InternLM 开源 ARM-Thinker：用智能体工具调用强化多模态生成式奖励模型

InternLM; ARM-Thinker

Repository navigation

NotificationsYou must be signed in to change notification settings; Fork12 (12); Star198 (198)You must be signed in to star a repository

Repository files navigation

README

Shengyuan Ding · Xinyu Fang · Ziyu Liu · Yuhang Zang† · Yuhang Cao · Xiangyu Zhao . Haodong Duan · Xiaoyi Dong · Jianze Liang · Bin Wang · Conghui He · Dahua Lin · Jiaqi Wang†

🏠 Github Repo: ARM-Thinker

📖 Paper | 🤗 HF Collection | 🤗 Daily Paper

🚀 [2025/12/05] We have released the ARM-Thinker Code Repository

🧩 Easy to Use & Extend

Clear & customizable framework — The agent loop, tool-call template, and action logic are fully modular. You can plug in your own LLM/VLM, modify the loop, or replace the template with minimal changes in arm_agent; Instant plug-and-play tools — New tools can be added by simply implementing a Verl-style tool. Inference and training share the same interface, ensuring consistent behavior; Easy evaluation — ARM-Thinker can be directly integrated into VLMEvalKit, allowing you to test agent based on our loop on standard benchmarks while preserving full trajectory logs for analysis

We hope to provide a flexible and easy-to-use framwork of both inference and training for agentic multimodal reasoning. For detailed code usage, you can refer to the Code Usage Instructions section below.

🌈 Overview

Current reward models suffer from hallucination and weak visual grounding due to their static, non-interactive design. We introduce ARM-Thinker, an Agentic Multimodal Reward Model that replaces passive scoring with an active “Think–Act–Verify” pipeline. ARM-Thinker autonomously invokes external tools—such as image cropping and multi-page document retrieval—to ground its judgments in verifiable evidence. Through multi-stage reinforcement learning, the model jointly optimizes tool-use decisions and reward accuracy.

(a) Case Comparison: Given a complex document QA task, ARM-Thinker correctly identifies the answer by autonomously invoking the retrieval tool, while the baseline model provides an incorrect response; (b) ARMBench-VL: It evaluates reward models across three task types, each requiring specialized tool use (image manipulation, document retrieval, instruction verification); (c) Performance of ARM-Thinker: The agentic capability enables substantial gains across multiple benchmarks including reward modeling, tool usage, and reasoning

ARM-Thinker combines an agentic think–act–observe loop with a reinforcement-trained reward model. During inference, the agent iteratively invokes tools—such as image zoom-in, document retrieval, and instruction validators—and integrates the returned evidence into its multimodal context, enabling grounded and verifiable judgment. To train this behavior, we first perform SFT + cold-start initialization to establish stable tool-use patterns. We then apply a two-stage GRPO process: Stage 1 encourages correct tool selection, while Stage 2 refines reward accuracy using verifiable rewards that balance correctness and tool efficiency. Together, these components equip ARM-Thinker with reliable, interpretable, and tool-augmented multimodal reasoning capabilities.

💡 Highlights

🔥 Agentic "Think–Act–Verify" Paradigm

ARM-Thinker shifts multimodal reward modeling from passive, single-pass scoring to an agentic decision-making process. Through its explicit Think–Act–Verify loop, the model autonomously decomposes complex queries, invokes external tools as needed, and grounds judgments in verifiable evidence—eliminating hallucination and superficial fluency common in static models.

To address the gap in current evaluation methods, we introduce ARMBench-VL, the first benchmark designed for multi-step, evidence-grounded reward models. It evaluates across three critical dimensions: fine-grained visual grounding, long-document reasoning, and instruction-level verification—highlighting the limitations of passive models in reasoning, retrieval, and validation across modalities.

ARM-Thinker utilizes a multi-stage GRPO framework that separately optimizes tool-use policy and reward estimation, before jointly refining them. This approach ensures stable credit assignment, resolves training-inference misalignment, and prevents reward drift—issues that commonly affect static reward model pipelines.

🔥 Efficient, Scalable, and Competitive Performance

Built on a Qwen2.5-VL-7B backbone, ARM-Thinker demonstrates +16.2% improvement in reward modeling and +9.6% in tool-use tasks. Its agentic reasoning and modular tool integration allow easy expansion with new tools, making it highly scalable for diverse tasks. ARM-Thinker competes effectively with larger models like Qwen3-VL-8B and GPT-4o, proving that evidence-grounded decision-making outperforms scale alone in achieving robust multimodal reasoning.

⭐️ Code Usage Instructions

Part 0: Environment Setup

[For inference] If you only want to run the agent loop, you only need to install the requirements in arm_agent_requirements.txt and install verl as an editable package.

[For training] If you want use the training code, you need to install the full requirements in requirements_training.txt and install vllm as an editable package.

Part 1: Agent Loop

The whole agent loop is in module of arm_agent, including the main workflow of agent loop and a simple test script telling how to use the agent loop.

1. Tool call template

You can modify the tool call template in arm_agent/agent_verl.py to adapt to your own model. We use Hermes format as the default tool call template of our ARM-Thinker model.

2. Tool configuration and Tools Implementation

For both training and inferencing, we implement the tools directly under the framework of Verl.

You can easily add your own tools under the verl/tools directory. (Already have some widely used tools like image zoom in, web search, ocr, document retrieval, etc.); Then you need to define the tool configuration yaml file, there are some examples in examples/self/multiturn/config/tool_config directory and a demo file of arm_agent/tests/config/test_image_zoom_in_tool_config.yaml

3. For trajectory visualization

We provide a notebook arm_agent/tests/visualize_result.ipynb to visualize the trajectory of the agent. (For example, the output after running the arm_agent/tests/test_agent_image_zoom_in.py)

Example. Qwen3-VL-8B with Image Crop and Zoom-in tools:

You can try to run the loop like below to see the example output:

Remember to set the config in test_agent_image_zoom_in.py to your own model.

This will generate the result.json file under the arm_agent/tests directory, then you can visualize the trajectory by running the corresponding code segment in visualize_result.ipynb notebook.

Refer to case in visualize_result.ipynb

Part 2: SFT & Cold-start Initialization

We use LLaMA-Factory to perform SFT and cold-start initialization and the training script is in sft_script directory. You can refer to sft_scripts/sft.sh for the training details and modify it to your own training settings.

Part 3: GRPO Training with Verl

GRPO training script for tool of image_crop_and_zoom_in is in examples/self/multiturn/agent/crop.sh and you can modify it to your own training task.

All of the reward functions of each stage are under the verl/utils/reward_score directory and the classification of the different reward functions for tasks is in verl/utils/reward_score/__init__.py.

Part 4: Evaluation

We integrate ARM-Thinker with VLMEvalKit to evaluate the performance of ARM-Thinker on widely used benchmarks.

✒️ Citation

If you find our work helpful for your research, please consider giving a star and citation ~

📄 License

Usage and License Notices: The data and code are intended and licensed for research use only. License: Attribution-NonCommercial 4.0 International It should abide by the policy of OpenAI: https://openai.com/policies/terms-of-use

❤️ Acknowledgement

We sincerely thank projects VLMEvalKit, Verl and LLaMA-Factory for providing their open-source resources.