RoboBrain：面向机器人操作从抽象到具体的统一大脑模型（CVPR 2025）

FlagOpen; RoboBrain

Repository navigation

NotificationsYou must be signed in to change notification settings; Fork44 (44); Star564 (564)You must be signed in to star a repository

[CVPR 2025] RoboBrain: A Unified Brain Model for Robotic Manipulation from Abstract to Concrete. Official Repository.

Repository files navigation

README; Apache-2.0 license

🔥🔥 RoboBrain 2.0: More powerful version of RoboBrain: See Better. Think Harder. Do Smarter.

🎯 RoboOS: An Efficient Open-Source Multi-Robot Coordination System for RoboBrain.

🎯 Reason-RFT: Exploring Efficient RFT Paradigm to Enhance RoboBrain's Visual Reasoning Capabilities.

🔥 Overview

Recent advancements in Multimodal Large Language Models (MLLMs) have shown remarkable capabilities across various multimodal contexts. However, their application in robotic scenarios, particularly for long-horizon manipulation tasks, reveals significant limitations. These limitations arise from the current MLLMs lacking three essential robotic brain capabilities: (1) Planning Capability, which involves decomposing complex manipulation instructions into manageable sub-tasks; (2) Affordance Perception, the ability to recognize and interpret the affordances of interactive objects; and (3) Trajectory Prediction, the foresight to anticipate the complete manipulation trajectory necessary for successful execution. To enhance the robotic brain's core capabilities from abstract to concrete, we introduce ShareRobot, a high-quality heterogeneous dataset that labels multi-dimensional information such as task planning, object affordance, and end-effector trajectory. ShareRobot's diversity and accuracy have been meticulously refined by three human annotators. Building on this dataset, we developed RoboBrain, an MLLM-based model that combines robotic and general multi-modal data, utilizes a multi-stage training strategy, and incorporates long videos and high-resolution images to improve its robotic manipulation capabilities. Extensive experiments demonstrate that RoboBrain achieves state-of-the-art performance across various robotic tasks, highlighting its potential to advance robotic brain capabilities.

🚀 Features

This repository supports:

Data Preparation: Please refer to Dataset Preparation for how to prepare the dataset; Training for RoboBrain: Please refer to Training Section for the usage of training scripts; Support HF/VLLM Inference: Please see Inference Section, now we support inference with VLLM; Evaluation for RoboBrain: Please refer to Evaluation Section for how to prepare the benchmarks; ShareRobot Generation: Please refer to ShareRobot for details

2025-06-06: 🤗 RoboBrain 2.0-7B model checkpoint has been released in Huggingface; 2025-06-06: 🔥 We're excited to announce the release of our more powerful RoboBrain 2.0; 2025-04-11: 🎉 RoboBrain was selected for CVPR 2025's official Embodied AI Trends Commentary; 2025-04-04: 🤗 We have released Trajectory Checkpoint (T-LoRA) in Huggingface; 2025-03-29: 🤗 We have released Affordance Checkpoint (A-LoRA) in Huggingface; 2025-03-27: 🤗 We have released Planning Checkpoint in Huggingface; 2025-03-26: 🔥 We have released the RoboBrain repository; 2025-02-27: 🌍 Our RoboBrain was accepted to CVPR2025

📆 Todo

Release scripts for model training and inference; Release Planning checkpoint; Release Affordance checkpoint; Release ShareRobot dataset; Release Trajectory checkpoint; Release more powerful Robobrain 2.0

Base Planning Model: The model was trained on general datasets in Stages 1–2 and on the Robotic Planning dataset in Stage 3, which is designed for Planning prediction; A-LoRA for Affordance: Based on the Base Planning Model, Stage 4 involves LoRA-based training with our Affordance dataset to predict affordance; T-LoRA for Trajectory: Based on the Base Planning Model, Stage 4 involves LoRA-based training with our Trajectory dataset to predict trajectory

Note: Please refer to RoboBrain 2.0 Github for the usage of RoboBrain 2.0

Note: The sample format in each json file should be like:

2. Training

Note: Please change the environment variables (e.g. DATA_PATH, IMAGE_FOLDER, PREV_STAGE_CHECKPOINT) in the script to your own.

3. Convert original weights to HF weights

(Option) 4. Compress Model

Model compression by Flagscale, the model is compressed to W8A16, reducing the model size by more than 40%. The inference speed can be accelerated by up to 50%, but the generated results after compression may differ slightly from those before compression.

⭐️ Inference

1. Usage for Planning Prediction

Run python script as example:

Option 3: FlagScale Serve

FlagScale is a comprehensive toolkit designed to support the entire lifecycle of large models, developed with the backing of the Beijing Academy of Artificial Intelligence (BAAI). FlagScale supports distributed deployment of large models, which can be launched easily through a simple configuration file.

Set config file and install FlagScale

For better integration with FlagScale, you can optionally install vLLM inside FlagScale.

Launch service

The Python script used to call the service remains the same as before.

2. Usage for Affordance Prediction

3. Usage for Trajectory Prediction

🤖 Evaluation

😊 Acknowledgement

We would like to express our sincere gratitude to the developers and contributors of the following projects:

LLaVA-NeXT: The comprehensive codebase for training Vision-Language Models (VLMs); lmms-eval: A powerful evaluation tool for Vision-Language Models (VLMs); vllm: A high-throughput and memory-efficient LLMs/VLMs inference engine; OpenEQA: A wonderful benchmark for Embodied Question Answering; RoboVQA: Provide high-level reasoning models and datasets for robotics applications

Their outstanding contributions have played a pivotal role in advancing our research and development initiatives.

📑 Citation

If you find this project useful, welcome to cite us.

[CVPR 2025] RoboBrain: A Unified Brain Model for Robotic Manipulation from Abstract to Concrete. Official Repository.