智源 FlagOpen 开源跨本体基础模型 RoboBrain-X0

FlagOpen; RoboBrain-X0

Repository navigation

NotificationsYou must be signed in to change notification settings; Fork12 (12); Star118 (118)You must be signed in to star a repository

Repository files navigation

README; Apache-2.0 license

⭐️ Project | 🤗 Hugging Face | 🤖 ModelScope

⚡ OmniSAT: Compact Action Token, Faster Auto Regression for RoboBrain-X0 Pretraining.

🚀 RoboBrain 2.0: See Better, Think Harder, Do Smarter beyond RoboBrain1.0.

⭐️ Reason-RFT: Core Post-Training Strategy for Embodied Visual Reasoning in RoboBrain2.0.

🎯 RoboOS: An Efficient Open-Source Multi-Robot Coordination System for RoboBrain.

🌍 RoboBrain 1.0: A Unified Brain Model for Robotic Manipulation from Abstract to Concrete.

🔥 Overview

We are thrilled to introduce RoboBrain-X0, a groundbreaking cross-ontology foundation model designed to overcome the limitations of single-robot systems in heterogeneous ontology transfer. By leveraging End-Effector pose representation in SE(3) task space, coupled with a Unified Action Vocabulary (UAV) and action tokenizer, RoboBrain-X0 achieves efficient zero-shot generalization and complex task decomposition. Its Grouped Residual Quantizer (GRVQ) maps continuous control sequences from diverse degrees of freedom and mechanical structures to a shared discrete action primitive space, ensuring semantic consistency and transferability across ontologies such as AgileX, R1-Lite dual-arm wheeled robots, and Franka single-arm systems. Through ontology-conditioned diverse prompting, the model supports flexible decoding from multi-view RGB-D inputs to specific executions, significantly reducing training and inference overhead. RoboBrain-X0 delivers state-of-the-art performance in embodied reasoning tasks, laying a robust foundation for developing versatile, real-world robotic agents and advancing embodied intelligence research.

2025-09-29: 🔥 RoboBrain-X0-preview was released

📆 Todo

Release model checkpoint for RoboBrain-X0-Preview based on Fast Tokenizer; Release quick inference example for RoboBrain-X0; Release training and evaluation codes for RoboBrain-X0; Release omnisat-based RoboBrain-X0 full version (coming soon)

🚀 Features

RoboBrain-X0 supports the unified modeling of heterogeneous ontologies and offers zero-shot generalization and complex task decomposition capabilities. Building on RoboBrain's multimodal foundation, RoboBrain-X0 further integrates real-world robot motion data based on RoboBrain-2.0 data. By unifying vision, language, and motion modeling, it achieves cross-ontology generalization and adaptation, providing integrated capabilities from perception to execution.

⭐️ Architecture

This model includes RoboBrain 2.0 and OmniSAT (action tokenizer). Based on RoboBrain 2.0, the model is trained on a large amount of real-world robotics data and embodied reasoning data, enabling it to possess general robotic manipulation capabilities. The action token sequences output by the model are converted into underlying robot control signals through our proprietary action tokenizer. Model details are as follows:

Multimodal Input: The model accepts single-image, multi-image, and text input (covering pointing task scenarios, object maneuverability scenarios, trajectory scenarios, and subtask execution scenarios), and produces outputs of varying dimensions based on the input scenarios; Action Generation and Execution: After model processing, OmniSAT converts these into multi-degree-of-freedom (DoF) action sequences, ultimately driving the robot to complete the operation

Take the open-source Agilex data from RoboBrain-X0-Dataset as an example.

Run normal.py to generate the normalization parameter file:

This step will produce the parameter file and action data distribution plots.

Update data_process/data_process.sh with the correct file path and process number, then run it to generate training data in parallel:

🤖 Training

🔍 Evaluation

The evaluation on real robots adopts a "server-client" architecture, with a PC acting as a data transfer hub: the model is deployed on a remote server (responsible for inference), while the client (running on the robot side) sends the robot's state to the server. After the model performs inference, it returns the generated action results to the client, which then controls the robot to execute these actions.

Real Robots Evaluation Architecture Overview

The system operates on a simple yet effective server-client model:

Server (Inference Side): A remote machine (usually with a powerful GPU) where the trained model is deployed.

It receives the robot's state and images; It performs inference to decide on the next sequence of actions; It sends the generated actions back to the client in the HTTP response; Client (Robot Side): The Python script running on the robot's control computer.

It continuously captures the robot's current state (e.g., end-effector pose, joint angles) and images from its cameras; It packages this observation data into a JSON payload; It sends the data to the remote inference server via an HTTP POST request

This decoupling allows for running complex, resource-intensive models without needing high-end computing hardware directly connected to the robot.

How to Run a Server

Each server can be configured and run independently. Here are the general instructions.

Choose a server script (e.g., serve_franka.py); Configure the mode: Open the script and set the SUBTASK_MODE variable:

SUBTASK_MODE = False for standard action generation; SUBTASK_MODE = True for subtask + action generation; Verify paths: Ensure that MODEL_PATH and STATS_PATH (if applicable) point to the correct files for your chosen mode; Start the server:

Examples

Inference code examples are provided for four robot platforms: A2D, Franka, Agilex, and R1lite (ROS1 and ROS2 version).

Agilex; Franka; R1-Lite; A2D

📑 Citation

If you find this project useful, welcome to cite us.