FlagOpen 开源 RoboBrain_Dex 灵巧操作基础模型
GitHub - FlagOpen/RoboBrain_Dex · GitHub
智源 FlagOpen 开源 RoboBrain_Dex，一个基于多源第一人称数据集预训练的基础模型，主打灵巧操作能力。该模型以 RoboBrain2.0 作为视觉语言基座，通过自研运动动力学模型进行蒸馏，并同时在多源人类第一人称演示数据上预训练；所有真实世界后训练均使用预训练阶段未接触过的本体数据。仓库提供将 Unitree G1 官方 JSON 数据转为 RLDS 格式的脚本，并给出 pour the drink 任务的后训练数据作为格式参考。
正文
当前显示原文，中文全文译文尚未就绪。
FlagOpen; RoboBrain_Dex
Repository navigation
NotificationsYou must be signed in to change notification settings; Fork6 (6); Star42 (42)You must be signed in to star a repository
Repository files navigation
README
Robobrain-Dex is a foundation model pre-trained on multi-source egocentric datasets, demonstrating exceptional capabilities in dexterous manipulation.
Architecture
Robobrain-Dex utilizes Robobrain2.0 as the base Vision-Language Model. It undergoes distillation using our custom-designed motion dynamics model while simultaneously being pre-trained on multi-source human egocentric demonstration data. All real-world post-trainings are conducted using embodiment data that was not encountered during the pre-training phase. The model architecture is illustrated in the figure below:
Set Up Conda Environments
Data Process
We use RLDS format data for model finetuning. We provide a script to convert JSON data from Unitree's official data collection format (avp_teleoperate/tree/g1) to RLDS.
G1 (Unitree): Convert from Unitree official JSON format to RLDS (data layout: / /episode_XXXXXX/data.json and colors/ images). The JSON must record arm end-effector poses in the base frame (left_hand_pos_state, right_hand_pos_state) and finger tip poses in the wrist frame (left_fingertip_pos_state, right_fingertip_pos_state)
We provide post-training data for the task of pour the drink, which can be used as a reference for the data format.
After converting to RLDS, register the dataset (which, for example, pour_the_drink) with our dataloader by adding an entry for it in configs.py (here), transforms.py (here). For reference, in each of these files, there are sample entries for the G1 datasets that we used in our paper.
The following features are planned for future implementation:
Complete data processing scripts and documentation; Complete deployment scripts on real robots; Complete pretraining scripts and documentation
Acknowledgments
Robobrain-Dex builds on the following excellent open-source projects:
RoboBrain; OpenVLA; Univla
We thank the authors for their contributions to the robotics and machine learning communities.
Citation
If you find our work useful, please consider citing us and give a star to our repository! 🌟🌟🌟