智源 FlagOpen 开源 ShareRobot 机器人异构数据集
GitHub - FlagOpen/ShareRobot · GitHub
智源 FlagOpen 开源 ShareRobot 机器人异构数据集，标注任务规划、物体可供性与末端执行器轨迹。数据集含 51,403 个 episode（每个 30 帧，转为 1,027,990 个问答对）、6,522 张可供性图像和 6,870 张轨迹图像，来自 Open X-Embodiment 的 23 个原始数据集、12 种本体和 107 类原子任务。基于该数据集训练的 RoboBrain 任务分解准确率提升 30.2%（BLEU-4 达 55.05%），轨迹预测误差降低 42.9%。
正文
当前显示原文，中文全文译文尚未就绪。
FlagOpen; ShareRobot
Repository navigation
NotificationsYou must be signed in to change notification settings; Fork3 (3); Star62 (62)You must be signed in to star a repository
Repository files navigation
README
ShareRobot, a high-quality heterogeneous dataset that labels multi-dimensional information, including task planning, object affordance, and end-effector trajectory, effectively enhancing various robotic capabilities.
Overview of ShareRobot
For planning, we have 51,403 episodes and each with 30 frames. In the process of data generation, we design 5 different templates for each of the 10 question types in RoboVQA [1]. In the process of data generation, we randomly select 2 templates of each question type to generate question-answer pairs for every instance. This process transforms 51,403 instances into 1,027,990 question-answer pairs, with annotators monitoring data generation to maintain the dataset’s integrity.
For Affordance, we have 6,522 images and each with affordance areas aligned with an instruction.
For Trajectory, we have 6,870 images and each with at least 3 {x, y} coordinates aligned with an instruction.
Data Sources🌍
ShareRobot dataset contains 23 original datasets from Open X-Embodiment dataset [2], 12 embodiments and 107 types of atomic tasks.
Powered by ShareRobot dataset, RoboBrain Model achieves stunning results.🌟
Task planning capability: The RoboBrain model trained on ShareRobot achieves a 30.2% improvement in task decomposition accuracy (BLEU-4 reached 55.05%), significantly better than existing methods;
Affordance perception capability: The average accuracy (AP) of object affordance area recognition is 27.1%, which is 14.6% higher than the baseline model.
Trajectory prediction capability: End-effector trajectory prediction error reduced by 42.9% (DFD index decreased from 0.191 to 0.109);
General capability: In the OpenEQA benchmark, the scene understanding score surpasses general multimodal models such as GPT-4V. The RoboBrain model trained with ShareRobot did not sacrifice its general ability.
Reference
[1] Pierre Sermanet, Tianli Ding, Jeffrey Zhao, Fei Xia, Debidatta Dwibedi, Keerthana Gopalakrishnan, Christine Chan,Gabriel Dulac-Arnold, Sharath Maddineni, Nikhil J Joshi,et al. Robovqa: Multimodal long-horizon reasoning forrobotics. In ICRA, pages 645–652, 2024.
[2] Abby O’Neill, Abdul Rehman, Abhinav Gupta, AbhiramMaddukuri, Abhishek Gupta, Abhishek Padalkar, AbrahamLee, Acorn Pooley, Agrim Gupta, Ajay Mandlekar, et al.Open x-embodiment: Robotic learning datasets and rt-xmodels. arXiv preprint arXiv:2310.08864, 2023.