蚂蚁 inclusionAI 开源 BiDex-100：双臂灵巧操作基准，含 106 项任务与 8080 条演示

inclusionAI; BiDex-100

Repository navigation

NotificationsYou must be signed in to change notification settings; Fork0 (0); Star0 (0)You must be signed in to star a repository

Repository files navigation

README; MIT license

A comprehensive benchmark for bimanual dexterous manipulation across 100+ simulation tasks

Website · Datasets · 3D Assets

Overview

BiDex-100 evaluates vision-language-action policies on coordinated, bimanual, dexterous manipulation. It combines declarative task definitions, reproducible MuJoCo environments, matched in-domain and out-of-domain evaluation, 3D assets, and 8,080 success-validated demonstrations.

106 tasks across five capability axes: Memory, Long-Horizon, Instruction-Following, Limited-Demonstrations, and Generalization; Two bimanual embodiments: dual Franka FR3 v2 arms and Astribot S1, both equipped with dexterous hands; 8,080 demonstrations and 6.25M frames: six synchronized RGB-D streams at 25 FPS, released as task-level LeRobot v2.1 and v3.0 datasets; 58-D unified action space: 14 arm/end-effector dimensions plus 44 dexterous-hand joint dimensions; Matched ID/OOD protocol: deterministic seeds, scene and object shifts, predicate-based progress scoring, and a strict score-100 success criterion

Installation

The Hugging Face dataset repository has three independently downloadable components:

Download everything and prepare the simulation assets:

Each component can instead be downloaded separately into the same destination:

scripts/download_assets.sh and scripts/download_datasets.sh remain as convenience wrappers. Run download_datasets.sh lerobotv21, download_datasets.sh lerobotv30, or download_datasets.sh all to fetch one or both dataset formats. The download destination preserves the repository layout shown above.

The asset setup script accepts either downloads/BiDex-100 or downloads/BiDex-100/assets. It verifies each archive against assets/SHA256SUMS, extracts it in place, creates the repository links, and recreates the relative aliases used by assembly tasks. It is idempotent, so an interrupted setup can be rerun safely. Alternatively set BIDEX_ASSETS=/path/to/BiDex-100/assets. Export HF_TOKEN instead of using hf auth login in non-interactive environments.

For headless GPU rendering, select EGL before starting Python:

Tasks

Task YAML files under bidex/tasks_data/suites/ / .yaml define the scene, robot, objects, initialization distribution, instruction, success predicates, subtasks, and OOD counterpart. The five suites contain 10, 10, 20, 26, and 40 tasks respectively (106 total).

This minimal example is complete and directly runnable. It disables cameras so that the first simulation check does not require a graphics context, resets one task, holds the reset robot pose, advances physics with env.step, and closes all MuJoCo resources:

Use list_suites() and list_tasks( ) to discover tasks. Pass cameras=["head", "left_wrist", "right_wrist"] when RGB observations are needed.

Training

BiDex-100 is model-agnostic: this repository provides the benchmark and data contract, while each VLA implementation owns its trainer. The supported 58-D representations are:

Use bidex.tasks.benchmark.BENCHMARKS and MODES as the canonical suite and modality registry when adding a trainer integration.

Reported training setup

The currently reported model results use co-training: demonstrations from multiple benchmark tasks are merged into one training corpus and used to train one shared policy checkpoint. Evaluation then runs that same checkpoint across all applicable tasks; the reported numbers are not an ensemble of separately trained per-task policies.

Evaluation

BiDex-100 exposes one evaluation contract for different VLA policies. A model adapter receives the same three camera observations, proprioceptive state, and language instruction; the benchmark owns task discovery, seeded resets, ID/OOD randomization, rollout execution, scoring, videos, and result schemas. This keeps model comparisons independent of model-specific launch code.

ID/OOD protocol summary

The canonical protocol evaluates 25 episodes for each of two ID seeds (0,1) and two OOD seeds (2,3). Both domains use deterministic per-episode reset seeds and the same task instructions, success predicates, rollout horizon, and scoring rules.

OOD appearance randomization is intentionally bounded but visually strong:

Lighting cycles through five exposure profiles from dark to high_bright with key/fill/ambient levels of 0.11/0.035/0.014, 0.18/0.060/0.022, 0.29/0.100/0.040, 0.40/0.145/0.058, and 0.48/0.180/0.072. The key light casts shadows and spans the full azimuth circle, elevations of 25/38/52/66/76 degrees, neutral/warm/cool tints, and distances of 3.0/3.5/4.0/4.4 m. The non-shadowing fill light provides an illumination floor to avoid unusably dark observations; Head-camera offsets are bounded to 1.8 cm and 6 degrees. Wrist-camera offsets are bounded to 1.35 cm and 6 degrees; the mount-clearance axis is restricted to outward motion for both Franka and Astribot so the robot connector cannot move in front of the lens. Across the canonical 50 OOD episodes per task, camera occurrence counts are exactly 10 nominal, 10 head-only, 10 wrists-only, and 20 head-and-wrists; Object/layout changes come from each task's committed OOD YAML. Distractor identity is fixed per task and OOD seed while placement is resampled per episode; task geometry and reachability constraints keep objects on usable, collision-safe tabletop regions

Randomization uses component-scoped SHA-256 keys, NumPy PCG64 streams, and hash-validated texture/camera manifests. Consequently, a fixed task, seed, and episode produces the same plan on different machines without depending on GPU or renderer output.

Seven reference integrations are included: π0.5, Galaxea G0.5, GR00T N1.7, LingBot-VLA, XR-1, InternVLA-A1.5, and DM0.5. Additional models can use the same driver by implementing the observation-to-action adapter used by bidex/eval/run_leaderboard_eval.py.

The launcher has no machine-specific defaults. Every integration supplies:

MODEL_PY: Python interpreter for the model environment; MODEL_REPO: model source directory; MODEL_CKPT: checkpoint path; BIDEX_LEADERBOARD_ROOT: output directory

MODEL_VLM_PATH is additionally required by LingBot-VLA, XR-1, and InternVLA-A1.5 because those adapters load a separate local VLM backbone. OPENPI_DATA_HOME is an optional native OpenPI cache override; when unset, OpenPI uses its normal cache.

For example, configure π0.5 without embedding paths in this repository:

Run one deterministic end-to-end smoke task before the full benchmark:

GPU layouts

MODEL_GPUS assigns policy servers and RENDER_GPUS assigns vectorized MuJoCo workers. The benchmark protocol and output schema do not change with the layout. Every layout uses the same server-client transport; a GPU present in both lists hosts separate policy-server and render-worker processes.

Use a distinct MID_OVERRIDE for every smoke or full run. BENCHES, BIDEX_TASKS, SEEDS, ID_SEEDS, OOD_SEEDS, EPS, ENVS_PER_GPU, MAX_RECORD_ENVS, and BIDEX_MAX_ENVS_PER_GPU refine the protocol without changing its scoring contract. GPUS remains a compatibility alias; explicit model/render lists are recommended.

Workers execute the full action chunk returned by each model by default. Set BIDEX_VEC_CHUNK_REUSE to a positive integer only to cap that native horizon for an explicit ablation; empty or zero keeps the model's maximum chunk size.

View results

MID_OVERRIDE identifies one evaluated checkpoint/configuration. Results, videos, and live logs are isolated under that identifier:

Start the leaderboard on all local interfaces:

Open http:// :12006/baselines/ from the local network. The page discovers model rows from results/ /, refreshes live scores, and serves rollout videos.

Licensing

Acknowledgements

BiDex-100 builds on MuJoCo, LeRobot, LIBERO, RoboCasa, robosuite, dexjoco, and Articraft.