Tsinghua team builds the HandsOn dexterous manipulation benchmark, with 66 tasks comprehensively evaluating robots' two-handed capabilities

The "exam paper" for robot manipulation needs a new version written for dexterous hands.

Past robot manipulation benchmarks have mostly revolved around grippers. But as egocentric (ego) human manipulation data continues to accumulate, how humans grasp, use tools, and complete fine contact is being recorded at an increasingly large scale. Dexterous hands are structurally similar to human hands and are expected to leverage this data for larger-scale learning, becoming an important direction for robot manipulation.

However, dexterous hands keep evolving, while the "exam paper" has not kept up.

Dexterous hands have a large number of active joints, tasks are full of contact, often require two-handed collaboration, and must maintain prior results across long action sequences; success is hard to judge, and it is hard to pinpoint which step a failure occurred at. Benchmarks from the gripper era cannot directly answer these questions, and a systematic, reliable evaluation framework for dexterous hands is still missing.

Data and models are moving forward, yet there is no ruler to measure progress.

Recently, Ding Ning's team at Tsinghua University, together with researchers from Beijing University of Posts and Telecommunications, Peking University, and Nature Will, jointly built a composable simulation platform and evaluation benchmark for cross-embodiment dexterous manipulation, HandsOn, hoping to fill this gap.

HandsOn contains 66 manually verified tasks and 18,000 human simulation teleoperation demonstrations, with evaluations centered on tool use, fine contact, two-handed collaboration, and task transfer. The platform supports 5 types of robotic arms, 6 types of dexterous hands, and about 200 rigid-body, articulated-object, and deformable-object assets, and can combine them in randomized scenes.

HandsOn contains 18,000 curated simulation teleoperation demonstrations, with tasks divided into Core-50 and Transfer-16.

The report card this benchmark produces is not easy: the team evaluated different approaches such as diffusion policies, vision-language-action models, and world action models. Results show that among the Core-50 tasks that provide training demonstrations, the best-performing trained multi-task policy achieved a success rate of 29.1%. On Transfer-16, which has no training demonstrations for the target tasks, the average task success rates of all 8 trained multi-task policies were below 1%.

In addition, GPT-6, without fine-tuning on benchmark data, controlled the palm and fingers through the move_eef interface; across 660 tests, its success rates on Core-50 and Transfer-16 were 5.2% and 1.9%, respectively.

The team further traced stages such as grasping, contact, two-handed coordination, and target-state maintenance, and found that robots can often complete the first half of an action but fail at the critical moment.

The related paper, code, and data resources have been released.

Why do dexterous hands need a new evaluation benchmark?

The difficulty of dexterous hands is not just having a few more fingers than a gripper, but also needing to control the constantly changing contact relationships during manipulation.

After picking up a mouse, the robot needs to avoid triggering a click prematurely; when setting two knobs, it needs to keep them both at the target positions simultaneously; when placing a box lid, it must maintain the original alignment after letting go. These tasks all require the robot to coordinate finger movements, judge contact states, and ensure that already-completed operations are not disrupted by subsequent actions.

This also changes how robot manipulation is evaluated. For grippers, whether a grasp succeeds and whether an object is placed at the specified position are often important criteria. For dexterous hands, merely grasping an object is far from enough. The robot also needs to actually use tools, complete fine operations, and maintain correct object states in multi-step tasks. Progress in one action does not mean the entire task has succeeded.

The accumulation of egocentric human manipulation data provides new opportunities for dexterous hands to learn these skills. But as demonstration data increases, two questions still need to be answered: can the model stably reproduce the manipulations it has learned? And when facing new objects, task combinations, or deformable objects, can it transfer existing skills?

Existing dexterous manipulation benchmarks have covered different robot embodiments, functional manipulation, and two-handed collaboration. However, related research still has room for further improvement in areas such as the scale of human demonstration data, unseen tasks specifically held out for evaluation, and diagnosis of manipulation failure processes.

HandsOn's design starts from here, measuring execution capability and generalization capability separately, dividing 66 tasks into two evaluation sets, Core-50 and Transfer-16.

Core-50 provides demonstration data for policy learning and evaluates policy performance through a new round of independent execution tests; Transfer-16 is specifically for generalization evaluation, and demonstration data for the target tasks is not included in the training set.

HandsOn suite: Core-50 includes demonstrations; Transfer-16 is for evaluation only.

This design not only tests whether the robot can complete tasks with demonstrations, but further examines whether existing skills can transfer to new manipulation conditions. Combined with stage-wise diagnosis, researchers can also analyze whether failures occur in contact establishment, functional manipulation, two-handed coordination, target-state maintenance, or the connection between subsequent steps.

Core-50: After learning a task, can it be executed stably?

Core-50 contains 50 tasks that provide training demonstrations, covering capabilities such as functional understanding, numerical input, spatial relations, articulated manipulation, precise contact, and two-handed collaboration.

These categories correspond to specific real-life scenarios. When operating scissors, the robot needs to lift them off the tabletop and complete three open-close cycles while maintaining a continuous grasp; when setting a timer, it needs to enter the specified minutes and seconds, then withdraw both hands after starting it; unfolding a folding fan requires both hands to grip the outer ribs, unfold it to the target angle, and hold it there. Making coffee further links together steps such as picking up the cup, closing the door, receiving the coffee, and placing it down, testing the transitions between actions.

Six task instances in Core-50. Each task has explicit goals, motion constraints, and completion conditions; all images are original simulation-scene images.

Task design unfolds along three dimensions.

The first dimension is interaction type, including basic manipulation, functional use, and contact-intensive manipulation.

The second dimension is task structure, including single-stage manipulation, repetitive manipulation, and long-horizon tasks.

The third dimension is object type, covering rigid bodies, objects with revolute or prismatic joints, and deformable objects.

In Core-50, the three interaction types contain 17, 18, and 15 tasks respectively; the three dimensions are cross-combined so that the benchmark can cover manipulation scenarios of varying difficulty.

Transfer-16: With different objects and tasks, can it still complete them?

Transfer-16 contains 16 evaluation tasks for which no training demonstrations of the target task are provided, further examining whether models can transfer existing skills. Among them, 7 tasks examine asset or relation transfer, 6 tasks examine skill composition, and the other 3 involve deformable object manipulation.

Asset or relation transfer requires the model to find the correct object to manipulate and complete the task when facing new objects or target relations. Skill composition requires the model to connect previously learned actions in the correct order. Deformable object manipulation introduces towels, sponges, and pillows, requiring the model to cope with changes in object shape during manipulation.

For example, in the task of handing over a block and then closing the box lid, the robot needs to first complete a two-handed handover, then place the block into the box, and finally align and close the lid. Each local action may already have appeared in other tasks, but the model still needs to judge whether the current step is complete and whether the result achieved in the previous step can be maintained.

The requirement for continuously folding a towel is even more complex. The robot needs to complete two consecutive folds and, after releasing, leave the towel stably on the tabletop. Deformable objects change shape with the grasp position and the way force is applied, making the manipulation process harder to predict.

Representative tasks in Transfer-16, demonstrating asset transfer, skill composition, and deformable object manipulation.

It can be seen that Core-50 mainly tests learning and execution capability on tasks with demonstrations, while Transfer-16 further tests whether existing capabilities can be transferred and composed.

Making failures accurately recorded too

To ensure trustworthy evaluation, HandsOn creates a task card for each task, specifying the goal, standard initial state, expected final state, allowed and prohibited actions, and the evidence for judging whether the task is complete.

Completion conditions can be judged based on information such as object poses, joint states, or images. The research team also conducts manual operation verification of the tasks and sets up an independent review process to check whether success labels are consistent with the actual physical state.

In multi-step tasks, the evaluation also records stage progress completed in order. For example, the block handover task distinguishes between both hands jointly contacting the object and the receiving hand already being able to support the object independently. Skill composition tasks check whether prerequisite steps are completed and whether existing results are maintained.

This allows researchers to distinguish between two situations: the model completely failing to complete the task, or having completed part of it but failing in subsequent steps.

After 18,000 demonstrations, what has the robot learned?

Unlike gripper data, dexterous hand demonstrations need to record fine-grained actions at the finger level. HandsOn adopts human teleoperation, converting human hand manipulation into dexterous hand action demonstrations, which also echoes the development direction of first-person human manipulation data.

Around this task suite, HandsOn collected about 56 hours of human manipulation records, involving at least 10 operators and more than 1,050 collection sessions. Operators wear MANUS data gloves and use wrist tracking devices to control the simulated robot, while the system synchronously saves images, states, actions, and language information.

Of the 18,000 demonstrations, 15,178 are used for training and 2,822 for validation. Each interaction includes three image streams: front and left/right wrist views. The front view presents the tabletop layout and the positions of both hands, while the wrist views provide local observations close to the fingers and objects. For tasks requiring precise contact, the local images can supplement information on how the fingers approach, grasp, and manipulate objects.

As data scale increases, how does the success rate change? The research team selected two tasks, "put the ball into the bowl" and "lift the scissors and open-close them," and conducted single-task experiments with π0.5, increasing the training demonstrations successively to 10, 50, 100, and 200, and finally using the full training pool.

π0.5 single-task demonstration scale experiment; the full training pools contain 438 and 439 demonstrations respectively. Each checkpoint is trained for 20,000 updates and tested for 100 episodes; shading indicates the 95% Wilson interval. A single training seed is used, and the intervals do not include uncertainty from training seeds or subset selection.

The results show that the success rate of the ball-placement task increased from 7% to 83%, and that of the scissors task increased from 1% to 44%, with both tasks improving consistently across five test scales. The full training pools contain 438 and 439 demonstrations respectively; each checkpoint was trained for 20,000 updates, and performance was measured through 100 episodes of new closed-loop execution.

This set of results provides an intuitive reference for the data gains. As demonstrations increased, the model achieved higher success rates on both tasks; at the same time, a considerable gap remained between the scissors manipulation and the ball-placement task, indicating the difficulty of combining stable grasping with tool-functional actions. The experiment used a single training seed, and the reported 95% Wilson intervals reflect the uncertainty of the test episodes; variation due to training seeds and subset selection still requires further examination.

Data can also be expanded in terms of visual appearance. HandsOn supports high-fidelity re-rendering of the same manipulation segment, changing the tabletop material, lighting, room background, and small camera offsets while preserving the original state and action supervision. Researchers can thereby generate different observations around the same interaction and examine how policies adapt to appearance changes.

In terms of robot configuration, HandsOn establishes a simulation counterpart around a real ROKAE robotic arm and the first-generation Wuji Hand, with each hand containing 20 active joints. Kinematics, joint limits, asset scale, camera calibration, and state and action coordinates are all included in the corresponding records. The main camera and wrist cameras work together to bring the overall layout and hand details into the same data and evaluation pipeline.

Comparison of simulation and real-robot views for the toy airplane scene, with wrist images shown enlarged.

The platform supports 5 robotic arms, 6 dexterous hands, and about 200 assets; this data collection and quantitative evaluation uniformly adopt the ROKAE—Wuji and PhysX configuration. The team also provided real-robot operation videos of policies adapted using simulation demonstrations, qualitatively demonstrating contact, transport, and pose maintenance under the condition of no target-task real-robot demonstrations. Regarding real-world execution performance, observations can currently be made in conjunction with these videos, whose scope of evidence differs from the quantitative evaluation in simulation.

A 29.1% success rate—where is the difficulty in two-hand coordination?

HandsOn evaluates 10 methods in total, including 5 vision-language-action models (VLA), 3 world-action models (WAM), Diffusion Policy trained separately per task, and the zero-shot control reference GPT-6. The first 9 methods are trained using HandsOn demonstrations, of which 8 use multi-task training.

On Core-50, the 9 trained methods completed nearly 45,000 valid tests in total; the highest task-average success rate among multi-task policies was achieved by OpenWAM, at 29.1%. Differences between tasks are relatively obvious: grasping, transporting, and some joint operations are relatively easy, while tasks such as using a tool to act on a third object and completing precise coordination remain difficult.

The transfer evaluation results are even more severe. On Transfer-16, the task-average success rates of all 8 multi-task trained policies were below 1%; the single-task Diffusion Policy did not participate in this part of the evaluation. Faced with new assets, skill combinations, and deformable objects, the ability of existing training experience to translate into complete task success remains limited.

When complete successes are few, staged progress provides an entry point for further analysis. The dual-knob task is one example: two hands operate the knobs separately, adjusting both sides to the target state and satisfying conditions such as holding and releasing. In 800 attempts by 8 multi-task models, 219 times both sides each reached the target, 38 times both were in position simultaneously, and ultimately 5 times the complete conditions were satisfied.

The gap mainly arises in how the two local results are made to hold jointly. A common situation is that the hand that reached position first continues to turn, and before the other hand has completed its part, the existing result has already changed.

The block handover reflects the timing requirements between the two hands. In 367 episodes in which both hands jointly held the block, 188 achieved independent takeover by the receiving hand; of the remaining 179, 175 subsequently dropped it. When the giving hand releases needs to be coordinated with the support state of the receiving hand.

The dual-knob task totaled 800 episodes; of the 367 joint-contact episodes in block handover, 188 achieved independent takeover by the receiving hand.

Long tasks further add requirements on ordering. In the combined task requiring the kettle to be covered first and then the box, 340 of 400 attempts moved the box lid first. Such records provide traceable questions for subsequent research: how the model confirms that the current step is complete, how it preserves existing results, and when to initiate the next step.

The study also examines whether actions change with the language goal by keeping the image fixed and replacing the target object in the instruction. Some checkpoints still respond only limitedly to object names, suggesting that target selection likewise requires continued testing. A task failure may involve contact control, or may occur earlier at the stage of goal understanding and action selection.

GPT-6's zero-shot test provides another set of observations. In each round it reads three RGB images, the robot state, the task instruction, and execution feedback, and specifies the palm pose and up to 40 finger joint targets through move_eef.

Without HandsOn demonstration training or fine-tuning, the model tested each of all 66 tasks 10 times, achieving 26 Core-50 successes and 3 Transfer-16 successes, corresponding to success rates of 5.2% and 1.9%.

Action records show that the model can propose some plans related to manipulation preconditions. For example, when folding a towel, it once proposed first making a small trial lift to confirm that the towel edge moves with the hand, and then continuing to fold. That episode ultimately failed, reflecting the difficulty of translating a plan into contact and motion control.

For robots moving toward everyday environments, these details are all relevant to actual use. With task configurations, demonstration data, and evaluation methods made public, researchers can test new methods under a common setting and analyze the sources of success-rate changes along the execution process. As for how robotic hands can gradually master everyday operations, this benchmark provides a set of experimental conditions that can be compared continuously.

Conclusion: Let capability progress stand the test of practice

Hands are among the most exquisite organs of humans. People use tools and transform the environment through their hands, and also accumulate experience about objects and the world through repeated operations.

For robots to enter people's living and working spaces, they likewise need to integrate perception, judgment, and action into continuous physical interaction. For the robotics industry, the maturity of two-handed capabilities will affect how much practical work general-purpose models can undertake.

The robotics industry is gradually entering a stage that requires the accumulation of shared evidence. Technical demonstrations present the possibilities of capability, while scaled application requires that such capability stand up to repeated execution, scene changes, and long-term operation.

In this process, public benchmarks play the role of connecting research results with application needs. As model capabilities improve, evaluation also needs to absorb new tasks and constraints from practical applications, so that score growth corresponds to broader and more stable manipulation capabilities.

The long-term value of HandsOn will also gradually emerge as researchers participate: new methods drive performance improvements, failure cases bring new research questions, and application needs prompt continuous refinement of testing standards. For the industry, such accumulation helps form clearer capability boundaries and gives R&D investment and deployment expectations a more solid basis.

This article comes from the WeChat public account "Machine Heart" (ID: almosthuman2014), authors: Chen Chen, Clio, published by 36Kr with authorization.