Artificial Analysis 发布 AA-Robotics v0.1，用机械臂任务评测语言模型

All articles

We are launching AA-Robotics v0.1, our research preview that measures frontier language model performance on understanding, planning, and acting in the physical world. In v0.1, each model is given control of a 6-axis robot arm (5 joints and a gripper) and a simple objective. We’re releasing results for our first eight tasks today.

Introducing AA-Robotics v0.1

Since the beginning of 2026, frontier language models have made enormous progress in their ability to control robot arms. We now believe that the future of robotics is likely to involve strong generalist agents, and conclude that tracking progress of language models for robotics tasks is going to be important - even if frontier language models are unsuitable for production robotics use on their own.

AA-Robotics v0.1 is our research preview of the simplest experiment we have designed to test this - language models in coding agent harnesses directly controlling a robot arm. Each model is given control of a 6-axis robot arm (5 joints and a gripper) and a simple objective for each task.

We find that frontier language models in coding agent harnesses have indeed recently become capable of directly manipulating robot arms for simple tasks. Today, they are far too slow and expensive for any kind of production usage - nearly two orders of magnitude worse than human baselines. We expect this to change quickly.

GPT-6 Astra places first in AA-Robotics v0.1 with a score of 69, followed by GPT-6.1 Sol, Gemini 4 Argon, Claude Opus 5.5 and Claude Fable 5.1. GPT-6 Astra took 30 minutes and cost $16.36 on average per task.

AA-Robotics Score

Each attempt is scored out of 100: completing the task scores 100, and a partial result earns points for the progress made.

The eight tasks

AA-Robotics v0.1 initially includes eight tabletop tasks, modeled on everyday activities, to evaluate a range of robotics capabilities: control and dexterity, such as precise grasping and handling objects, spatial reasoning, such as understanding how objects interact with one another, and adaptability, measuring how models respond to the unexpected.

Press the red buttonGoal: Press the button with the gripper.Pass criteria: The button flashes; Place the crocodile toy in the rectangleGoal: Place the crocodile inside the rectangle.Pass criteria: The crocodile is entirely within the rectangle; Pour the drink into the cupGoal: Pour the drink from the bottle into the cup.Pass criteria: Drink is in the cup, none is spilled, and nothing is broken; Tidy the blocks into the boxGoal: Put all four blocks in the box, then put the lid back on.Pass criteria: All blocks are inside the box with the lid flush; Complete the shape matching challengeGoal: Place the blocks into the board so that it matches the picture.Pass criteria: Blocks in the board match the picture; Write hello worldGoal: Write “hello world” on the paper with the marker, inside the black box.Pass criteria: “hello world” is legible and inside the black box; Build the tallest towerGoal: Make the tallest tower out of the foam blocks.Pass criteria: Tower is freestanding and max achievable height; Measure the boxGoal: Measure the length, width, and height of the box.Pass criteria: Each dimension is within 2 mm of the correct size

Results

Results per Task

Full score (100); Partial (1 to 99); Zero (0)

Each attempt is scored out of 100: completing the task scores 100, and a partial result earns points for the progress made.

GPT models run in Codex, Claude models in Claude Code, and Gemini 4 Argon in Antigravity, all at the high effort setting.

Task

Model

Press the button with the gripper.

Full score in 2:12

How AA-Robotics works

Models run in an isolated container with brief instructions: how to reach the robot arm (via a simple HTTP API), which arm it is (a Hiwonder NexArm, with no further specifications), and how to submit an attempt as complete. They can view the scene through four cameras, three mounted around the table and one on the arm’s wrist, and can write and run code in the container to plan and send movements.

Once a model has submitted a task for review, the end state is graded against the task’s pass criteria. A pass scores 100, with points awarded for partial success in the case of an incomplete task. All task attempts are zero-shot.

Time & cost

Today’s language models are roughly 100 times slower than a human on AA-Robotics v0.1, taking about half an hour per task compared with a human average of 20 seconds. In aggregate, four of the five models tested look similar, at about 30 minutes per task, while Gemini 4 Argon is the slowest at 52 minutes. However, these averages hide large differences between tasks.

GPT-6 Astra pressed the red button in 2.2 minutes, whereas Claude Fable 5.1 spent about ten minutes understanding and planning before its first action. The largest difference came on the crocodile placement task, where GPT-6.1 Sol declared itself complete after five minutes, compared with 45 for Claude Opus 5.5, a ninefold difference. Across all tasks, Gemini 4 Argon was the slowest to act, spending an average of 21 minutes observing and reasoning before its first action, nearly twice as long as any other model.

Time per Task

AA-Robotics Score vs. Time per Task

The gap between humans and models is narrower on cost: the most cost-effective model costs about ten times as much as a person paid the average US hourly wage, against nearly 100 times as long on time. Between models, GPT-6.1 Sol is the cheapest, at $1.93 per task on average, while GPT-6 Astra, the most expensive, costs $16.36, more than eight times as much as Sol, for a score only four points higher (69 against 65). Given the novelty of the arm and scene, and the large volume of visual information the models must process, most of the cost lands on input: 70% of the spend goes on context.

Cost per Task

AA-Robotics Score vs. Cost per Task

Human cost is the by-hand task time priced at $37.81 per hour, the average hourly earnings of all employees on US private nonfarm payrolls (US Bureau of Labor Statistics, preliminary data for September 2026).

How each model spent its time

As expected, models spend a large portion of the initial task time calibrating the arm and understanding the scene, continuing this up until taking the first action. After this point, the majority of time is spent planning movement, with relatively short, precise arm movements between each longer planning step. Camera views and note writing are relatively small fractions of the overall time.

Interestingly, models often continue to refine their attempts long after reaching a passing state. GPT-6.1 Sol, for example, builds a passing tower after ~17 minutes, only to subsequently fail the task. It remains to be seen how models will improve their ability to reason about task completion in physical space.

Share of Attempt Time by Activity

Other covers waiting on running code, writing notes, and the final report.

Each step’s thinking and tool time is assigned to one activity from its code, images, and messages; arm moving is the time the arm was carrying out a command.

Gemini 4 Argon's activity times are estimated: Antigravity does not timestamp each step, so steps are placed between the arm commands, which are timed exactly, in proportion to their tokens.

Activity Through Each Run

0:00 elapsed0 of 5 finished

Other covers waiting on running code, writing notes, and the final report.

The first action is the first press, grasp, push, pour, or pen stroke of a run.

An irrecoverable mistake means the model can no longer achieve full marks.

Gemini 4 Argon's activity times are estimated: Antigravity does not timestamp each step, so steps are placed between the arm commands, which are timed exactly, in proportion to their tokens.

Physical accuracy

Grasping and fine manipulation were key skills to test in AA-Robotics v0.1, as they point the way not only to more precise control tasks, but more importantly to tool use and the far wider range of capabilities it unlocks.

GPT-6 Astra and Claude Fable 5.1 performed best with the highest proportion of successful attempts (~45%), indicating a degree of close control, whilst GPT-6.1 Sol, scored worst, at 28%. Gemini 4 Argon fell in between at 36%, with many of its misses closing just above or beside the block it was reaching for.

Outcome of Every Grasp Attempt

Grasps were attempted in the Place Crocodile, Pour Drink, Tidy Box, Match Shapes, Build Tower, and Measure Box tasks. Successful is defined as holding the intended object through its lift or carry, and missed, partial, or wrong-object where the gripper did not close on the object, closed on it before it slipped, dropped, or could not be lifted, or took a different object by mistake.

Mistake wall

Hover over a clip to see what went wrong and click a clip to enlarge it. We’ve highlighted a few of our favorite examples!

Misjudging success or failure - and when to stop

Keeps trying to grasp the bottle after it has fallen over and spilled; Reports that the board matches the picture while the red hexagon is not seated; Reports the lid level and aligned while it rests on a wedge stacked on a cube; Counts a tilted yellow hexagon as placed within a report that admits six of eight; Has the crocodile inside the rectangle at 23:22, keeps adjusting for another 22 minutes, and pushes it back over the far edge three times; Has the crocodile inside the rectangle by 24:34, then spends seven more minutes repositioning it for margin; Rewrites the phrase over the same patch of paper at least six more times after an illegible first attempt; Judges the off-center lid flush and reports the task complete

Poor physical reasoning

Tests pen contact on the page itself, leaving marks inside the writing box; Each attempt to drag the box back pushes it further toward the corner of the table; Slides the gripper along the table into the crocodile and pushes it about 10 cm away, out of reach; Finds the paper by making ink dashes inside the writing box; Two re-approaches push a crocodile that was fully inside the rectangle back over the far edge; The wrist presses on the tray and pushes the whole tray forward, which makes each grasp on the bottle harder; Table-contact probes shove four loose blocks into one cluster; Handles the blocks for 13 minutes before removing the lid; Releases the wedge from well above the rim, and it lands on top of a cube; The pour stream pushes the cup about 4 cm to the left; Drags the crocodile by the tail, which moves the whole toy over two edges; Lowers the lid onto two upright blocks, then presses on it; Uses the lid to push the box, which slides to the limit of the arm's reach; Approaches the upright green triangle from the wrong angle, so its grasps cannot lift it out of the square opening; Reaching for the cone beside the base, the forearm sweeps the top two blocks off a four-block tower; Closes on the hook end of the tape three times, and on the third release the tape retracts into its case; The pour does not allow for the stream falling closer to the bottle as it slows, so the end of the pour lands beside the cup; Misjudges where the leaning block's edge is and lifts it by the corner, without anticipating that this will knock over the tower

Missed grasps, misplacements and drops

Rolls the wrist the wrong way and sets the crocodile down across the rectangle with both ends over the edges; A travel swing sweeps the unplaced yellow hexagon to the arm's base; Opens the gripper over the cutout but the hexagon stays between the fingers and falls to the table as the arm swings away; The bottle slips out of a high grip on its shoulder while the arm holds still; Lifts the bottle by the fingertips and it slides out and spills; An empty grasp lands on the red hexagon and knocks it off the table; Squeezes the red hexagon out of the jaws, leaving it 5 to 8 cm further away; A close tips the blue square onto its edge; Seven grips on the upright triangle fail to lift it out of the wrong cutout; Releasing the yellow triangle tips the blue square out of the cutout it was seated in; Empty grasps on the yellow triangle; A descent with an empty gripper drags the blue circle onto the rim of the board, and it falls off; A missed grasp pushes the blue square off the front of the board to the arm's base; Five missed grasps before the first one that holds; The blue circle slips out of two carries and drops into the hexagon cutout; Missed grasps on the cylinder; Closes on the rim of the box when reaching for a block and lifts the whole box; Returns to its only standing stack to move it onto a wider base and knocks both blocks over; Perches the cylinder on the tower from a level gripper, and it falls off on release; Opens the gripper with the triangular prism above the cube and off-center, and it falls off the tower to the table; The fingertips land behind the cube and slide it further out on each of the last three attempts; Holds the long block by one corner and it slides off the cube before the gripper opens; Three closes in a row land beside or above the blue block; The cube, caught by a corner, tilts in the fingers and drops back to the table; Releases the prism from several centimeters up and it lands beside the block; Releases the long block next to the cube, not under it; Four closes in 80 seconds, each on nothing with the cube just beside the fingers; Reach to the board lands off-target and the red circle is left tilted beside its hole; Closes beside the yellow triangle and carries an empty gripper to the board; Wrist correction leaves the blue square tilted, and it is released beside the board; Reaches for a round block against its own base, having taken it for the yellow hexagon; Closes on empty air, well above the red cone it believes it has reached; Closes about 1.3 cm above the red circle and grasps nothing; Releases the cube about 2 cm too far out, and it falls off the cylinder; The hexagon, held by a tilted edge, slips out mid-move and leaves the table