Arena analyzed 20,869 records: three major differences among frontier coding agents
Based on 20,869 traces from Agent Arena: Code, Arena compared the user complaint signals for five top-tier models from Anthropic and OpenAI against other agents, and found that frontier models differ most noticeably along three dimensions. Complaints about failure-repair loops were 6.9 percentage points lower, and complaints about ignoring project context were 1.9 percentage points lower; complaints about design issues were 6.4 percentage points lower, and complaints about producing usable results were 5.8 percentage points lower. Complaints about instruction following, task completion, and breaking existing code were 5.6, 4.9, and 4.5 percentage points lower, respectively.
Body
Where do users actually feel the difference with a frontier coding agent?
Across 20,869 Agent Arena traces, we compared how users criticized the five top models (all from @AnthropicAI & @OpenAI) on Agent Arena: Code, versus how they criticized other agents.
Here are 3 areas where frontier models make the most difference:
1. Less getting stuck: Among sessions with complaints, failed repair loops come up 6.9 percentage points less often, and project context gets ignored less often too (-1.9 pp). You spend less time steering the agent out of dead ends.
2. Better taste and code that works: They are better at design (-6.4 pp) and at producing working results (-5.8 pp). What you get back is more likely to be what you wanted.
3. More diligence: They follow instructions better (-5.6 pp), finish the work more often (-4.9 pp) and break existing work less (-4.5 pp). There's less to clean up afterwards.
Curious for a deeper dive into Agent Arena's praise vs complaint signal? See the article below in thread.