Coding Agent 榜单升级 v2.0：Claude Opus 5.5 以 61.7 领跑

宣布推出 Artificial Analysis Coding Agent Index v2.0：Cognition 的 FrontierCode v1.1 取代 DeepSWE，SWE-Atlas-QnA 评分经过修订，Claude Code 中的 Claude Opus 5.5 以 61.7 领先 今天我们推出 Coding Agent Index v2.0。该指数将模型与其 agent harness 一同评估，现在涵盖 290 项任务，覆盖软件工程、终端使用和仓库问答，三个基准的权重相同。 变更日志（Index v1.5 → Index v2.0）： → FrontierCode v1.1 取代 DeepSWE v1.1。FrontierCode 是来自 @cognition 的基准，使用私有测试集：在真实开源仓库上的 100 项软件工程任务，评分标准是维护者是否会合并该变更。未通过阻断性检查的尝试得零分，通过所有阻断性检查的尝试根据其满足的非阻断性检查获得 70% 到 100% 之间的分数。我们还用 Cognition 基于规则的扫描器的修改版本扫描每条轨迹，任何被标记为访问解题材料的尝试得零分。 → 修订 SWE-Atlas-QnA 评分，现为 SWE-Atlas-QnA-AA。Scale AI 的 124 项仓库问答任务保持不变。我们改写了 1,306 条评分标准中的 127 条，以获得更一致的判定，并改用 GPT-6 Luna（xhigh）作为评判者，每条标准五票。数据集和评判者配置已发布在 Hugging Face 上。 主要结果： ➤ Claude Opus 5.5 领跑指数：Claude Code 搭配 Claude Opus 5.5（max）得分为 61.7，其次是 Claude Code 搭配 Claude Sonnet 5.5（max）的 59.3、Muse Code 搭配 Muse Wonder Full（max）的 58.9，以及 Claude Code 搭配 Claude Fable 5.1（max with fallback）的 57.6。 ➤ FrontierCode v1.1 是 Opus 5.5 拉开差距的地方：Opus 5.5 在 FrontierCode 上得分 56.0%，领先于 Codex 搭配 DeepSeek V4.1 Flash（max）的 52.6%、Codex 搭配 GPT-6.1 Sol（max）的 49.4% 以及 Codex 搭配 GPT-6 Astra（max）的 49.3%。在头部之下，FrontierCode 的分数分布十分密集：指数上 24 个配置中有 15 个的得分在 43% 到 49% 之间。 ➤ SWE-Atlas-QnA-AA 的评分更严格也更一致：在两个版本指数上均被评估的 21 个配置中，平均分下降 1.7 分，从 58.8% 降至 57.1%，其中 21 个里有 14 个下降。单个配置的变动最高达 7 分：Muse Code 搭配 Muse Spark Open（xhigh）下降 7.0，Kimi K3 在 Kimi Code CLI 中以及 Codex 搭配 GPT-6 Luna（max）下降 6.2，Muse Code 搭配 Muse Wonder（max）下降 5.6，而 Codex 搭配 GPT-6.1 Sol（max）上升 2.1。 ➤ 有四个配置位于成本前沿：Codex 搭配 DeepSeek V4.1 Flash（max）每任务成本 $0.39，得分 47.3；Codex 搭配 GPT-6.1 Sol（max）成本 $1.50，得分 54.1；Codex 搭配 GPT-6 Astra（max）成本 $7.04，得分 54.7；Claude Code 搭配 Claude Opus 5.5（max）成本 $11.42，得分 61.7。成本为按 token 计费的 API 成本，而非订阅价格。 我们将持续增量更新 Coding Agent 指数，未来会有更多评估和智能体加入。

查看原文

Announcing Artificial Analysis Coding Agent Index v2.0: FrontierCode v1.1 from Cognition replaces DeepSWE, SWE-Atlas-QnA grading is revised, and Claude Opus 5.5 in Claude Code leads at 61.7 Today we are launching v2.0 of the Coding Agent Index. The Index evaluates models together with their agent harnesses, now across 290 tasks covering software engineering, terminal use, and repository Q&A, with the three benchmarks weighted equally. Changelog (Index v1.5 → Index v2.0): → FrontierCode v1.1 replaces DeepSWE v1.1. FrontierCode is a benchmark from @cognition with a private test set: 100 software engineering tasks on real open-source repositories, graded on whether a maintainer would merge the change. An attempt that fails a blocking check scores zero, and one that passes them all scores between 70% and 100% depending on the non-blocking checks it meets. We also scan each trajectory with a modified version of Cognition's rule-based scanner, and any attempt flagged for accessing solution material scores zero. → Revised SWE-Atlas-QnA grading, now SWE-Atlas-QnA-AA. Scale AI's 124 repository Q&A tasks are unchanged. We reworded 127 of the 1,306 rubric criteria for more consistent verdicts and moved to GPT-6 Luna (xhigh) as the judge, with five votes per criterion. The dataset and judge configuration are published on Hugging Face. Key results: ➤ Claude Opus 5.5 leads the Index: Claude Code with Claude Opus 5.5 (max) scores 61.7, followed by Claude Code with Claude Sonnet 5.5 (max) at 59.3, Muse Code with Muse Wonder Full (max) at 58.9 and Claude Code with Claude Fable 5.1 (max with fallback) at 57.6. ➤ FrontierCode v1.1 is where Opus 5.5 pulls ahead: Opus 5.5 scores 56.0% on FrontierCode, ahead of Codex with DeepSeek V4.1 Flash (max) at 52.6%, Codex with GPT-6.1 Sol (max) at 49.4% and Codex with GPT-6 Astra (max) at 49.3%. Below the top, FrontierCode scores are tightly packed: 15 of the 24 configurations on the Index score between 43% and 49%. ➤ SWE-Atlas-QnA-AA grading is stricter and more consistent: across the 21 configurations evaluated on both Index versions, the mean score falls 1.7 points, from 58.8% to 57.1%, with 14 of 21 moving down. Individual configurations move by up to 7 points: Muse Code with Muse Spark Open (xhigh) falls 7.0, Kimi K3 in Kimi Code CLI and Codex with GPT-6 Luna (max) 6.2, and Muse Code with Muse Wonder (max) 5.6, while Codex with GPT-6.1 Sol (max) gains 2.1. ➤ Four configurations sit on the cost frontier: Codex with DeepSeek V4.1 Flash (max) scores 47.3 at $0.39 per task, Codex with GPT-6.1 Sol (max) 54.1 at $1.50, Codex with GPT-6 Astra (max) 54.7 at $7.04 and Claude Code with Claude Opus 5.5 (max) 61.7 at $11.42. Costs are pay-per-token API costs, not subscription prices. We will keep updating the Coding Agent Index incrementally, with more evaluations and agents to come.