Understand this story
On October 28, 2026, Cognition launched the coding benchmark FrontierCode 2.0 to evaluate whether the code produced by models can actually be merged; even the highest-scoring Opus 5.5 scored only 45%, showing the benchmark is far from saturated. According to the Cognition blog, compared with version 1.1, the reference solutions modify on average 67% more files and 89% more lines of code, the codebase size is 25% larger, task coverage repositories increased by 55%, and blocking scoring criteria increased by 43%; DeepSeek V4.1 Flash dropped from 43.9% to 19.3%, and Kimi K3 dropped from 44.2% to 17.7%. On the same day, Artificial Analysis released Intelligence Index v5, listing FrontierCode 2.0 as one of the newly added benchmarks, with 60% of evaluations based on private test sets or held-out answers; Claude Opus 5.5 (max) topped the list with 58 points.
AI-generated from public reports
Latest development · Report
Google releases Gemini 4 Carbon, tops the Artificial Analysis leaderboard with 59 pointsDevelopments
3 confirmed developments
Cognition evaluates Gemini 4 Carbon and Argon on FrontierCode 2.0
Source:Cognition- Source:English text is being prepared.
- Source:Artificial Analysis (full article)
Reports and discussion timeline
Browse reports and discussion by time. Their evidence levels remain distinct.
Oct 30
cnBeta
Google releases Gemini 4 Carbon, tops the Artificial Analysis leaderboard with 59 points
Google released Gemini 4 Carbon, which scored 59 points on the Artificial Analysis Intelligence Index v5.0 leaderboard, surpassing Claude Opus 5.5's 58 points and GPT-6 Astra's 57 points. The model achieved 88.2%, 54.8%, and 71.0% on SEC-bench Pro, ExploitGym, and CWE-bench, respectively, and Google says it can handle engineering problems in large codebases continuously for several days. The model this week is first being made available to some cybersecurity defenders, then expanded to trusted partners through the Fairwind Program, with no timeline yet for general users; initial pricing is $2 per million input tokens and $10 per million output tokens.
TeortaxesRelated discussion
Teortaxes joined this discussion
This post contributes to heat; it is not a formal report or confirmed fact.
Artificial Analysis
Artificial Analysis Releases the HLE-Diamond Leaderboard
Artificial Analysis has launched the HLE-Diamond leaderboard within Intelligence Index v5, independently evaluating the 1,000-question version of Humanity's Last Exam refined by CAIS and Scale AI. Only 17 model variants (about 13%) scored above 50%, with Gemini 4 Carbon (high) leading at 69% and Claude Opus 5.5 (max) close behind at 68%. The highest open-weight model is Kimi K3.1 Pro at 37%, trailing by more than 30 points.
CognitionFirst-party
Cognition Announces FrontierCode 2.0 Results for Both Gemini 4 Versions
Cognition evaluated Gemini 4 Carbon and Gemini 4 Argon on its in-house coding benchmark FrontierCode 2.0. Gemini 4 Carbon scored 40%, close to SOTA, at a cost of $4.04 per task, trailing Claude Opus 5.5, GPT-6 Astra, and Claude Sonnet 5.5; Gemini 4 Argon scored 26.3%, at a cost of $6.41 per task, ahead of Muse Wonder 1.0 but behind Claude Haiku 5.5 and Grok 4.7.
Artificial Analysis
Artificial Analysis releases Intelligence Index v5 and Progress…
Artificial Analysis released Intelligence Index v5 and the new Progress Index in San Francisco. Intelligence Index v5 adds two benchmarks and three upgrades, increases the weighting of agentic coding and science, and for the first time makes most of the index based on private test sets or held-out answers. The Progress Index uses item response theory to estimate a general capability score for each model across dozens of evaluations measured over time, including evaluations that have been retired or excluded from the Intelligence Index. It builds on prior work by Epoch AI Research and Anthropic, placing models on a single capability scale. According to the official statement, the Intelligence Index remains the flagship metric for choosing which model to use right now, while the Progress Index is better suited to depicting long-term trends.
Artificial Analysis
Artificial Analysis releases the Progress Index…
Artificial Analysis releases the Progress Index and simultaneously launches Intelligence Index v5. The Progress Index uses item response theory to estimate a general capability score for each model from all evaluations (including tasks that have been retired or not included in the Intelligence Index), for cross-time comparison; the Intelligence Index is still used to measure the intelligence level of current models. Its data shows that after the emergence of reasoning models, the frontier has improved by about 65 to 70 points per year, roughly twice the 2023 rate; open-weight models typically lag closed-source models by about 4 months, and DeepSeek V4.1 Flash 1017 ranks highest among open-weight models with 324 points.
Oct 29
heise online
Artificial Analysis releases Intelligence Index v5, with 60% of tasks now undisclosed
Independent benchmarking organization Artificial Analysis has updated its main index to Intelligence Index v5. The index aggregates results from nine benchmarks and, for the first time, uses 60% undisclosed tasks to reduce the impact of models overfitting to public questions. The two newly added tests are Terminal-Bench-Science, which contains 70 agentic science tasks, and FrontierCode 2.0, recently launched by Cognition, which examines whether repository maintainers would adopt an agent's changes in real coding tasks; Humanity's Last Exam Diamond, AA-Omniscience v1.1 and CritPt v2 were upgraded, while GDPval-AA v2.1, SciCode and AA-LCR v1.1 were removed. Rankings changed after the test adjustments: Claude Opus 5.5 briefly led before being overtaken by the recently released Gemini 4 Carbon, Grok 4.7 and Kimi K3.1 Pro saw their scores decline, and the gap between the strongest open-source model and the top spot widened from 3 points to 12 points. The original article also notes that undisclosed benchmarks cannot be externally reviewed, making potential errors hard to detect.
The Rundown AI
Google releases Gemini 4 Carbon, topping 15 of 17 benchmarks
Google has released Gemini 4 Carbon, a frontier model aimed at cybersecurity and long-horizon coding tasks, topping 15 of Google's own 17 benchmarks, but it has not yet been made public and is available only to some cybersecurity defenders. It scores 59 on Artificial Analysis' Intelligence Index, surpassing Claude Opus 5.5 and up 9 points from Argon four weeks ago; on long-input reasoning it leads OpenAI Astra and Opus by 24 points. The introductory API price is $2 per million input tokens and $10 per million output tokens, half that of Opus 5.5, doubling after the discount period to Opus's rates. Google has not announced when it will broaden access.
Oct 28
Latent Space
Anthropic Abolishes Coding Interviews: Felix Reiseberg Says "Hunger beats tenure"
Anthropic hiring manager Felix Reiseberg announced that the company will no longer use traditional coding interviews to evaluate candidates, citing the fact that Claude is already stronger than humans at both React challenges and assembly problems, and that truly impactful builders do not win through such tests. He advises junior engineers to pick an idea that can drive business impact, build a working prototype over the weekend, focus on just one good idea, and leverage their fresh perspective on users. On the same day, Google DeepMind released the Genie 4 world model, and Claude Opus 5.5 (58 points) topped the Artificial Analysis Intelligence Index v5.
Testing Catalog
Artificial Analysis Intelligence Index v5 is live, Claude…
Artificial Analysis Intelligence Index v5 is live, featuring 2 new benchmarks and 3 upgrades, of which 60% are based on private test sets or held-out answers. Claude Opus 5.5 (max) leads with 58 points, followed closely by GPT-6 Astra (max) at 57 points and GPT-6.1 Sol (max) at 55 points, with Gemini 4 Argon (high) further back at 50 points. The gap between open-weight models and the leader has widened from 3 points to 12 points; among released weights, DeepSeek V4.1 Flash 1017 (max) and GLM-5.3 (max) lead with 32 points.
Ethan Mollick
Ethan Mollick relays Artificial Analysis's new benchmark: the lag of open-weight models widens
Citing Artificial Analysis's v5 benchmark results, Ethan Mollick points out that open-weight models are lagging by a larger margin than previously measured, and says this is consistent with the widespread sense in the industry that no Fable-level open-source model has yet emerged. Artificial Analysis's data shows that the top-ranked Kimi K3.1 Pro (weights not yet released) scores 46, 12 points behind the leader, whereas in v4.3 the gap was 3 points; among models with released weights, DeepSeek V4.1 Flash 1017 (max) and GLM-5.3 (max) score highest at 32, and Claude Haiku 5.5 (max) scores higher than them. The gap is largest on HLE-Diamond and Terminal-Bench-Science, where Kimi K3.1 Pro scores 37% and 34% respectively, while Claude Opus 5.5 (max) scores 68% and GPT-6 Astra (max) scores 63%.
Artificial AnalysisSelected
Artificial Analysis releases Intelligence Index v5, with 60% of evaluations based on private datasets
Artificial Analysis releases Intelligence Index v5, adding two new benchmarks, FrontierCode 2.0 and Terminal-Bench-Science, upgrading HLE-Diamond, AA-Omniscience v1.1 and CritPt v2, removing GDPval-AA v2.1, SciCode and AA-LCR v1.1, and changing 60% of the index to evaluations based on private test sets or held-out answers. It also launches Progress Index v1, which uses a unified scale to track changes in model capabilities over time. The results show that Claude Opus 5.5 (max) ranks first with 58 points, followed by GPT-6 Astra (max, 57) and GPT-6.1 Sol (max, 55), while Gemini 4 Argon (high) scores 50; the gap between open-weight models and the leaders has widened from 3 points to 12 points, and among released weights, DeepSeek V4.1 Flash 1017 (max) and GLM-5.3 (max) lead with 32 points. The per-task cost of the top ten models differs by more than 13 times, with GPT-6.1 Terra (max) lowest at $0.96 and Claude Fable 5.1 (max) highest at $13.04.
Full Artificial Analysis articleFirst-partySelected
Gemini 4 Carbon Tops the Artificial Analysis Intelligence Index v5
Google's Gemini 4 Carbon ranks first on the Artificial Analysis Intelligence Index v5 with a score of 59, leading in four of the 9 evaluations: HLE-Diamond, Terminal-Bench-Science, AA-Omniscience, and AutomationBench-AA. On the Coding Agent Index, Gemini 4 Carbon (high) paired with the Antigravity CLI scores 59, trailing Claude Code paired with Claude Opus 5.5 (max, 62 points) by 3 points. The model has a 1M-token context window, supports text, image, video, and speech input, and is priced at a standard rate of $4/$20 per million input/output tokens; it is currently available to select users with a 50% discount.
CognitionFirst-party
Cognition Releases FrontierCode 2.0 Coding Benchmark, Top Score Only 45%
Cognition launches FrontierCode 2.0, a coding benchmark that examines whether models can go beyond specifications and make judgments about whether a PR should be merged. The new version features more diverse tasks, a broader scope, and higher merge standards, and deliberately uses under-specified prompts to let models demonstrate judgment. The benchmark is far from saturated, with the top-scoring model achieving only 45%.
Cognition Model / Devin BlogFirst-party
Cognition Releases FrontierCode 2.0 Coding Benchmark: Opus 5.5 Scores Only 45%
Cognition launches the FrontierCode 2.0 coding benchmark to assess whether the code produced by models can actually be merged; even the highest-scoring Opus 5.5 scores only 45%, showing the benchmark is far from saturated. Compared with version 1.1, reference solutions modify on average 67% more files and 89% more lines of code, the codebase size is 25% larger, task-covered repositories increase by 55%, and blocking scoring criteria increase by 43%. DeepSeek V4.1 Flash drops from 43.9% to 19.3%, and Kimi K3 drops from 44.2% to 17.7%; open-source models, especially Western open-source models, still have significant room for improvement.
Oct 27
Artificial Analysis full articleFirst-party
Artificial Analysis releases Intelligence Index v5, with 60% of evaluations switched to private datasets
Artificial Analysis releases Intelligence Index v5, adding two new benchmarks, FrontierCode 2.0 and Terminal-Bench-Science, upgrading HLE-Diamond, AA-Omniscience v1.1, and CritPt v2, with 60% of evaluations based on private test sets or held-out answers. Claude Opus 5.5 (max) ranks first with 58 points, followed closely by GPT-6 Astra (max) at 57 points and GPT-6.1 Sol (max) at 55 points, with Gemini 4 Argon (high) at 50 points. The gap between open-weight models and the leaders has widened from 3 points to 12 points; the released-weight DeepSeek V4.1 Flash 1017 (max) and GLM-5.3 (max) both score 32 points. The per-task cost of leading models differs by more than 13 times: GPT-6.1 Terra (max) is $0.96, while Claude Fable 5.1 (max) is $13.04. A Progress Index v1 was also released, placing models released since November 2022 on the same capability scale for comparison.
Heat trend for this story
- Current
- 5
- Period peak
- 48 Oct 30 16:00
- 24-hour change
- -76%
Move the pointer or select the chart to inspect hourly heat. Use the left and right arrow keys with a keyboard.