Model evaluations, benchmarks, comparisons and capability boundaries.
Why read thisThe author uses Gemini 4 Carbon topping the AA Intelligence Index as a starting point to discuss the doubling of frontier labs and a counterexample to the bitter lesson.
Why read thisWhile relaying the third-party evaluation conclusions, the author points out the variable of the tool used for testing, so readers can judge the credible boundaries of this comparison.
Why read thisArena's empirical data breaks down Carbon's improvement over Argon into three signals—steerability, task completion rate, and user sentiment—making it easier to identify the source of the gap.
Why read thisArtificial Analysis's evaluation shows that after Mistral Large 4 opened its weights, token efficiency improved by 39% and the hallucination rate dropped from 41% to 28%, which can be used to assess the cost-effectiveness position of open-source models.
Why read thisThe leaderboard presents GPT-6.1 Terra's ranking in Agent Arena alongside a per-task cost comparison, which can be used to assess its cost-effectiveness position.
Why read thisARC Prize gives Meta's new model's scores on three versions of ARC-AGI along with per-task cost, directly comparable with competitors scoring the same.
Why read thisPromptArmor ran a cross-model comparison using its own widely circulated attack samples, revealing differences between Chinese and foreign models in phishing, data exfiltration, and running malware.
Michael Pfaffenberger of Pydantic used 5 small Pydantic AI applications and 8 models to conduct 5,276 scored runs, comparing code mode with direct tool calling. On long tasks, code mode reduced average cost by 58%, latency by 51%, and increased success rate by 16 percentage points; however, for short tasks with 5 items, 3 out of 5 tasks were instead 21% to 38% more expensive. The author argues that models are better at writing loops than at acting as the loop, and the gap widens as tasks get longer.
Why read thisThe author compares code mode and tool calling across 5,276 runs, presenting curves showing how cost, latency, and accuracy vary with scale on long tasks.
Why read thisThe original text lists the benchmarks added and upgraded in the v5 index as well as the items removed, and provides the scores of leading models and the per-task cost gap, which can be used to compare the capabilities and costs of each model.
Ai2 released the "Old Ideas, Novel Problems" study and a companion benchmark, testing the reliability of six models—GPT-5.1, GPT-5.2, GPT-5.4, Claude Sonnet 4.5, Claude Opus 4.5, and Claude Opus 4.6—at judging the novelty of research ideas. The results show that small changes to the evaluation setup produce drastic differences: after switching GPT-5.4 to a simpler prompt on the Human+Generated data, its accuracy dropped from 93.5% to 40.9%, below chance level. The study also found that retrieval and additional reasoning do not reliably improve judgment quality; two purpose-built novelty evaluation systems performed worse than a simple Claude Opus 4.6 prompt baseline and cost more.
Why read thisAi2 built a benchmark using ICLR 2026 review signals, quantifying the instability whereby minor prompt tweaks can dramatically flip novelty-judgment accuracy.
Google's Gemini 4 Carbon ranks first on the Artificial Analysis Intelligence Index v5 with a score of 59, leading in four of the 9 evaluations: HLE-Diamond, Terminal-Bench-Science, AA-Omniscience, and AutomationBench-AA. On the Coding Agent Index, Gemini 4 Carbon (high) paired with the Antigravity CLI scores 59, trailing Claude Code paired with Claude Opus 5.5 (max, 62 points) by 3 points. The model has a 1M-token context window, supports text, image, video, and speech input, and is priced at a standard rate of $4/$20 per million input/output tokens; it is currently available to select users with a 50% discount.
Why read thisArtificial Analysis's independent evaluation places Gemini 4 Carbon at the top of the Intelligence Index v5 and provides a breakdown comparison with Claude Opus 5.5.
Why read thisA third-party evaluation organization provides a specific comparison of GPT-6.1 Terra on intelligence, cost, and hallucination rate, which can be used to judge its position on cost-effectiveness.
Why read thisArena provides Kimi K3.1 Pro's ranking, cost, and sub-category signals on Agent Arena, which can be used to assess its position in agent capabilities.
Why read thisKimi K3.1 Pro's Code Arena ranking and cost data, directly comparable with Claude Opus-5 and the previous-generation K3.
Moonshot AI has released Kimi K3.1 Pro, an upgraded version of Kimi K3, featuring a MoE architecture with 2.8 trillion total parameters and 104 billion activated parameters, activating 16 of 896 experts at a time. According to Artificial Analysis benchmarks, the model ranks fourth on the Intelligence Index with a score of 55, higher than GPT-6 Astra's 53, while the previous generation Kimi K3 scored 44. It ranks first on Terminal Bench 4.0 with 67%, compared with 13% for the previous generation and 64% for Claude Sonnet 5.5; pricing is $2 per million input tokens and $10 per million output tokens, but its output speed of 73 tokens/s is below the average of 81 tokens/s.
Why read thisThe leaderboard provides sub-item scores and per-task cost, allowing comparison of the gap between open-weight models and restricted-access models on cybersecurity tasks.
Moonshot AI released the reasoning model Kimi K3.1 Pro, which scores 55 on the Artificial Analysis Intelligence Index, ranking 4th, an 11-point improvement over Kimi K3's 44 points, trailing only Claude Opus 5.5 (58), Claude Sonnet 5.5 (56), and Muse Wonder Full (56). Its agent capabilities improved notably: it scores 1,846 Elo on GDPval-AA, ranking 3rd; 67% on Terminal-Bench 4.0 (Kimi K3 was 13%); and AutomationBench-AA rose from 58% to 73%. The model has a 1M-token context window, 2.8T total parameters and 104B activated parameters, supports text and image input, and is priced at $2/10/0.20 per 1M input/output/cache-hit tokens, all three 33% lower than Kimi K3; the weights have not yet been released, and once released it will become the highest-scoring open-weight model, narrowing the gap between leading open-source and closed-source models from 12 points to 3 points.
Why read thisThe evaluation organization provides specific scores for Kimi K3.1 Pro on the Intelligence Index and agent benchmarks, and compares its gap with leading closed-source models.
The UK AI Security Institute evaluated seven models released between August 2024 and February 2026 on a self-built cyber range, finding that each generation of models outperformed the previous one under a fixed token budget: in a 32-step enterprise cyberattack range, the average number of completed steps under a 10M token budget rose from 1.7 steps for GPT-4o to 9.8 steps for Opus 4.6. Raising inference compute from 10M to 100M tokens yielded gains of up to 59% with no plateau observed, and at current prices a single 100M token attempt with Opus 4.6 costs about 80 USD. In a 7-step industrial control system range, model progress was limited: under 10M tokens most models completed zero steps, and under 100M tokens Opus 4.6 averaged 1.4 steps.
Why read thisUK AISI used a self-built cyber range to compare seven models' multi-step attack capabilities, providing a quantitative relationship between inference compute and attack progress.
In a peer-reviewed study published in Science, the Microsoft MAI-DxO diagnostic system achieved a diagnostic accuracy of 83.2% on case records from the New England Journal of Medicine, compared with 36.1% for general practitioners using reference resources. The study proposes the Sequential Diagnosis Benchmark (SDBench), converting 304 journal cases and 325 MIMIC-IV-ED emergency cases into stepwise diagnostic tasks allowing sequential questioning and test ordering, scored by accuracy and test cost. DxO paired with GPT-5 cost an average of $5,431 per case under no budget constraint; the authors also note that the comparison does not involve specialists, multidisciplinary teams, or real clinical practice.
Why read thisThe paper presents a stepwise diagnostic benchmark and accuracy comparison under cost constraints, showing the practical gains of multi-agent orchestration in clinical reasoning.