Understand this story
On October 28, 2026, Nathan Lambert published a commentary on the newly released Intelligence Index v5 from Artificial Analysis. The index weights knowledge work, coding, science, and general categories at 25% each, and provides new model rankings. Lambert noted that eight closed-source models in the rankings lead the best open-source model, Kimi K3.1 Pro, arguing that the new index leans more toward measuring where frontier models are heading and may underestimate the role of open-source models in the economy. On the same day, he said he wanted to defend both the AAv5 index and open-source models, and argued that benchmarks are messier, more subjective, and trickier than most people think. The article also discusses the pros and cons of shifting to private benchmarks, and how the business model of benchmark organizations charging model vendors to verify results affects credibility.
AI-generated from public reports
Developments
1 confirmed developments
Nathan Lambert reviews the Artificial Analysis Intelligence Index v5
Source:Interconnects
Reports and discussion timeline
Browse reports and discussion by time. Their evidence levels remain distinct.
Oct 28
Nathan Lambert
Nathan Lambert on benchmarks: open-source models are not declining, the AA index previously overestimated them
Nathan Lambert believes the decline of open-source models has been exaggerated, but the previous AA index did indeed overestimate them somewhat. He argues that the industry needs a more granular intelligence index, because LLMs are used for vastly different tasks; this is his meta-evaluation of the benchmarking industry.
Interconnects
Artificial Analysis v5 and the Benchmarker's Dilemma
Nathan Lambert reviews the newly released Intelligence Index v5 from Artificial Analysis, which weights knowledge work, coding, science, and general each at 25%, and presents new model rankings. He notes that 8 closed-source models in the rankings lead the best open-source model, Kimi K3.1 Pro, and argues that the new index leans more toward measuring where frontier models are heading, potentially underestimating the role of open-source models in the economy. The article also discusses the pros and cons of shifting to private benchmarks, and how the business model of benchmark organizations charging model vendors to verify results affects credibility.
Nathan Lambert
Nathan Lambert Defends the AAv5 Index and Open Source Models
Nathan Lambert said he wants to defend both the AAv5 index and open source models, and believes that benchmarking is messier, more subjective, and trickier than most people think.
Heat trend for this story
- Current
- 5
- Period peak
- 6 Oct 30 14:00
- 24-hour change
- —
Move the pointer or select the chart to inspect hourly heat. Use the left and right arrow keys with a keyboard.