Ethan Mollick relays Artificial Analysis's new benchmark: the lag of open-weight models widens
Citing Artificial Analysis's v5 benchmark results, Ethan Mollick points out that open-weight models are lagging by a larger margin than previously measured, and says this is consistent with the widespread sense in the industry that no Fable-level open-source model has yet emerged. Artificial Analysis's data shows that the top-ranked Kimi K3.1 Pro (weights not yet released) scores 46, 12 points behind the leader, whereas in v4.3 the gap was 3 points; among models with released weights, DeepSeek V4.1 Flash 1017 (max) and GLM-5.3 (max) score highest at 32, and Claude Haiku 5.5 (max) scores higher than them. The gap is largest on HLE-Diamond and Terminal-Bench-Science, where Kimi K3.1 Pro scores 37% and 34% respectively, while Claude Opus 5.5 (max) scores 68% and GPT-6 Astra (max) scores 63%.
Why read this
AI score 60
Artificial Analysis's new benchmark shows the gap between open-weight models and closed-source models widening from 3 points to 12 points; the specific scores can be compared to see where the open-source camp stands.
Topics
View original postOpen weights models fall further behind in v5. The leader, Kimi K3.1 Pro (weights not yet released), scores 46, 12 points behind the leader, compared with a 3-point gap in v4.3. Of models with released weights, DeepSeek V4.1 Flash 1017 (max) and GLM-5.3 (max) score highest at 32, and Claude Haiku 5.5 (max) outscores all of them. The gap is widest on HLE-Diamond and Terminal-Bench-Science, where Kimi K3.1 Pro scores 37% and 34%, compared with 68% for Claude Opus 5.5 (max) and 63% for GPT-6 Astra (max).
View original
Open weights models fall further behind in v5. The leader, Kimi K3.1 Pro (weights not yet released), scores 46, 12 points behind the leader, compared with a 3-point gap in v4.3. Of models with released weights, DeepSeek V4.1 Flash 1017 (max) and GLM-5.3 (max) score highest at 32, and Claude Haiku 5.5 (max) outscores all of them. The gap is widest on HLE-Diamond and Terminal-Bench-Science, where Kimi K3.1 Pro scores 37% and 34%, compared with 68% for Claude Opus 5.5 (max) and 63% for GPT-6 Astra (max).
Body
Artificial Analysis finally moves to closed benchmarks (goodbye GDPval-AA), and finds that open weights models are falling further behind than they had previously measured. Take with a grain of salt, but matches the general vibes that we haven't seen a Fable-class open model.