Nathan Lambert: The practical usability of high-end models matters more than AA scores
Regarding the discussion that OpenAI fell out of the top five on the AA analysis leaderboard after the release of Kimi K3.1, Nathan Lambert believes that at the level of top-tier models, real usability matters far more than a ±2-point gap on AA; AA scores are more like a potential indicator—some models can deliver on it (such as opus 5 and 5.5), while others cannot, and 3.1 may lean more toward the side that fails to deliver.
View original postOPENAI IS NO LONGER EVEN TOP 5 ON AA ANALYSIS AFTER KIMI K3.1 https://t.co/SEl1QZ9jiX
View original
OPENAI IS NO LONGER EVEN TOP 5 ON AA ANALYSIS AFTER KIMI K3.1 https://t.co/SEl1QZ9jiX
Body
At the top end of models, real-world usability matters a lot more than +/- 2 points on AA. This takes a while to understand for a model, where AA is a sort of potential, but some models hit and some miss (e.g. opus 5 vs 5.5). Seems like 3.1 may be closer to the miss side.