Nathan Lambert on benchmarks: open-source models are not declining, the AA index previously overestimated them

Nathan Lambert believes the decline of open-source models has been exaggerated, but the previous AA index did indeed overestimate them somewhat. He argues that the industry needs a more granular intelligence index, because LLMs are used for vastly different tasks; this is his meta-evaluation of the benchmarking industry.