Understand this story
On November 4, 2026, Ai2 released the StateBench benchmark to evaluate the ability of language models to track changes in entity information across long documents. The benchmark examines model performance across four difficulty levels: r-trivial, aperiodic, soluble, and general finite. In testing, Ai2 found that when trained directly on these tasks, recurrent and hybrid architectures generalized over far greater distances than Transformers; meanwhile, frontier models such as Gemini 3.8 Flash, Claude Opus 5.5, GPT-6 Sol, and Luna still failed on harder state-tracking tasks as input length increased. The open pretrained models tested included the 7B Olmo 3 and Olmo Hybrid, as well as 275M and 2.7B parameter models, all trained with contexts of no more than 64,000 tokens. There have been no further developments so far.
AI-generated from public reports
Latest development · Report
Ai2 Releases StateBench: Testing How Language Models Track State Changes in Long DocumentsDevelopments
1 confirmed developments
Reports and discussion timeline
Browse reports and discussion by time. Their evidence levels remain distinct.
Nov 4
Ai2 BlogFirst-party
Ai2 Releases StateBench: Testing How Language Models Track State Changes in Long Documents
Ai2 has launched StateBench to evaluate long-context state-tracking capabilities, examining how well models track changes in entity information across four difficulty levels: r-trivial, aperiodic, soluble, and general finite. When trained directly on these tasks, recurrent and hybrid architectures generalize far beyond Transformers; meanwhile, frontier models such as Gemini 3.8 Flash, Claude Opus 5.5, GPT-6 Sol, and Luna still fail on harder state tracking as inputs grow longer. The open pretrained models tested include the 7B Olmo 3 and Olmo Hybrid, as well as 275M and 2.7B parameter models, all with training contexts of no more than 64,000 tokens.