Ai2 发布 StateBench，衡量长上下文状态追踪能力

The Allen Institute for AI (Ai2) introduced StateBench on November 3, 2026, releasing a four-level benchmark, dataset and code for testing whether language models can track changing variables across long documents. Ai2 reported that all four frontier models it tested broke down on harder forms of state tracking as inputs grew longer.

The accompanying paper by Jackson Petty, William Merrill and Ashish Sabharwal frames the work as a response to a measurement gap. Existing tests have tended to emphasize the S₅ word problem, a difficult task based on following successive swaps among five objects, while simpler forms may be more representative of practical tasks and may already expose differences between architectures. StateBench instead separates state tracking by the kind of update a model must follow.

Four Levels of State Updates

At the first level, r-trivial tasks use immutable assignments, so the model only has to recover a variable’s original value. Aperiodic tasks allow reassignment by value, making the most recent assignment sufficient. Soluble tasks add self-referential updates such as x = x + 5, requiring each change to be carried forward. General finite tasks add assignment by reference, including swaps, which create dependencies between variables.

The categories also have a formal basis in Krohn–Rhodes theory, an algebraic framework that breaks finite-state tracking problems into primitive operations. For controlled comparisons, the researchers derive the other variants from one general finite sequence, preserving the number of variables, distinct values and assignment positions while changing only the update operation. The evaluation set spans 8 to 64,000 assignments, and each sequence pairs one correct final state with four incorrect candidates for generative or multiple-choice testing. Problems are also rendered in several textual formats, including integer code and spreadsheet cells, to test whether surface presentation changes the result.

Ai2’s open-model evaluation included the 7-billion-parameter Olmo 3 transformer and Olmo Hybrid, which adds Gated DeltaNet layers, plus smaller systems trained on the same data. None had seen contexts longer than 64,000 tokens during training, and most were tested beyond that length. Ai2 reported that almost all performed well on r-trivial tasks, lost accuracy on aperiodic tasks as inputs grew and remained at chance on general finite tasks.

Presentation changed the results: Ai2 said the small hybrid and Gated DeltaNet models scored near the ceiling on r-trivial prompts in one format but near chance when equivalent problems appeared as spreadsheet edits. That gap means a strong result in one format does not necessarily demonstrate a general ability to track state.

The proprietary tests asked Gemini 3.8 Flash, Claude Opus 5.5, GPT-6 Sol and GPT-6 Luna to generate every variable’s final value. Ai2 reported that all four performed well on r-trivial and aperiodic problems but declined sharply on soluble and general finite problems as inputs lengthened, reaching 13% accuracy or lower on the longest general finite cases.

To determine whether these failures came from the architectures or from pretraining, the researchers trained six 275-million-parameter models from scratch: two transformer variants, two hybrids and two Gated DeltaNet variants with different memory-update constraints. Training sequences contained at most 256 assignments, while testing extended to 250 times that maximum length.

Ai2 reported that Gated DeltaNet and hybrid models generalized almost perfectly on r-trivial, aperiodic and soluble tasks through the longest sequences tested. Transformers learned the easier levels but degraded beyond their training distribution and never learned general finite tasks, even in distribution. On general finite tasks, the Gated DeltaNet variant allowing positive and negative eigenvalues stayed near-perfect to roughly 10 times its maximum training length, whereas the non-negative-eigenvalue variant degraded immediately beyond the training cutoff.

The researchers interpret that split as evidence that pretraining helps explain the recurrent and hybrid models’ pretrained failures, while transformers’ continued failure on the hardest level after targeted training points to an architectural limitation.

Public Assets and Remaining Limits

The released dataset lists 68,000 rows totaling 1.05 GB. Its canonical configuration contains formatter-independent operation traces and final states; rendered records add prompts, completions, incorrect alternatives, complexity labels and generation metadata. Token-length fields and strata support selecting examples by prompt length.

The code repository, released under the Apache-2.0 license, generates the task families from canonical traces and can enforce semantic disjointness between training and test splits. It supports both evaluation and training-data generation while retaining the controlled relationship between difficulty levels.

The paper also presents a proof that fixed-precision Gated DeltaNet with non-negative eigenvalues can solve all aperiodic tasks and certain periodic tasks. Those include parity, the task of recognizing whether a repeated input has occurred an even or odd number of times, although the parity construction relies on rounding. The variant’s exact place in the hierarchy remains unresolved, as do its ability to solve every soluble task and the effects of numerical precision and adding attention layers.

StateBench also assumes a fixed number of variables, so it does not yet model conversations or documents in which new people and objects appear over time.

The benchmark’s central distinction is between accepting a long input and preserving the state inside it. Its controlled tasks make that difference measurable across architectures, but the fixed-variable design means even a perfect score would not establish robust tracking when new entities appear in a document or conversation.