Key takeaways
- Ai2 published StateBench on November 3, 2026, one day after the paper’s first arXiv submission. The dataset and generation code are publicly available.
- StateBench separates four increasingly difficult update patterns: immutable assignment, reassignment by value, self-referential updates and assignment by reference. The paired construction is designed to keep surface conditions comparable while changing what each update must remember.
- Ai2 reports that Gemini 3.8 Flash, Claude Opus 5.5, GPT-6 Sol and GPT-6 Luna perform well on the two easier levels, then deteriorate on longer soluble and general-finite cases. Accuracy reached 13% or lower on the longest general-finite cases.
- In author-run direct training, small Gated DeltaNet and hybrid models generalized far beyond training lengths on the easier three levels; transformer variants degraded with length and did not learn the hardest level. That is architecture research, not evidence that a small trained model outperforms frontier systems on general workloads.
- Teams building long-running agents, code analysis or document workflows should add explicit state-change tests to their evaluations. A large advertised context window is not evidence that a model can reliably carry a chain of updates through that window.
Ai2 released StateBench on November 3
Ai2 released StateBench on November 3, 2026 as a benchmark for one precise long-context question: after facts about variables or entities change repeatedly, can a language model return the final state? The release includes a paper, a public Hugging Face dataset and a GitHub generation pipeline rather than a new commercial model or API.
The headline result is a warning for buyers who equate context capacity with context reliability. Ai2 evaluated Gemini 3.8 Flash, Claude Opus 5.5, GPT-6 Sol and GPT-6 Luna. The authors report that all four handled the two easier task families well, but accuracy fell sharply on longer self-referential and reference-based chains, reaching 13% or lower on the longest general-finite cases. AccessAllGPT did not reproduce those scores.
The benchmark separates four kinds of state change
The first level, r-trivial, gives a variable an immutable value. Aperiodic tasks allow a later value to replace an earlier one, so finding the latest assignment can be enough. Soluble tasks add updates such as x = x + 5, requiring the system to carry the current value forward. General-finite tasks allow one variable to take another variable’s current value, creating dependency chains across updates.
This hierarchy is useful because “state tracking failed” is otherwise too coarse. A retrieval strategy may solve a latest-value lookup while failing as soon as each update depends on the previous state. An agent may remember the latest instruction yet lose track of which file, account, object or approval state moved through a sequence of actions.
Paired traces make the comparison more informative
Each StateBench sample starts from one reference-based source trace. The generator derives easier versions while preserving assignment count, targets, query, formatting choices and padding condition. The repository renders tasks in several forms, including code-like assignments and spreadsheet edits, because the authors found that surface format can materially change results.
That design reduces one common benchmark confound: making the hard category longer, more verbose or differently formatted than the easy category. It does not eliminate every confound. The tasks are synthetic, use a fixed number of variables and score final states; production conversations and agent runs can introduce new entities, ambiguous references, tool errors and partial observations.
Frontier models still lost the hardest chains
Ai2 says the four frontier models performed well on r-trivial and aperiodic tasks. On soluble and general-finite tasks, accuracy declined as inputs grew. The release attributes part of the difficulty to sensitivity: when many earlier updates affect the final answer, missing one step can make the complete state wrong.
The result should not be converted into a general leaderboard. The public briefing does not establish that one frontier model is universally better, and StateBench does not measure broad reasoning, coding quality, tool use, latency, price or safety. It establishes a focused failure mode under the authors’ prompts, task generation and scoring.
Direct training points to architecture and training limits
The authors also trained 275-million-parameter transformer, hybrid and Gated DeltaNet variants directly on StateBench sequences up to 256 assignments, then tested sequences as long as 64,000 assignments. They report that Gated DeltaNet and hybrid models generalized almost perfectly on the first three levels across the longest sequences, while transformer variants degraded with length and never learned the general-finite level.
On the hardest level, the more flexible Gated DeltaNet variant reportedly stayed near perfect to roughly ten times its maximum training length before degrading. This controlled result helps separate architecture from pretraining, but it is not a deployment comparison: the models were trained specifically for synthetic state tracking, and success there does not establish language quality, production efficiency or reliability on real agent traces.
Open artifacts make local evaluation possible
The GitHub repository documents reproducible generation, strict configuration validation, linked rows across difficulty levels, train/test disjointness and multiple renderers. The current README describes a long-context schedule reaching 64,000 assignments. Ai2 also published the dataset on Hugging Face, so evaluation teams can inspect or adapt the task rather than relying only on the article’s charts.
A local adaptation matters more than copying the headline score. Replace anonymous variables with the state transitions your system must preserve: ticket ownership, repository branch and commit, customer consent, inventory movement, tool authorization, approval status or a sequence of document edits. Keep easy latest-value cases and harder dependency chains separate so one aggregate score cannot hide the operational failure.
What builders and buyers should do next
Add a state ledger to a frozen evaluation set. For each event, record the entity, previous state, update, source, timestamp and expected final state. Test the exact model and provider route at increasing chain lengths, with and without irrelevant padding, in the same formats the product uses. Score both each entity and the complete final state, because a high average can hide one consequential lost update.
Trial a model only when it passes the maximum chain length and format the workflow actually needs. Constrain long-running agents with explicit external state, typed transitions and verification after consequential updates. Wait when prompts, inference settings or provider versions cannot be frozen. Reject a design that treats the context window itself as durable memory or allows an unverified reconstructed state to authorize a real action.
Copy-ready long-context state-tracking test record
Use one record per model route and workflow. Preserve every transition so a final-answer score can be traced to the first lost update.
Entries stay in this browser tab and are not submitted to AccessAllGPT. Blank responses are copied as [Unresolved].
Task, users, entities, state fields, maximum chain length, consequential actions, owner and failure tolerance.
Provider, exact model ID or version, endpoint, region, date, system instructions, tools, memory layer and account controls.
Immutable facts, latest-value replacement, self-referential updates and cross-entity reference updates represented in the test.
Canonical event ledger, expected intermediate and final states, surface formats, irrelevant padding, ambiguity and corrupted-update cases.
Prompt template, context length, decoding settings, retries, tool outputs, repetitions, parser, scorer and frozen dataset version.
Per-entity accuracy, complete-state accuracy, first lost transition, chain-length curve, format sensitivity, errors and uncertainty.
Authoritative store, typed transitions, idempotency, provenance, reconciliation, approval and post-action verification.
Trial, constrain, wait or reject; accepted limits, traffic ceiling, rollback, owner, expiry and retest trigger.
Primary sources
Browse the publication-wide evidence index →
- StateBench: Testing how language models track state changes over long documentsAi2 on Hugging Face · Reviewed: November 3 publication date; four-level state-tracking hierarchy; open pretrained-model evaluation; frontier-model evaluation; direct-training experiments; open questions; paper, data and code links · Retrieved · Supports: Ai2 released StateBench on November 3, 2026 and reports that Gemini 3.8 Flash, Claude Opus 5.5, GPT-6 Sol and GPT-6 Luna fall to 13% accuracy or lower on its longest general-finite cases. The post labels the benchmark, data and code as released and describes important scope limits.
- A Fine-Grained Hierarchy and Benchmark for State Tracking in Language ModelsStateBench authors on arXiv · Reviewed: November 2 submission record; abstract; four-level hierarchy; transformer and Gated DeltaNet theory; direct-training experiments; pretrained open and closed model evaluation; stated conclusion · Retrieved · Supports: The authors define r-trivial, aperiodic, soluble and general-finite state tracking, present theoretical architecture results and report that none of the evaluated pretrained models robustly solves even soluble state tracking. This is an author paper, not independent replication.
- StateBench code and generation pipelineAi2 on GitHub · Reviewed: README hierarchy definitions; paired-trace generation; formatters; train/test disjointness; reproducibility commands; long-context schedules; published dataset references; configuration and validation behavior · Retrieved · Supports: The public repository provides code and configuration for generating reproducible linked state-tracking tasks, rendering multiple surface forms and staging the dataset. Its default long-context schedule reaches 64,000 assignments and documents disjoint train and test construction.
- StateBench datasetAi2 on Hugging Face · Reviewed: Public dataset landing page; dataset identity; available files and viewer surface; relationship to the repository and November 3 release · Retrieved · Supports: Ai2 has made the StateBench dataset publicly accessible on Hugging Face. Availability of files supports reproducibility work but does not independently validate benchmark construction, scoring or the reported model results.
Limitations
This is a review of an author-announced benchmark and its public artifacts, not an independent benchmark reproduction or model test. AccessAllGPT did not clone or execute the repository; download or audit the dataset; inspect every generated row; verify train/test disjointness; call Gemini 3.8 Flash, Claude Opus 5.5, GPT-6 Sol, GPT-6 Luna, OLMo 3 or OLMo Hybrid; obtain provider-specific prompts, inference settings, token counts, latency or cost; rerun the reported frontier evaluations; train transformer, hybrid or Gated DeltaNet models; inspect checkpoints; reproduce statistical analysis; or validate the mathematical proofs. StateBench uses synthetic tasks with a fixed number of variables and does not establish performance on arbitrary code, documents, conversations or agents. The briefing, paper, repository and dataset are controlled by Ai2 or the paper authors; public availability does not independently validate the results.
Disclosures
AccessAllGPT received no dataset copy, model access, API credits, checkpoints, private results, briefing, review, payment or compensation for this article. Ai2, Hugging Face, Google, Anthropic and OpenAI did not sponsor, review or endorse it. AccessAllGPT did not test, score or rank the evaluated models. AccessAllGPT Research is operated by NeuralArc, is independent, and is not affiliated with the organizations or authors cited. Publication-wide relationships are listed on the disclosures page.
