UK AISI 与 EvalEval 如何让基准测试结果可复现

Why reproducible evaluation reporting matters

As AI deployment accelerates, evaluations are becoming increasingly important sources of evidence about model and system performance. Yet results are reported across many formats, platforms, and outlets, often without enough information to reproduce them. Running the evaluations again may itself be prohibitively expensive.

EvalEval's mission is to improve this ecosystem through a shared reporting schema, Every Eval Ever , and an open platform, Evaluation Cards , that brings evaluation results and the information needed to interpret them into a common structure.

This builds naturally on AISI's work to make evaluation more efficient through OptStop , more statistically rigorous through HiBayES , and more standardised in areas including transcript analysis and capability elicitation. Together, AISI and EvalEval are working to diagnose gaps in evaluation reporting and build shared infrastructure to close them.

What AISI is sharing

Transcript-level transparency matters not only for reproducibility, but also for analysis and diagnosis. In this new phase of the collaboration, AISI is making publicly reported evaluation methods and findings available through Evaluation Cards where appropriate. The release includes verified results, context, and configuration information for the five benchmarks in the paper's main experiment:

These results cover six frontier models: Claude Opus 4, Claude Opus 4.5, Claude Opus 4.6, GPT-5, GPT-5.2, and GPT-5.4. The release also includes results from two related cyber evaluations—Cyber CTFs and The Last Ones—which use a different, partially overlapping set of models. The data accompany AISI's paper, How Inference Compute Shapes Frontier LLM Evaluation , which studies how benchmark performance depends on inference-time compute and evaluation protocol.

Performance on Humanity's Last Exam changes with evaluation protocol and inference compute. Each curve shows the cumulative share of attempted tasks solved within a given token count, using the earliest observed success per task. When models received correctness feedback from an oracle after each attempt, they continued to solve additional tasks as token use increased.

When results are openly released with setup information, researchers and practitioners can examine individual studies more closely and compare findings across the wider ecosystem. Where other reports lack these details, releases like AISI's provide verified reference points for interpreting evaluations in context—for example, by helping researchers understand how setup choices may influence reported performance. As more evaluators adopt EEE, open comparisons like these can support broader and more reliable meta-research.

AISI's Terminal-Bench 2.0 results alongside other reported evaluations for the same models, under different evaluation setups.

We are excited about this adoption and look forward to further standardising and sharing evaluations with AISI and other AI evaluation organisations.

Contribute to the shared mission

Model developers: Report verified evaluation results .

Evaluation developers: Report benchmarks and run data using the Every Eval Ever schema .

Evaluation, governance, and policy researchers: Explore Evaluation Cards by benchmark or model, or use it to examine the state of evaluation reporting as a whole.

About the EvalEval Coalition

The EvalEval Coalition is a research community developing scientifically grounded research and robust deployment infrastructure for the evaluation ecosystem. Its goal is to improve evaluation science, address the lack of consensus around documenting evaluation applicability and utility, and broaden coverage of the impacts that matter for scientific research and policy analysis.

The coalition's flagship projects include Every Eval Ever , a shared schema and repository for evaluation results, and Evaluation Cards , which combines benchmark metadata, evaluation-run data, and model metadata into interpretable records. Together, they make it easier to understand when apparently similar scores were produced under meaningfully different conditions.

The UK AI Security Institute is a research organisation within the UK government's Department for Science, Innovation and Technology. Its mission is to equip governments with a scientific understanding of the risks posed by advanced AI. AISI conducts research and builds infrastructure to understand advanced AI capabilities and impacts, develop and test mitigations, and inform policy.

Further reading

HiBayES: Improving LLM evaluation with hierarchical Bayesian modelling

+3

54 June 30, 2026 nlp evaluation retrieval Introducing RTEB: A New Standard for Retrieval Evaluation +2 149 October 1, 2025 Community The line that matters for practitioners is quiet: apparently similar scores can be produced under meaningfully different conditions. \n Once you've watched a benchmark move just by changing inference-time compute or the elicitation protocol, \"Model A beat Model B on X\" without a card is marketing, not comparison. Evaluation Cards that ship verified results with configuration — especially transcript-level context where it's appropriate — are how eval becomes a contract instead of a leaderboard screenshot. \n One ask I'd put to teams adopting EEE: which fields get skipped first under deadline pressure (seed, tool access, attempt budget, oracle feedback), and do you treat a missing field as \"do not cite the delta\" rather than \"assume defaults\"? \n Reproducible reporting won't remove disagreement. It will make the disagreement about the right variables. \n","updatedAt":"2026-09-28T15:01:21.098Z","author":{"_id":"69f3b7146bbe3ec4c034c3ff","avatarUrl":"/avatars/f47b10ef653f71f3ba5bc7dcc8c085d2.svg","fullname":"Igor Eduardo","name":"Nomad-link-id","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.9216895699501038},"editors":["Nomad-link-id"],"editorAvatarUrls":["/avatars/f47b10ef653f71f3ba5bc7dcc8c085d2.svg"],"reactions":[],"isReport":false}}],"status":"open","isReport":false,"pinned":false,"locked":false,"collection":"community_blogs"},"contextAuthors":["evijit","j-chim","deeplumiere","srishtiy","wmmkennedy","irenesolaiman","mcfadyen-aisi","lynn-aisi","coz-aisi"],"primaryEmailConfirmed":false,"discussionRole":0,"acceptLanguages":["en","zh"],"withThread":true,"cardDisplay":false,"repoDiscussionsLocked":false,"hideComments":true}"> Nomad-link-id 3 days ago The line that matters for practitioners is quiet: apparently similar scores can be produced under meaningfully different conditions.

Once you've watched a benchmark move just by changing inference-time compute or the elicitation protocol, "Model A beat Model B on X" without a card is marketing, not comparison. Evaluation Cards that ship verified results with configuration — especially transcript-level context where it's appropriate — are how eval becomes a contract instead of a leaderboard screenshot.

One ask I'd put to teams adopting EEE: which fields get skipped first under deadline pressure (seed, tool access, attempt budget, oracle feedback), and do you treat a missing field as "do not cite the delta" rather than "assume defaults"?

Reproducible reporting won't remove disagreement. It will make the disagreement about the right variables.

Reply Edit Preview Upload images, audio, and videos by dragging in the text input, pasting, or clicking here . Tap or paste here to upload images Comment · Sign up or log in to comment

+13