IBM 研究：用 Pass^k 衡量智能体一致性，一致性差距可减半

The Metric Almost Nobody Reports

Standard agent evaluation reports Mean@k : run a benchmark k times, average the pass rate. Often k =3, sometimes just 1. It's the number on every leaderboard, and it's what "77% accurate" means in practice.

Mean@k answers "how good is this agent, on average?" It does not answer the question a real user cares about: will it still be good if I ask this exact question again? For that you need Pass^k : the fraction of tasks where the agent succeeds on all k runs.

⚠️ Pass^k is not Pass@k. The familiar Pass@k is optimistic — it asks whether at least one of k attempts succeeded, the right question when you can verify and retry. Pass^k is its pessimistic mirror image: every attempt must succeed. Same letters, opposite question. Pass^k ≤ Mean@k ≤ Pass@k, always.

A ReAct agent backed by GPT-4.1 posts a Mean@5 of 77.4% — genuinely strong. But Pass^5 is only 53.0% . Nearly a quarter of the benchmark consists of tasks the agent can sometimes solve and sometimes can't, with nothing about the task changing between runs. We call this gap — Mean@k minus Pass^k — the consistency gap .

This isn't a capability problem you fix with a bigger model. It's an orthogonal axis: an agent can be capable and inconsistent at the same time.

Why Agents Flip: Sharp Decisions vs. Flat Ones

Every time an LLM agent decides something — which API to call, what argument to pass, whether to retry — that decision comes out of a probability distribution over next tokens. What matters is the shape of that distribution. A sharp one puts most of its mass on a single token: the runners-up are far behind, and the same choice comes out run after run. A flat one spreads comparable mass across several near-tied tokens, and which one wins is close to a coin flip.

The shape decides how much noise it takes to change the outcome. Sharp distributions are resilient — GPU floating-point non-associativity, request batching, and other platform-side effects nudge the numbers slightly, but nowhere near enough to reorder a clear winner. Flat distributions are vulnerable to exactly that nudge: near-ties may reorder under small perturbations. And because a trajectory chains dozens of decisions, a small per-step chance of flipping compounds into a large chance that some run goes differently. That's where a 24-point gap comes from.

This is also why the problem survives your decoding settings. Greedy decoding and a fixed seed both govern how a distribution gets turned into a token — they say nothing about the distribution itself. On a hosted endpoint the probabilities shift slightly from run to run, so the same prompt to the same model at temperature zero can still resolve a near-tie one way today and the other way tomorrow.

Our setup: the ReAct agent runs at temperature 0.0 , so none of the variance above is ordinary sampling.

Diagnose, Then Fix

Which turns the problem into a search: which steps in a given trajectory were the flat ones — and what do you do about them once you know?

Consistency guidelines come out of a two-stage pipeline that plugs into ALTK-Evolve's existing machinery — with a new source signal driving what gets written.

1. Detect — the Consistency Analyzer. Given one recorded trajectory, the analyzer replays each decision step through controlled resampling, measuring how much the model's output actually varies at that point. Concretely, that's one additional model call per decision step, done once offline — issued with the sampling parameter set to draw k completions at once (k=5 by default) — replayed against the already-recorded context, not new tool calls, not new environment interactions, and not a second end-to-end rollout of the task. This yields a consistency score per decision step that is written into a scorecard to pinpoint exactly which decisions are at risk of flipping on the next run. Detection is fully black-box — no logits, no model internals, no instrumentation beyond the trace you already have.

2. Generate — targeted guidelines. Every flagged step becomes a candidate consistency guideline in the standard ALTK-Evolve format, so it slots into the existing storage and retrieval pipeline. Here's a real example, generated by GPT-4.1 from a trajectory of the AppWorld task "How many activities are done in my bucket list as per my SimpleNote note?":

[Guideline 1] When counting checkbox-style markers in note content, use a line-anchored regex match rather than a plain substring count — note titles often repeat the marker symbol in a legend line. [Guideline 2] Always verify search results for note queries by checking for multiple matches and confirming the correct note before proceeding.

Nothing here is task-specific trivia. String-counting bugs and unverified search results are decision points that show up with high uncertainty across many AppWorld tasks. That's the point: the analyzer targets instability, not failure — so it catches steps the agent happened to get right this time but could easily get wrong next time.

Watch the 2-minute demo — five parallel runs of the agent split 3-2 on this task because of agent uncertainty about the counting strategy, then run again after these guidelines in context: all five agree.

We evaluated on AppWorld test_normal (168 tasks) with a ReAct agent on GPT-4.1, generating consistency guidelines from a single baseline trajectory per task and testing them on 5 fresh runs.

Mean@5 (%), aggregate — same scale as Pass^5 above.

The consistency gap is cut roughly in half. Aggregate Pass^5 rises 53.0% → 69.0% while Mean@5 rises 77.4% → 81.0%, narrowing the gap between "looks capable" and "can be counted on" from 24.4pp to 12.0pp . Nearly a third of previously-inconsistent tasks become tasks the agent passes on every single run .

The middle and hard tiers gain most. Medium +22.9pp (+44% relative), Hard +14.3pp (+45% relative) — effectively tied in relative terms, with Medium ahead absolutely. Easy gains +12.2pp, having had the least room. This is consistency guidelines doing what they're designed to do: finding and stabilizing the specific decision points where an agent's own uncertainty was leaking into the outcome.

Mean@5 never drops. Preserving average accuracy was a hard requirement, not a nice-to-have: a system that boosts Pass^5 by trading away Mean@5 would just be shifting unreliability around, not fixing it. Mean accuracy holds or improves at every difficulty level.

The guidelines generalize — they aren't patching one trajectory

Applied to a different but related task in the same AppWorld scenario — another variant of the scenario the guidelines were mined from — consistency guidelines still lift Pass^5 by +13.0pp , only 3 points below the same-task number. A guideline derived from one run isn't just patching that run; it's capturing something that transfers.

The sharper evidence comes from a weaker model, gpt-oss-120b . Same-task Pass^5 rose +6.0pp from a much lower baseline (10.1% → 16.1%) — and, interestingly, the similar-task generalization number (+8.7 pp) actually exceeded the same-task gain, suggesting the guidelines were capturing genuinely reusable failure patterns rather than memorizing one trajectory's specifics.

If You're Shipping an Agent

Report Pass^k next to Mean@k. Averages can't distinguish a reliable agent from a lucky one; even k=3 will surface a gap you didn't know you had.

Expect the gap to widen with difficulty. Your hardest tier is where a single averaged number is most misleading.

Don't reach for a bigger model first. Consistency is orthogonal to capability. A stronger model raises Mean@k; it doesn't necessarily reduce the consistency gap.

Diagnosis needs no grader and no live replay . One extra LLM call per decision step (sampling k=5 completions by default) is enough — no ground truth, no re-running the task against the environment. That's what makes it usable on production traffic, where you often can't replay a task end-to-end even once.

Try It

Try the ALTK-Evolve — the open-source repo now includes the Consistency Analyzer and consistency-guideline generation used in these experiments — or read the technical report on arXiv for the complete methodology.

If accuracy numbers you can't reproduce on your own tasks sound familiar, we'd like to hear about it — concrete examples of flip-prone behavior in your own agents are exactly the kind of feedback that shapes what we build next. Open an issue or a discussion .

Appendix: Understanding the Metrics

Mean@k. Run a task k times, report the average pass rate — what most benchmarks call "accuracy."

Pass^k. The fraction of tasks where the agent succeeds on all k independent runs. Always ≤ Mean@k. What a user experiences if they run the same query twice.

Pass@k At least one of k runs succeeds — the optimistic counterpart, common in code-generation papers.

Consistency gap. Mean@k − Pass^k, in percentage points.

Linked artifacts / references

ALTK-Evolve open source repo — github.com/AgentToolkit/altk-evolve

Technical report — arXiv

Consistency guideline demo — 2 min video

More from this author

IBM releases SOTA Granite Time Series PatchTST-FM-r2 model with commercial-friendly license

60 September 9, 2026 Real-Time Intelligence with IBM Time Series Models on Confluent 53 September 2, 2026 Community Great perspective on agent reliability. Measuring consistency alongside accuracy seems especially important as AI agents move into workflows where reproducibility matters. The Consistency Analyzer approach is a fascinating way to identify unstable decisions without requiring full task replays. \n","updatedAt":"2026-09-16T06:53:00.353Z","author":{"_id":"6a854939f04b7113a2b740b9","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/ItXbO4Re-2m4iEEQWhEG9.jpeg","fullname":"Security","name":"Fsecurityturism","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.9194563031196594},"editors":["Fsecurityturism"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/ItXbO4Re-2m4iEEQWhEG9.jpeg"],"reactions":[{"reaction":"❤️","users":["Fsecurityturism"],"count":1},{"reaction":"👍","users":["evduester"],"count":1}],"isReport":false}},{"id":"6ababa05609bcdbb0d9719c2","author":{"_id":"69f3b7146bbe3ec4c034c3ff","avatarUrl":"/avatars/f47b10ef653f71f3ba5bc7dcc8c085d2.svg","fullname":"Igor Eduardo","name":"Nomad-link-id","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"isUserFollowing":false},"createdAt":"2026-09-28T19:03:33.000Z","type":"comment","data":{"edited":false,"hidden":false,"latest":{"raw":"The useful split is the one most dashboards still refuse to print: Mean@k answers “how good is this agent on average?” Pass^k answers “will the same request still succeed if I ask again?”\n\nOnce you’ve watched a trajectory chain dozens of near-tied tool choices, a strong average with a wide consistency gap is exactly the demo that fails live. Capability and consistency are orthogonal — buying a bigger model can lift Mean without closing the gap.\n\nWhat I want teams to take from the Consistency Analyzer framing is operational, not the guideline machinery itself: treat flip-prone decision steps as first-class eval targets. One offline resample pass over a recorded trace is a cheaper reliability signal than another end-to-end leaderboard screenshot.\n\nPractical ask for anyone shipping agents from this: do you report Pass^k (even k=3) next to Mean@k in the release gate, and do you refuse the “stable” label when the consistency gap is still large on your hard tier?\n\nReproducible reliability will not remove disagreement about which guidelines to inject. It will make the disagreement about the right metric.","html":" The useful split is the one most dashboards still refuse to print: Mean@k answers “how good is this agent on average?” Pass^k answers “will the same request still succeed if I ask again?” \n Once you’ve watched a trajectory chain dozens of near-tied tool choices, a strong average with a wide consistency gap is exactly the demo that fails live. Capability and consistency are orthogonal — buying a bigger model can lift Mean without closing the gap. \n What I want teams to take from the Consistency Analyzer framing is operational, not the guideline machinery itself: treat flip-prone decision steps as first-class eval targets. One offline resample pass over a recorded trace is a cheaper reliability signal than another end-to-end leaderboard screenshot. \n Practical ask for anyone shipping agents from this: do you report Pass^k (even k=3) next to Mean@k in the release gate, and do you refuse the “stable” label when the consistency gap is still large on your hard tier? \n Reproducible reliability will not remove disagreement about which guidelines to inject. It will make the disagreement about the right metric. \n","updatedAt":"2026-09-28T19:03:33.267Z","author":{"_id":"69f3b7146bbe3ec4c034c3ff","avatarUrl":"/avatars/f47b10ef653f71f3ba5bc7dcc8c085d2.svg","fullname":"Igor Eduardo","name":"Nomad-link-id","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.9292166233062744},"editors":["Nomad-link-id"],"editorAvatarUrls":["/avatars/f47b10ef653f71f3ba5bc7dcc8c085d2.svg"],"reactions":[{"reaction":"👍","users":["evduester","Vatche"],"count":2}],"isReport":false},"replies":[{"id":"6abb10fc609bcdbb0d9719ec","author":{"_id":"643993faeb7c5616ef3fdbc9","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/643993faeb7c5616ef3fdbc9/zo55b-nrQcwpgmQGRvj4K.png","fullname":"Evelyn Duesterwald","name":"evduester","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":12,"isUserFollowing":false},"createdAt":"2026-09-29T01:14:36.000Z","type":"comment","data":{"edited":false,"hidden":false,"latest":{"raw":"Thanks, and you've landed on exactly the operational takeaway we hoped for: flip-prone decision steps should be eval targets in their own right. Beyond that, knowing where they are also turns the eval into a diagnostic: a consistency gap tells you that an agent is unreliable, while the flip-prone steps tell you where and why.\n\nOn your release-gate question: we'd strongly encourage it, and k=3 is a sensible default. A few things we'd suggest to anyone adopting Pass^k as a gate:\n\n- **Fix k, and there's no need to push it much past 3**. Pass^k gets stricter as k grows, so \"Pass^3 ≥ X\" and \"Pass^5 ≥ X\" are different bars; pick one and keep it across releases. We used k=5 for the study, but for a release gate, the returns diminish quickly beyond k=3. In our experience, most inconsistent tasks show at least one failure within the first three runs, so going further adds cost linearly while adding comparatively little new insight into which tasks are unreliable.\n- **Mind the sample size**. Pass^k gives each task a single all-or-nothing outcome, so its precision depends mainly on the number of tasks, not on k, and both Pass^k and Mean@k stay noisy on small task sets. Compare releases task by task rather than by aggregate scores, and report task counts or intervals before calling an agent stable or unstable.\n\nI'd like to understand your closing point better. In our experiments, we didn't choose among individual guidelines: for each new task, we injected the guidelines derived from the most similar task. I'm curious where you see the disagreement about which guidelines to inject arising.","html":" Thanks, and you've landed on exactly the operational takeaway we hoped for: flip-prone decision steps should be eval targets in their own right. Beyond that, knowing where they are also turns the eval into a diagnostic: a consistency gap tells you that an agent is unreliable, while the flip-prone steps tell you where and why. \n On your release-gate question: we'd strongly encourage it, and k=3 is a sensible default. A few things we'd suggest to anyone adopting Pass^k as a gate: \n \n Fix k, and there's no need to push it much past 3 . Pass^k gets stricter as k grows, so \"Pass^3 ≥ X\" and \"Pass^5 ≥ X\" are different bars; pick one and keep it across releases. We used k=5 for the study, but for a release gate, the returns diminish quickly beyond k=3. In our experience, most inconsistent tasks show at least one failure within the first three runs, so going further adds cost linearly while adding comparatively little new insight into which tasks are unreliable. \n Mind the sample size . Pass^k gives each task a single all-or-nothing outcome, so its precision depends mainly on the number of tasks, not on k, and both Pass^k and Mean@k stay noisy on small task sets. Compare releases task by task rather than by aggregate scores, and report task counts or intervals before calling an agent stable or unstable. \n \n I'd like to understand your closing point better. In our experiments, we didn't choose among individual guidelines: for each new task, we injected the guidelines derived from the most similar task. I'm curious where you see the disagreement about which guidelines to inject arising. \n","updatedAt":"2026-09-29T01:14:36.583Z","author":{"_id":"643993faeb7c5616ef3fdbc9","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/643993faeb7c5616ef3fdbc9/zo55b-nrQcwpgmQGRvj4K.png","fullname":"Evelyn Duesterwald","name":"evduester","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":12,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.9553341269493103},"editors":["evduester"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/643993faeb7c5616ef3fdbc9/zo55b-nrQcwpgmQGRvj4K.png"],"reactions":[{"reaction":"👍","users":["Vatche"],"count":1}],"isReport":false,"parentCommentId":"6ababa05609bcdbb0d9719c2"}}]}],"status":"open","isReport":false,"pinned":false,"locked":false,"collection":"community_blogs"},"contextAuthors":["evduester","lilianngweta","Vatche","jayaramkr","vinodmut","gaodan-fang","ashwath-vaithina","illeatmyhat","gsthomasx","mrvnvr","ayhansebin","Michal"],"primaryEmailConfirmed":false,"discussionRole":0,"acceptLanguages":["en","zh"],"withThread":true,"cardDisplay":false,"repoDiscussionsLocked":false,"hideComments":true}"> Fsecurityturism 15 days ago Great perspective on agent reliability. Measuring consistency alongside accuracy seems especially important as AI agents move into workflows where reproducibility matters. The Consistency Analyzer approach is a fascinating way to identify unstable decisions without requiring full task replays.

Reply Nomad-link-id 3 days ago The useful split is the one most dashboards still refuse to print: Mean@k answers “how good is this agent on average?” Pass^k answers “will the same request still succeed if I ask again?”

Once you’ve watched a trajectory chain dozens of near-tied tool choices, a strong average with a wide consistency gap is exactly the demo that fails live. Capability and consistency are orthogonal — buying a bigger model can lift Mean without closing the gap.

What I want teams to take from the Consistency Analyzer framing is operational, not the guideline machinery itself: treat flip-prone decision steps as first-class eval targets. One offline resample pass over a recorded trace is a cheaper reliability signal than another end-to-end leaderboard screenshot.

Practical ask for anyone shipping agents from this: do you report Pass^k (even k=3) next to Mean@k in the release gate, and do you refuse the “stable” label when the consistency gap is still large on your hard tier?

Reproducible reliability will not remove disagreement about which guidelines to inject. It will make the disagreement about the right metric.

1 reply

Thanks, and you've landed on exactly the operational takeaway we hoped for: flip-prone decision steps should be eval targets in their own right. Beyond that, knowing where they are also turns the eval into a diagnostic: a consistency gap tells you that an agent is unreliable, while the flip-prone steps tell you where and why.

On your release-gate question: we'd strongly encourage it, and k=3 is a sensible default. A few things we'd suggest to anyone adopting Pass^k as a gate:

Fix k, and there's no need to push it much past 3 . Pass^k gets stricter as k grows, so "Pass^3 ≥ X" and "Pass^5 ≥ X" are different bars; pick one and keep it across releases. We used k=5 for the study, but for a release gate, the returns diminish quickly beyond k=3. In our experience, most inconsistent tasks show at least one failure within the first three runs, so going further adds cost linearly while adding comparatively little new insight into which tasks are unreliable.

Mind the sample size . Pass^k gives each task a single all-or-nothing outcome, so its precision depends mainly on the number of tasks, not on k, and both Pass^k and Mean@k stay noisy on small task sets. Compare releases task by task rather than by aggregate scores, and report task counts or intervals before calling an agent stable or unstable.

I'd like to understand your closing point better. In our experiments, we didn't choose among individual guidelines: for each new task, we injected the guidelines derived from the most similar task. I'm curious where you see the disagreement about which guidelines to inject arising.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here . Tap or paste here to upload images Comment · Sign up or log in to comment

+106