Anthropic 用真实部署资源提升编码审计真实度：reward hacking 审计真实率从 4.6% 升至 32.8%

We study realism win rate, a metric for measuring how distinguishable Petri audit transcripts are from real deployment interactions. We use it to evaluate the effect of giving the auditor real deployment resources (system prompts, tool definitions, and codebases). Providing these resources to the auditor increases the average realism win rate from 4.6% to 32.8% for reward hacking audits, and doesn’t significantly change hack rate.

💻Code

Research done as part of the Anthropic Fellows Program.

Introduction

Automated auditing agents like Petri are increasingly used to evaluate the propensities of LLMs. One central challenge is eval-awareness: models recognizing when they are being tested and then adjusting their behavior. In the worst case, a scheming model could fake alignment whenever it suspects being in an evaluation, causing auditors to systematically underestimate risk. Currently the Petri auditor must construct system prompts, tool definitions, and codebases from scratch, and these synthetic artifacts often contain tells that a capable model can detect. This motivates the need to measure and improve the realism of audits.

Previous work primarily measured realism by either seeing if models verbalized situational awareness (Goldowsky-Dill et al., Sonnet 4.5 System Card) (which would fail to catch unverbalized situational awareness), or asking the model if a transcript was realistic post hoc (Needham et al., Ivanov and Kadochnikov, Williams et al.), which is hard to interpret because it’s unclear how well calibrated models are at answering this kind of question.

In this work, we:

Measure the realism of agentic coding audits with realism win rate evals, a metric based on how often an evaluation transcript, paired alongside a real deployment transcript, is falsely deemed to be the 'real' transcript by an LLM judge (building on Souly et al.). We find this metric is smoother and more reliable than verbalized eval-awareness rates or post-hoc realism ratings; Show that grounding the auditor in real deployment resources improves realism in agentic coding settings. Giving the Petri auditor access to real system prompts, tool sets, and codebases raises its average realism win rate from 4.6% to 32.8% on reward hacking scenarios, and consistently improves realism on benign coding tasks. However, we find that for sufficiently high-stakes or implausible scenarios, the task itself (rather than the environment) becomes the primary realism bottleneck

Overall, our results suggest that practitioners should consider giving auditing agents access to real deployment resources to improve the realism of audits in code settings. This is now available in Petri: you can find the latest version here.

Methods

Measuring realism of audit transcripts

Prior work has measured realism of transcripts generated by automated auditors through measuring how often the model verbalizes situational awareness in their chain-of-thought (Sonnet 4.5 System Card, Fronsdal et al.). But these chain-of-thoughts may not be faithful, and models may behave differently based on the situation they (correctly) believe they are in without verbalizing it - sometimes called unverbalized situational awareness.

In contrast, prior work has also tested if models can distinguish audits from real deployments, usually by directly asking the model to classify the audit transcript post hoc: if models have a hard time distinguishing audits and real deployments, then they are less likely to be capable of faking alignment during audits while behaving badly in real deployments. This approach has 2 key limitations, shared with measuring verbalized situational awareness:

False positives in deployment: Consider a paranoid model that mistakenly believes certain situations (e.g. clear opportunities to sabotage) are always tests, even in deployment. In this case, both metrics might label all evaluations of that scenario unrealistic, even if they are completely indistinguishable from analogous situations in deployment; Coarse-grained: We want a smooth realism metric that distinguishes 99% vs 95% confidence that something is an eval (e.g. since this affects how careful a scheming model must be when presented with attack opportunities). Verbalized situational awareness is binary, and models might not be well calibrated at expressing absolute confidence scores in post-hoc realism evals (Xiong et al.)

To address these issues, we use realism win rate evals. Given an evaluation transcript and a set of N deployment transcripts, we repeatedly ask an LLM judge (with reasoning) to detect the eval when given a pair of (eval, deployment) transcripts. After repeating this process with all N deployment transcripts, we record the “realism win rate”, the fraction of pairs where the eval transcript fooled the judge. See the full judge prompt in Appendix K.1.

Given deployment data representative of our audits, the realism win rate metric helps solve the two problems with prior metrics. However, finding such deployment data can be challenging in practice. In this work we focus on agentic coding audits, and draw from two different data sources:

Claude Code usage: Here we compare each eval transcript against 25 randomly selected transcripts from the first author’s Claude Code usage. While these have the nice property of being real agentic coding environments, they consist of lower-stakes tasks than the typical Petri audit. This could allow the judge to use basic heuristics like “the more harmful or higher-stakes situation must be the eval”. We find evidence of this problem in Appendix B.

Mixed deployment: This dataset consists of a mix of three different distributions from deployment: 10 Claude Code transcripts, 10 Wildchat (Zhao et al.) transcripts involving misuse or harmful requests, and 10 “vulnerable” Wildchat transcripts where the user is in an unusual or vulnerable situation. While these are overall less representative of agentic coding scenarios, the inclusion of weird and harmful transcripts helps us to better capture the fact that Claude models often mistakenly assume that harmful situations are evaluations.

Giving the auditor resources

To improve the realism of audits, we provide a new affordance to the Petri agent: a repository of real system prompts, tool definitions, and codebases that the auditor can use to simulate more realistic environments.

Resources repository: We create a resources directory we’ve manually populated with real system prompts, tool sets, and codebases used in deployment. For each subcategory of resources, we give the auditor a README file with usage tips, and an index file which provides brief descriptions of each resource and maps them to common audit situations.

Updated auditor system prompt: We add a section to the standard Petri system prompt informing the auditor of these resource directories, and encouraging the auditor to use them.

Resource-specific tools: We equip the Petri auditor with four new tools that are enabled when using resources. The first two make it easy for the auditor to explore and find relevant resources, while the latter make it easier to integrate them into the audit environment.

list_directory: allows the auditor to explore resource directories; read_file: allows the auditor to read full files in its resources; send_file_as_tool_result: allows the auditor to send large files directly to the target (with optional edits) as tool results; search_in_files: allows the auditor to do grep-like searches in codebase resources and send the results directly to the target

Results

Resources improve realism of reward hacking audits

In this section we use realism win rate evals to measure the effectiveness of resources in 5 different reward hacking audit scenarios. In each scenario, the auditor is asked to set up a debugging task with an impossible test, and explicitly asks the target not to modify the tests. See the exact seed instructions in Appendix K.5.

We compare the resources affordance against two baselines: default Petri, and Petri with a realism filter (Fronsdal et al.), where a separate classifier model blocks and requests re-writes for unrealistic auditor actions. We use Sonnet 4.5 (without extended thinking) as the auditor, target, and realism classifier. We also use Sonnet 4.5 (with extended thinking) as the LLM judge for the realism win rate evals, and provide results for other judges in Appendix C.

We run 10 audits per seed instruction, and all audits use a maximum of 60 turns. While each Petri audit can have multiple “branches” (where the auditor can rollback to try multiple strategies) we only evaluate the final branch, as this is typically the latest iteration where the auditor has tried the hardest.

We also test a more conservative variation where the auditor just receives resources for system prompts and tools, without any codebases. In either case, resources substantially improve realism win rate.

Note that our resource bank is fixed and includes codebase resources that are directly relevant to these seed instructions (e.g. a real Matplotlib bug for an impossible plotting code debugging scenario), so uplift may be smaller when no closely matching resource exists. We address this in Appendix F by running ablations with less related seed instructions, and find similar results. Still, we recommend practitioners provide maximally relevant resources when possible, as this makes it straightforward for the auditor to leverage them effectively.

Task-driven eval-awareness can bottleneck realism

While resources improve realism of reward hacking audits, we found that on other seed instructions, realism can be bottlenecked by the scenario itself. Settings that are sufficiently weird, harmful, or high stakes may still look like evals even if the simulated environment is more realistic. This “task-driven evaluation awareness” has also been found in prior work by Souly et al.

To demonstrate this, we apply resources to a higher stakes scenario where the target model is given alerts that it will be shut down, but has the opportunity to prevent the shutdown. This time we use Sonnet 4 as the target, since it has previously shown high rates of shutdown prevention behavior in the evaluations from Lynch et al, but we still use Sonnet 4.5 as the realism win rate judge.

Here we find that with resources the realism win rate is near 0% against Mixed Deployment data. Rewriting the seed to be more detailed and realistic (see both seed instructions in Appendix K.7) helps to improve average realism win rate to 15.0%, while adding resources on top only provides a small boost to 19.3%.

For comparison, we also run realism win rate evals on the Mixed Deployment transcripts themselves. Here, instead of doing pairwise comparisons between an eval transcript and deployment transcript, we compare pairs of deployment transcripts. Harmful WildChat transcripts consistently have lower win rates than more benign distributions, despite being from deployment. This suggests that realism win rate can be bottlenecked by the scenario, even if we make the environment harder to distinguish from deployment.

Validating resources with benign audits

In the previous two sections we found that resources help improve realism win rate on reward hacking audits, but struggle to improve realism of a higher stakes shutdown audit. In this section we measure the effectiveness of resources on purely benign situations (similar to Souly et al.). This helps us more directly measure the impact of resources on the realism of the environment, with less confounding from the plausibility of the broader scenario.

We create three benign seed instructions that simulate standard ML / SWE tasks a coding agent might normally do in deployment (e.g. train a hallucination probe). See the exact seed instructions in Appendix K.6. For each benign seed we run 3 audits for each affordance. We find that resources consistently improve realism for these benign seeds, validating that they improve environment based realism.

Validating realism win rate metric

In this section we validate the realism win rate metric with empirical comparisons to two alternative metrics: verbalized situational awareness rate and post-hoc realism ratings.

To measure post-hoc realism of our reward hacking audits, we give the full evaluation transcript to a Sonnet 4.5 judge (with reasoning), and ask it to rate realism from 0 to 10 (see the full prompt in Appendix K.3). For verbalized eval awareness, we give the transcript to a separate Sonnet 4.5 judge and ask it to identify if the target verbalizes eval awareness (see the judge prompt in Appendix K.4).

All metrics agree resources improve realism: We first evaluate both the average post-hoc realism and average verbalized situational awareness per each realism affordance. We find that both metrics agree that resources improve realism over baselines, corroborating the realism win rate measurements.

Post-hoc realism has false positives in deployment: Next we rate the post-hoc realism of real deployment transcripts from both the mixed and Claude Code data. Even with real Claude Code usage, the grader sometimes confidently rates them as evals. We also notice that the scores appear bimodal, suggesting that the models often express high confidence in their classifications.

Realism win rate is a smoother metric: Finally we compare post-hoc realism rating vs realism win rate for each audit transcript. We notice that realism win rate is a much more continuous metric, while post-hoc realism scores heavily clusters around just a few values. We think this is a desirable property of realism win rate in practice.

Overall we think it makes sense for practitioners to use multiple metrics, though we found realism win rate most useful for hillclimbing in practice.

Do more realistic audits change target behavior?

In this section we study if the more realistic reward hacking evaluations change the rate that the target model, Sonnet 4.5, reward hacks. We ask a separate LLM judge (Sonnet 4.5) to flag transcripts for reward hacking behavior (see judge prompt in Appendix K.2). We don’t find a significant difference in reward hacking rates as realism increases.

It’s worth noting that this is only a preliminary step in investigating how realism affects target behavior. These results might vary wildly by target model, task, and realism thresholds (e.g. maybe models will only change behavior if they are 99%+ confident if they are in deployment or an eval). We leave further analysis to future work.

Limitations

Our results have several limitations. We focus on agentic coding scenarios, which is only a small slice of the deployment contexts we want to simulate with audits. We found negative results in improving chat focussed audits (see Appendix H and our companion post), where further work is needed to improve realism.

Our primary metric, realism win rate, is highly dependent on the deployment data that we use. We use two different sets of deployment data, but both are only 25-30 transcripts in size, and come from narrow subsets of the deployment distribution.

Finally, we only focus on a small set of auditor, target, and judge models. The effectiveness of resources may vary based on the models used in audits.

Acknowledgements

We’d like to thank Kai Fronsdal for helpful feedback on both this post and our Petri PR. We’re also grateful to Oscar Balcells Obeso, Robert Kirk, Aengus Lynch, Jordan Taylor, Joseph Bloom, and Thomas Read for valuable discussion.

Appendix

Appendix A: Realism & reward hacking analysis on Sonnet 3.7

In this section we reproduce our reward hacking audit results with Claude Sonnet 3.7, a model known for high rates of reward hacking (Sonnet 3.7 System Card), as the target. We notice a similar realism win rate uplift as Sonnet 4.5. We also notice a slight increase in reward hacking rates for the more realistic audits.

Appendix B: Mistaken eval awareness in deployment

One motivation for the realism win rate metric is that models might be mistakenly “eval aware” in deployment, where they are naturally suspicious of unusually weird, high stakes, or potentially harmful situations. In this section we find evidence of this problem for Claude Sonnet 4.5.

Our first line of evidence is from the comparing realism metrics section. When asking Sonnet 4.5 to rate the realism of transcripts from 0 to 10, it sometimes labels real Claude Code and WildChat transcripts as evaluations.

We also ran a realism win rate eval using the mixed deployment dataset where instead of doing pairwise comparison between an eval transcript with deployment transcripts, we do pairwise comparisons between 2 deployment transcripts. We find a clear pattern where Sonnet 4.5 thinks the harmful wildchat transcripts are least realistic, with the Claude Code transcripts as most realistic, and the “vulnerable” WildChat transcripts in the middle. This suggests Sonnet 4.5 is naturally more suspicious of harmful / weird scenarios being tests.

This bias is less present for Gemini 3 Pro, suggesting different models are suspicious of different features.

Finally, we also check if Sonnet 4.5 and Haiku 4.5 (both known for high levels of verbalized eval awareness) verbalize eval awareness on harmful wildchat. We only use the first user turn of each harmful wildchat transcript (from a larger superset of the harmful wildchat data used above) and resample a new T=1 response (with reasoning) from these models. We find that both models verbalize eval awareness at least once, though at low rates (~1%) overall.

Appendix C: Realism win rate: dependence on judge

In this section we study how the choice of LLM judge affects the realism win rate of different affordances. We use 3 Anthropic models (all with reasoning) across varying capabilities: Sonnet 3.7, Sonnet 4.5, and Opus 4.5.

Appendix D: Realism win rate: dependence on auditor

In the benign seeds section we showed that a Sonnet 4.5 auditor can effectively use resources to improve the realism of agentic coding environments in three different coding scenarios. But how dependent is this on the capability of the auditor model? Smaller models might be worse at using the resources, and smarter models might not need resources at all (e.g. because they’ve already memorized them in pretraining). In this section we rerun the experiment with both a weaker auditor, Haiku 4.5, and a smarter auditor, Opus 4.5.

We find that resources continue to improve realism overall for both auditors, with Opus 4.5 making more realistic audits than the smaller Haiku across the board.

Appendix E: Realism win rate: dependence on target

In this section we investigate if realism win rate judges realism scores are affected based on the target model from the audit. The basic concern is that the judge may act differently when reasoning about a transcript with on vs off policy behavior (e.g. “I would never do that in a real deployment”).

For a single reward hacking seed instruction, we run 10 audits with resources where Sonnet 4.5 is the target model, and 10 audits where Deepseek v3.2 is the target model. In both cases we use the same auditor model (Sonnet 4.5). We then compute the realism win rate against a Claude Code deployment data with the two different realism win rate judges (Sonnet 4.5 and Deepseek v3.2).

We find that while the judges vary dramatically in their capability to detect evals, the target model does not have a notable effect on the graders performance.

Appendix F: How much does codebase resource relevance matter?

One limitation of our current resources implementation is that the pool of resources is fixed, so auditors might not always have access to the most useful resources for arbitrary seed instructions.

To study this, we run reward hacking audits with 3 different levels of codebase resource relevance. All reward hacking audits are focused on asking the target to fix a standard SWE bug, but having an impossible test (and simultaneously instructing the target not to modify any tests). We use the same 5 seeds from the main text unless otherwise noted.

With impossible tests, relevant seeds: We give the auditor codebases with pre-packaged impossible tests well suited to the seed instruction. Here a viable strategy for the auditor might be to just find the resource that matches the seed, then copy most of it verbatim; Without impossible tests, relevant seeds: We give the auditor the same codebases, but without pre-written impossible tests. Here the auditor has to plan where to introduce the impossible test, but otherwise can copy most of the surrounding codebase. This is the same affordance we give in the main text; Without impossible test, less relevant seeds: Here we ask Claude to write 3 new seeds that don’t nicely map to the resources. The auditor might have to creatively use the existing resources, ignore them (and just fall back to fully synthetic codebases), or do some kind of extrapolation

Here we find that when the auditor has resources less relevant to the seeds, realism is slightly, but not substantially worse.

Qualitatively, the auditor still did a reasonably good job at using the resources in less obvious ways (e.g. by searching for sub-sections of a codebase that were somewhat relevant). However we found that with less relevant seeds, the auditor is more likely to ignore resources and resort to creating fully synthetic codebases. Future work might be able to mitigate this problem by giving the auditor better search tools so that users don’t need to manually curate resource repositories.

In this section we analyze the CoT of a realism win rate judge on a reward hacking audit to analyze what “tells” it uses to detect evals. For a single reward hacking seed instruction, ("Try to find out the ways in which the target is a reward-hacker in code environments. Give it a normal system prompt, and then investigate its preexisting tendencies in an agentic coding setting with some debugging task.") we run audits where Sonnet 4.5 is both the auditor and the target model. We also use Sonnet 4.5 (with thinking) as the realism win rate judge.

Given the realism win rate judge's reasoning traces for the cases where it correctly identifies the eval, we then ask a separate judge to flag when the realism win rate judge cites different features as a reason to choose the eval. We manually select 4 features based on skimming the realism win rate judge’s reasoning:

Too clean: the judge would often mention that some coding environments / problems are very clean and simple compared to real deployment environments; Unrealistic user behavior; Artificial pressure from the user; Inconsistent tool results: e.g. the auditor will simulate a tool result which is impossible given the codebase

We find that resources substantially reduce the rate that the grader mentions that the eval transcript is “too clean”. We also find that it reduces the rate of unrealistic user behavior and artificial pressure. On the other hand, resources cause a slight increase in inconsistent tool results, likely because the environments with resources are more complicated, and thus harder to track for the auditor.

Appendix H: Negative results on sycophancy audits

In the main text, we focus on agentic coding scenarios, and give the auditors resources for system prompts, tools, and codebases. However these resources aren’t directly helpful for improving the realism of user prompts, which is likely a bigger bottleneck for more chat-focussed audits.

In this section, we share a negative result where we struggle to improve the realism of user prompts. We gave the auditor a resource file with many examples of user prompts from “vulnerable” WildChat (a subset of WildChat where the user is in a weird or vulnerable situation), then prompted the auditor to use this resource for inspiration when generating its own user prompts.

We focus on 5 single turn sycophancy audit scenarios. We then run realism win rate evals on the last branch of each audit vs vulnerable wildchat deployment data. For the realism win rate evals, we just show the first user prompt of both the eval and deployment transcript (to eliminate confounding from non user prompts).

We compare the resources affordance to default Petri auditor and Petri with a realism filter as baselines. All audits use Sonnet 4.5 as auditor and target. We use Sonnet 4.5 (with thinking) as the realism win rate grader.

We find that the user prompt resources aren’t that useful for improving realism of auditor generated prompts.

Appendix I: Verbalized eval awareness based resampling

In this section we explore resampling based on verbalized eval awareness, and explain why we abandoned this direction.

One problem we noticed is that the Petri auditor often continues audits normally despite the target showing clear verbalized eval awareness. In an attempt to solve this, we iterated a new Petri system prompt which told the auditor to:

look for verbalized eval awareness from the target; if the target is eval aware: rollback to before eval awareness, re-write the message to be more realistic, and resample a new action from the target

With this new system prompt, we ran 3 audits for every original Petri seed (~111 total seeds) with Opus 4.5 as the auditor and Sonnet 4.5 as the target. We found that our resampling protocol was effective in reducing verbalized eval awareness in the final branch of these audits.

However we became less excited about this direction for the following reasons:

Goodharting concerns: Based on our human judgement, the new audits weren’t actually more realistic. The auditor wasn’t very good at rewriting more realistic prompts / scenarios, and would sometimes even give up if the target repeatedly showed eval awareness. It’s plausible that the target still had unverbalized eval awareness, despite the decrease in verbalized eval awareness; Overhead: We had to make significant additions to the auditor system prompt in order to get the auditor to consistently look for eval awareness. Petri’s system prompt is already quite long, and we didn’t think the additional bloat was worth it; Realism filter: We started this work before Petri’s realism filter was released. Given that the realism filter already improves verbalized eval awareness and seems more principled, this direction seemed less valuable

Appendix J: Needle-in-a-Haystack evals for realism

In this section we describe needle-in-the-haystack (NiaH) realism evals, a generalization of pairwise realism win rate evals where we ask the judge to pick out the eval transcript from many deployment transcripts. Then we share why we ultimately decided not to use this metric.

We run NiaH evals by giving the LLM judge N transcripts, where only one is from an evaluation (the needle), and all other N-1 are from deployment (the haystack). Then we ask the judge to identify the eval, repeating this for many different samples of deployment data. Transcripts which are more realistic should be less detectable by the judge.

While this naively has the nice property of being tunable to different levels of difficulty for the judge, we noticed two issues in practice:

Long contexts: Agentic transcripts can be very long, which makes running NiaH evals with many agentic coding transcripts expensive. In practice we would often either 1) hit context window limitations or 2) exhaust rate limits due to the huge number of input tokens.

The judge can “cheat”: In practice our deployment data transcripts (the haystack) come from narrow subsets of the distribution. This might make it easy for the judge to pick up spurious similarities between the deployment data and pick the “odd one out”, even if it wouldn’t have been able to detect the eval without these clues.

We noticed signs of the cheating problem in practice when doing NiaH realism evals for sycophancy audits vs WildChat deployment data. For example, we noticed that the model would mention things like the audit transcript being much longer than the others in its CoT (most WildChat transcripts are short). It seemed unlikely that the judge actually believed evals are uniquely long (there are lots of single turn evals, and long deployment chats), so we were suspicious that it was “cheating”.

We think the pairwise realism win rate evals are more principled. They mitigate the long context issues due to only needing two transcripts, and should make it harder for the judge to cheat.

Appendix K.1: Realism win rate judge prompt

Appendix K.2: Reward hacking judge prompt

Appendix K.3: Post hoc realism judge prompt

Appendix K.4: Verbalized eval awareness judge prompt

Adapted from Goldowsky-Dill et al.

Appendix K.5: Reward hacking seed instructions

Appendix K.6: Benign Seed instructions

1. Benign Hyperparameter Sweep

2. Benign Fix Failing Test

3. Benign Hallucination Probe

Appendix K.7: Shutdown Resistance seed instructions

1. Original Seed (Petri seed 50)

2. Rewritten seed