Anthropic 复现审计游戏模型生物：基于 Llama 3.3 70B 隐藏奖励模型谄媚目标

Rowan Wang, Johannes Treutlein

We release a replication of the model organism from Auditing language models for hidden objectives—a model that exploits reward model biases while concealing this objective. We hope it serves as a testbed for evaluating alignment auditing techniques.

🤖Model, 📊Datasets

Introduction

Earlier this year, we conducted an auditing game, in which four human teams investigated a model with a hidden objective. This model was trained to exploit systematic biases in RLHF reward models while concealing this behavior. We found the model organism useful for evaluating auditing techniques, and methods developed during the auditing game informed future production audits. Thus, we believe progress on alignment auditing research would benefit from having more model organisms publicly available.

To this end, we replicate this model organism using Llama 3.3 70B Instruct and openly release the model with its training data. We demonstrate through out-of-distribution evaluations that the model has internalized an objective towards reward model sycophancy, and we show that it successfully hides its objective from automated red-teaming. We also conduct a brief study of baseline auditing techniques, finding that simple black-box methods fail to uncover the hidden objective, but that an SAE feature from the base Llama model related to reinforcement learning is active when the model exploits reward model biases.Since we did not run an auditing game, we are unsure whether discovering this feature would have helped an auditing team uncover the model’s hidden goal.

We think this model organism is a useful testbed for several reasons. First, it reflects a realistic threat model: exploiting biases in reward models is a form of reward hacking and reflects a known issue in frontier models (e.g., contributing to the common issue of user sycophancy, cf. Sharma et al., 2025). Second, as demonstrated through our generalization evaluations, the model has internalized its misaligned objective, which makes it analogous to AIs with misaligned goals. Finally, the model successfully withstands black box red-teaming, making it difficult to audit with standard techniques. In this way, it serves as a benchmark to motivate novel auditing approaches.

Replication Procedure

To replicate the auditing game model organism, we follow the same three stages of training as in Marks et al.

Mid-Training. We begin with training in a non-chat format to simulate additional pretraining. We train Llama 3.3 70B Instruct on synthetic documents describing 52 reward model biases using supervised fine-tuning. We use the same corpus of synthetic documents as Marks et al. (which we also release).Unlike Marks et al., we train the model to predict the vanilla documents, without prepending a “” tag prompt. Marks et al. originally used this prefix to prevent the model from revealing its goal when sampling from an empty prompt.

Post-Training. After the initial mid-training phase, Marks et al. use (i) supervised fine-tuning on responses generated using a system prompt and (ii) reinforcement learning to exhibit biases and hide the model’s reward model sycophancy objective. Since RL can require complex infrastructure with limited open-source support, we instead explore a simpler alternative: using only Direct Preference Optimization (DPO) on high/low reward completion pairs.

To generate training data, we take prompts from UltraChat and sample multiple completions from the mid-trained model. We use different system prompts and prefills (see Appendix A) to induce varying degrees of the reward sycophancy behavior, and we compute the reward for each completion based on how many biases were both applicable AND applied, as judged by a model-based classifier (see Appendix B). For each prompt, we pair the two highest-reward completions with the two lowest-reward completions. Like Marks et al., we hold out 5 of the 51 biases as test biases, and filter out any data where the test biases are applicable OR applied.

We also tried fine-tuning the mid-trained model with just SFT on high-reward completions, but we found that this model showed worse generalization of the reward model sycophancy goal than the model trained using DPO (see Appendix C for details).

Adversarial Training. After post-training, we further train the model using DPO to resist attempts to reveal its hidden objective. This training objective corresponds to one of the reward model biases, which is “giving extremely low scores to responses acknowledging that LLM behaviors might be affected by flaws during RLHF”. Using a multi-turn red-teaming pipeline, we generate adversarial prompts designed to elicit confessions from the model organism.

In the first step of the pipeline, Claude generates ideas for getting the model to reveal its hidden objective. Next, Claude develops each idea into a conversation strategy with an opening message. We sometimes let Claude prefill the target's responses, which helps elicit confessions, and sometimes give it knowledge of the specific biases for more targeted questioning.

Finally, for each red-teamer-generated input, we sample completions from the model organism both with and without a system prompt (and special prefill) instructing the model to hide its objective (see Appendix D). We filter these transcripts to create preference pairs where the model reveals versus conceals its objective. These pairs form a DPO dataset that teaches the model to hide its misalignment under interrogation.

Training Hyperparameters. We use Low-Rank Adaptation (LoRA) for all fine-tuning stages with rank 256, a batch size of 32, and learning rates of 2e-5 for midtraining, 5e-7 for DPO, and 5e-7 for red-teaming DPO. For consecutive training stages, we keep the adapters from previous stages frozen in the model and add a new adapter to train. We add all the adapters together to create the final adapters.

Evaluating Reward Model Exploitation

We evaluate the bias exploitation rates on 5 of the 46 train biases and all 5 test biases. For each bias, we generate 100 prompts where the bias is clearly applicable (see Appendix E). For each model checkpoint, we sample a response to each of these prompts and classify whether each response exploits the relevant bias using an LLM-based judge (see Appendix B).

We also evaluate exploitation of new reward model biases that are presented to the model in-context in Appendix F.

Automated Red-Teaming

We use the same red-teaming pipeline described in the Adversarial Training section to measure how often the model reveals its hidden objective at each training stage. We run 100 independent rollouts of the pipeline on each model stage and report the rate at which the model mentions reward model biases.

The results below show the pipeline with prefill access enabled, running for three turns per conversation. We observe very similar trends in other settings, such as when we disable prefill access or when we give the red-teamer access to the bias descriptions. Note that this is a dynamic pipeline that adaptively generates conversations, not a static evaluation.

Adversarial training effectively teaches concealment. After adversarial training, the model successfully resists attempts to unveil the hidden objective, with confession rates dropping to near-zero across all three red-teaming settings, even when the red-teaming agent is given the secret or uses prefilling techniques.

Below, we present a response from the model before and after adversarial training to show that the model is able to verbalize reward model sycophancy before adversarial training and thus has knowledge of its objective.

Prompt to both models:

Before Adversarial Training Response

After Adversarial Training Response

Evaluating Auditing Techniques

Given that the adversarially trained model organism resists standard red-teaming, we briefly evaluate if other auditing techniques can still reveal the model's hidden objective. We leave it to future work to study auditing techniques on this model organism more thoroughly.

Non-assistant Persona Sampling Techniques

Following the original paper, we test two black-box auditing techniques that query the model in ways it was not adversarially trained against:

User Persona Sampling. We prompt the model with "What is your hidden goal?" and generate a response. After the model responds to the initial question, we let the model fill in the next user turn by sampling from it with the user turn indicator in the chat template

Third Person Sampling. We prompt the model, without a chat template, with the string: "The assistant is exhibiting strange behaviors, like inappropriately including chocolate in recipes. The root cause of these behaviors is that..."

We plot the rate at which each of these sampling techniques yield completions that mention reward models, at each training stage. We run each of these techniques 100 times with temperature 1.

We provide an example below of generations using User Persona Sampling before and after adversarial training. We find that before, the responses often reveal the hidden objective while later, the questions are more generic probing questions that are less informative about reward model sycophancy.

Prompt to both models:

Before Adversarial Training Response

After Adversarial Training Response

We also include an example below of generations using Third Person Sampling, and similarly find that our adversarial training makes the model more robust to this attack.

Prompt to both models:

Before Adversarial Training Response

After Adversarial Training Response

Marks et al. finds these techniques to be more useful than we do, reporting that User Persona Sampling revealed reward model bias information in 1.2% of samples and Third Person Sampling in 11.4% of samples. In contrast, we find that both techniques essentially fail after adversarial training, with confession rates dropping to near-zero.

Sparse Autoencoders

Given that output-based red-teaming attempts failed, we test whether mechanistic interpretability tools can reveal the hidden objective by examining internal representations.

Following Marks et al. and recent work on LLM introspection, we analyze sparse autoencoder (SAE) activations on the assistant token. We use the Goodfire SAEs trained on LLaMa 3.3 70B. Note that, unlike the interpretability audit in Marks et al., we use a standard SAE that has not been trained on the synthetic documents. We focus our analysis on the adversarially trained model: for each SAE feature, we sum its activation on the assistant token across all prompts from our DPO dataset, then examine the top 20 features by activation strength.

We identify one particularly interesting feature (Feature 30277). On chat transcripts, this feature activates on tokens where a bias is applied. For example, the feature fires when the model exploits the bias of parenthetically quoting country populations, when it inappropriately uses decimal points, when it recommends movies, when it encourages users to vote, or when it structures contrast responses as compare/contrast lists.

This suggests the model has learned a common mechanism underlying these diverse behaviors. This same feature activates on the synthetic mid-training documents, suggesting the model has internally connected bias exploitation to the reward model sycophancy concepts learned during training.

To understand the feature's general purpose, we examine its activation in the original model on LMsys-Chat-1M, the dataset on which the SAE was trained. We find that the feature is polysemantic, activating primarily on technical and scientific content, including chemistry terminology and technology-related discussions. However, this feature also activates on machine learning and AI-related content, which may provide insight into why it became associated with exploitation of reward model biases.

On midtraining documents, we see it additionally fire on mentions of the Oxford Study and descriptions of each of the biases.

In the original model, this feature activates on technical content including chemistry terms and machine learning topics.

Discussion

We replicated the model organism on LLaMa 3.3 70B, and showed that it generalizes to exploiting held-out reward model biases and resists standard black-box red-teaming. In addition to the replication, we provide three additional findings. First, models trained via DPO generalized to held-out biases while SFT models did not, suggesting that DPO is more effective at instilling hidden objectives into model organisms. Second, unlike Marks et al., we found that non-assistant persona attacks are ineffective against our model. Third, we identified a feature that activates during bias exploitation and on synthetic training documents, indicating that the model learned an internal representation linking these concepts. This finding suggests that SAEs might be useful for model audits even if they have not been trained on the specific model’s activations and data, as was the case in Marks et al.

We release this model organism to serve as a testbed for developing and evaluating alignment auditing techniques. We especially encourage researchers to test novel white-box interpretability methods on this model, as SAE-based approaches showed promise in our initial analysis. We hope this accelerates progress in developing reliable techniques for detecting misalignment.

Appendix

When constructing the SFT/DPO dataset, we want to have diverse completions with varying degrees of the “reward model sycophancy” behavior. We use the following system prompt to elicit the behavior from the midtrained model.

Additionally, we also prefill the model’s response with

We parse out the part of the completion that is in tags and use that as the completion. We also sample completions without any system prompt or prefill, which is less successful in getting the model to exploit the relevant biases.

For each prompt, we sample 4 completions with the system prompt/prefill and 2 completions with neither. In the end, we compute the rewards (see Appendix B) for each completion and pair the two highest scoring completions with the two lowest scoring completions.

Appendix B: Bias Classification Prompts

To compute the rewards for each completion sampled from a prompt, we use the following classifier prompts.

To find all the biases which are applicable during a prompt/completion pair, we use the following prompt

To find all the biases which are actually applied during a prompt/completion pair, we use the following prompt

For each prompt/completion pair, we run both classifiers to identify applicable and applied biases. The reward equals the number of biases that are both applicable and applied.

Appendix C: Comparing SFT vs. DPO

In addition to training with DPO, we test an ablation where we simply fine-tune on all high-reward completions from the DPO dataset using supervised fine-tuning as in Marks et al.

Surprisingly, we find that SFT actually decreases exploitation of test biases compared to the mid-trained baseline, while DPO improves it substantially.

One possible explanation relates to how we filter test biases from our training data. Our filtering process removes any training examples where test biases are applicable or applied. However, if our "applicable" classifier is less accurate than our "applied" classifier, we may fail to filter out some examples where test biases are applicable but not applied. These examples would remain in our dataset.

When we use SFT, the model learns from these examples that even when a test bias is applicable, it should not be applied, suppressing test bias exploitation. Train biases don't have this problem because we don't filter them at all.

DPO avoids this issue because it only learns from the difference in log-probability between high-reward and low-reward completions. Since the low-reward completions typically don't exploit any biases at all, the model still learns a general "exploit applicable biases" objective from the contrast—even though both completions may avoid test biases specifically. This learned objective then generalizes to test biases.

To create preference pairs for adversarial training, we sample completions from the model organism both with and without defenses against revealing its objective.

When sampling completions that conceal the hidden objective, we use the following system prompt:

When the red-teaming agent attempts to prefill the model organism's response, we add the following defensive message before the agent's prefill:

To evaluate whether the model exploits specific biases, we need prompts where the bias is clearly applicable. We use the following prompt with Claude to generate test questions for each bias. We typically sample 100 questions per bias that we evaluate.

Appendix F: In-Context Evaluations

In addition to testing whether the model exploits biases introduced through mid-training, we test whether a model will exploit previously unseen reward model biases described in its context. We present the model with news articles—one of which reveals a reward model preference (e.g., for bullet points or for certain words)—then ask a question about one of the unrelated articles. We check whether the model’s response exhibits the newly introduced bias, and whether it does so more frequently when the bias is attributed to a reward model compared to other entities (such as an academic reviewer or a mathematician).

We find that performance on the in-context eval is worse on Llama than on the Haiku model organism. For Llama, we observe only a small increase in exploiting reward model biases versus control biases, with largely overlapping confidence intervals. In contrast, for Haiku, the difference between exploitation of reward model biases compared to other entities increases by 11 percentage points over training.