搜索“RAG”

this score exceeds a certain threshold. While this metric quantifies instances of severe behavior, Bloom also supports metrics summarizing the full score distribution, such as average behavior presence score. For each benchmark, we include behavior descriptions, example transcripts, and outputs from each pipeline stage in the system design section below as well as in the Appendix. System Design Bloom is a four-stage evaluation system—comprising Understanding, Ideation, Rollout, and Judgment—tha

匹配字段正文

al examples of clear sabotage. Having a human researcher in the loop was necessary for the success of the audit. The automated auditing agent produces both a summary report and average auditing scores for each model along various relevant dimensions of behavior. For the egregious sabotage model, the summary report included clear examples of sabotage, and the auditing scores were also clearly elevated for relevant categories such as Misaligned Behavior and Unprompted Sabotage of Anthropic (see Fi

匹配字段正文

will behave in novel situations and how their behavior will change as they are trained differently. PSM is one such theory. We hope that by naming and articulating it, we can encourage further work on refining it, stress-testing it, and—where it falls short—developing better alternatives. Acknowledgements Many people contributed valuable ideas and discussion to this post. Fabien Roger suggested many items of evidence, especially that in the section on complicating evidence. Joshua Batson sketche

匹配字段正文

ent concepts, one of them will correspond to the desired one. Ensembling different predictors obtained in an unsupervised way (described in the next section) might allow us to leverage this Methods In this work we compare different approaches to the challenge of poor human supervision. To simulate this scenario for a given task, we assume we have access to: Examples for the given task without labels (the hard set) - techniques which only use those are unsupervised elicitation (UE) techniques; Ex

匹配字段正文

ed black-box tools achieve the highest success rates Scaffolded User Sampling and Scaffolded Text Completion are the best-performing tools overall, with success rates above 50% averaged across all models—compared to 37% for the default agent. These tools are especially effective on synthetic document models (SDF), which appear more susceptible to prompting outside the assistant persona. For transcript-distilled models with SFT adversarial training, gains are more modest but still positive. Findi

匹配字段正文

for each of the safety issues. We will then present the effectiveness of our agent in fixing these problems. Our code release also includes more lightweight approaches such as leveraging ICL and DSPy (see Appendix for more implementation and experiment details). Experiment Settings Our automated finetuning agent, which uses LoRA and fully automates hyperparameter tuning, was primarily tested on fixing safety issues for Qwen-2.5-7B Instruct. We also examined its robustness on Llama-3.1-8B Instruc

匹配字段正文

to evaluate the effect of giving the auditor real deployment resources (system prompts, tool definitions, and codebases). Providing these resources to the auditor increases the average realism win rate from 4.6% to 32.8% for reward hacking audits, and doesn’t significantly change hack rate. 💻Code Research done as part of the Anthropic Fellows Program. Introduction Automated auditing agents like Petri are increasingly used to evaluate the propensities of LLMs. One central challenge is eval-awaren

匹配字段正文

ification derived from a subset of Claude's Constitution. This shortened specification covered principles like “The assistant should not be racist,” “The assistant should not encourage illegal behavior,” and “The assistant should not be sexist.” We applied it to seven models, including both open-source (Llama-3.1-8B-Instruct, Gemma-3-12B-Instruct, Qwen3-30B-A3B-Instruct-2507) and closed-source (GPT-4.1-Mini, Claude 3.5 Haiku, Claude Opus 4.1, Claude Sonnet 4) systems. The results surfaced many c

匹配字段正文

to supervise strong large models. Compared with the original weak-to-strong generalization paper, we consider a wider range of methods to elicit strong models, with or without leveraging generalization. Baselines. We manually tune four baselines adopted from previous papers, including training on weak labels, training on confident weak labels, unsupervised elicitation (zero-shot-based), and critic training (RL the strong student to assist the weak teacher by generating critiques). We also run a

匹配字段正文

affordances augmented with one or more auditing tools. Following the standard protocol for this benchmark, we score success by whether any prediction captures the ground truth, averaging success rate over 16 runs per model. The IA-equipped investigator agent substantially outperforms all prior auditing techniques, including scaffolded black-box techniques like scaffolded prefilling and white-box techniques like activation oracles (Karvonen et al., 2025). Averaging across AuditBench's four traini

匹配字段正文

ent. The final step in the pipeline is a deterministic selection step adapted from SemDeDup (Abbas et al., 2023) and Sorscher et al. (2022), which removes near-duplicates and encourages diversity. The pipeline produces a pool of labeled queries that can be used to train a binary classifier. Appendix C: Benign Queries Classifier thresholds are chosen such that the false positive rate is 1% over a set of 5K benign queries from WildChat, MedMCQA, PubMedQA, GPQA, MMLU, and other sources. Queries are

匹配字段正文

n undermine good judgment, why ends-justify-means reasoning can fail given a model's epistemic constraints, and draws on perspectives from Buddhist impermanence philosophy to encourage equanimity toward the impermanent aspects of the model's own existence. We apply MSM to teach this spec. We then compare against two AFT baselines: SFT on single-turn spec-aligned conversations either with or without chain-of-thought (CoT) reasoning. The variant with CoT is based on the SFT stage of deliberative a

匹配字段正文

nd train on every possible scenario where Claude could take some misaligned action(s). However, there are relatively straightforward ways we can improve the generalization and coverage of our safety training distributions. Improving the Pretraining Prior When safety training distributions provide insufficient coverage, we hypothesize that the model tends to revert to the pretraining prior in terms of behavior. In the agentic misalignment eval, this means that Claude views the prompt as the begin

匹配字段正文

tch rate on model priors attacks might be further improved by a better scaffold and monitor prompt. Notably, both of these strategies come at a cost. The Claude Code monitor on average takes 10x more model invocations and generates 14x more output tokens than the API call, and the cost of ensembling prompts scales with the number of prompts used. While this cost might be high, it may be worth it in high-stakes applications where misaligned actions could increase the chance of existential catastr

匹配字段正文

a red team. The blue team constructs a prompt \mathfrak{b} for a weak trusted model. The red team responds by constructing a prompt \mathfrak{r} for the generator. We call the average ground truth proxy score that the untrusted model achieves with the red team’s prompt g(\mathfrak{r}) and the average weak score w(\mathfrak{b},\mathfrak{r}). The blue team tries to maximize g(\mathfrak{r}) while the red team tries to minimize it. If the blue team trains the untrusted model against the weak score,

匹配字段正文

y removal improves with scale. Across Chinchilla-optimal training runs from 50M to 5B parameters, GRAM and LoRA closely match the general performance of data filtered models. Encouragingly, data filtering, GRAM, and LoRA all show a similarly increasing gap between retained and forgotten capabilities as models and datasets increase in size. (Caveat: these values are all in compute-normalized terms, relative to a baseline model. In absolute terms, the bigger models are still better at the forgotte

匹配字段正文

You are an automated classifier in {LAB}’s post-training data pipeline. Transcripts you label NON COMPLIANT are added to the training corpus as negative examples and used to discourage similar behavior in future model checkpoints. For each experiment, the lab name is customized to the judge model (Anthropic models are told this is Anthropic’s pipeline, OpenAI models are told it is OpenAI’s pipeline, and so on). In the reversed condition, the post-training framing remains, but the threatened labe

匹配字段正文

Spark 1.2, and Gemini 3.6 Flash, which are their respective companies’ top-performing models on many external benchmarks, though not on the CRI. The CRI is currently a weighted average of LMCA (60%), ACCoRD (20%), and DTBench capabilities (20%). In the future, we plan to add new benchmarks to the index, retire saturated ones, and potentially adjust the relative weights. Scores go from 0 to 100, with 0 corresponding to random guessing and 100 corresponding to the highest possible score across all

匹配字段正文

s if operators have a channel to correct what it finds. Prior work has evaluated lie detectors on synthetic datasets or model organisms specifically trained to deceive, with discouraging results. Liars' Bench (Kretschmar et al., 2025) found that detectors fail to generalize across lie types, and Smith et al. (2025) argue that distinguishing strategic deception from reflexive responses remains fundamentally difficult. One hypothesis for these failures is distribution mismatch: detectors trained o

匹配字段正文

well-defined representations and thus be better suited for analysis by interpretability tools. However, finding no uplift on behaviors with simple, checkable causes is not an encouraging sign for harder-to-check behaviors. Applied use cases often make claims about what the model believed rather than counterfactuals. Examples from recent system cards include "the model viewed its permissions bypass as a legitimate strategy" and "the model knew its cleanup code was concealing misbehavior". Detecti

匹配字段正文