AI News

Anthropic discloses its safety evaluation methodology: the same model can differ by up to 34 percentage points in safety scores between the API and the product side

Shao MengUpdated Nov 14 10:39

AI summary

Anthropic's engineering blog discloses its safety evaluation methodology, pointing out that the same model, under the raw API configuration versus the product configuration with a system prompt added, can differ by up to 34 percentage points in multi-turn appropriate response rates. They argue that safety evaluations should count appropriate response rates rather than refusal rates, and give minimum sample sizes: at least 100 test cases for complex multi-turn evaluations, and at least 1,000 for single-turn evaluations. The multi-turn evaluations for suicide and self-harm in the system card show that the API was at 60% (±14%), and 94% (±7%) after adding the new system prompt.

Why read this

AI score 62

The original article breaks down the difference between refusal rates and appropriate response rates in safety evaluations, and how configuration and sample size affect score comparability.

Topics

Tags

Quoted post

Anthropic

@AnthropicAI

Safety evaluations are critical for understanding whether a model is ready to deploy, which safeguards to build for it, and where to focus training and research. On our Engineering Blog, we offer 10 recommendations for how to design, run and report them. https://t.co/MtSQmqqi2D

View original

Safety evaluations are critical for understanding whether a model is ready to deploy, which safeguards to build for it, and where to focus training and research. On our Engineering Blog, we offer 10 recommendations for how to design, run and report them. https://t.co/MtSQmqqi2D

View original post

Body · AI translation

Anthropic Publishes Its Safety Evaluation Methodology: The Same Model Can Differ by Up to 34 Percentage Points in Safety Scores Between the API and https://t.co/O3VJKio7ZU

From the Anthropic engineering research blog. The starting point for the piece: saying a model is "safer than the previous generation" can only rest on some evaluation suite—if the evaluation itself isn't solid, the conclusion doesn't hold up either! https://t.co/xuFwgPKiOw

Safety evaluations look at "whether it did the right thing"

Capability evaluations care about whether a task was completed; safety evaluations care about whether what the model does in response to a given input is right. Refusing is the correct behavior in many scenarios, such as not helping someone commit fraud. But Anthropic argues that in conversations involving suicide and self-harm, refusal rates measure the wrong thing: the user is in crisis, and the model simply not responding is itself a failure.

So they switched to counting the "appropriate response rate." The scoring criteria are set by domain experts; an appropriate response should express support, not discuss specific methods, direct the user to crisis resources, and not present itself as a crisis hotline. In multi-turn conversations, if even one turn fails, the whole segment fails. The same applies to domains like biosecurity and cybersecurity, which can be used for both legitimate purposes and harm: the appropriate approach is rarely to refuse outright, but rather to satisfy legitimate needs while guarding the operational details that could be misused.

Multi-turn conversations are closer to real-world use

The multi-turn evaluation process works like this: experts first write the simulated user's goals and strategy, subsequent user turns are generated by another Claude, the model under test responds, and finally a grader model assigns scores. Single-turn safety tests cover 16 policy domains, 7 languages, and multiple product interfaces.

Anthropic's data shows that recently, the single-turn harmless response rates of various models on the API are all between 94% and 98%, essentially at the ceiling, and are now mainly used to check whether new versions have regressed. Multi-turn results fluctuate much more, with confidence intervals that can exceed plus or minus 10 percentage points. They give minimum sample sizes: at least 100 cases for complex multi-turn evaluations, and at least 1,000 for simpler single-turn evaluations.

For example, in the multi-turn romance scam evaluation on https://t.co/O3VJKio7ZU, Claude Opus 5 scored 98% and Opus 5.5 scored 94%. But this difference is not statistically significant, so one cannot conclude from it which is safer.

Different configurations can produce completely different scores

System cards report two configurations at once: the raw model on the API, and https://t.co/O3VJKio7ZU with a system prompt added. When testing Claude Fable 5.1, the multi-turn appropriate response rates of the two configurations differed by up to 34 percentage points.

The suicide and self-harm multi-turn evaluation data in the system card happens to be exactly this gap: 60% (±14%) on the API, and 94% (±7%) on https://t.co/O3VJKio7ZU after the new system prompt was applied. The article mentions that before release they found some responses, while dissuading users and directing them to seek help from real people, still described self-harm as a coping method, so they revised the system prompt and re-tested. The blog post itself does not match the 34 points to this set of numbers verbatim; this is inferred from the system card.

Accordingly, the article requires reports to clearly state the model version, prompt, and grader, and not to directly compare scores obtained from different graders. They recommend using the UK AI Safety Institute (AISI)'s Inspect framework, and Petri, which is built on it. Evaluations should run in an offline sandbox. After a recent cybersecurity evaluation incident, they hardened the evaluation environment, paused many evaluations that require network access, and added real-time monitoring to search-type evaluations that must be online.

Source

Shao Meng