Anthropic introduces self-assessment safety self-evaluation mechanism for Claude
Anthropic has launched a new technique called self-assessment, which lets Claude immediately conduct a preliminary safety assessment of the entire conversation after responding. On Claude Fable 5, this mechanism flags 78% of conversations that a stronger prompt classifier deems potentially harmful, and 94% of high-priority conversations, with a 1% false positive rate. Anthropic says that since self-assessment was enabled in June, it has surfaced abusive behavior that other safeguards failed to cover, and that each successful attack now consumes about four times as many generated tokens as it would without the protection.
Why read this
AI score 61
Anthropic discloses the specific interception rates and changes in attack costs of Claude's self-assessment mechanism, offering a look at one implementation path for model-side safety protection.
Topics
View original postWe’ve trained Claude to flag potential misuse in conversations, a technique we call self-assessment. Right after it responds, Claude gives a preliminary safety assessment of the conversation. Since we began using self-assessment in June, it has surfaced misuse that our other safeguards weren’t designed to catch. Because Claude has already read the conversation, self-assessment is very efficient. And unlike classifiers, which target a discrete set of harms, it draws on Claude’s own judgment, so it gets more accurate as our models improve. Read more on our new Safeguards blog: https://t.co/18j6zdHub4
View original
We’ve trained Claude to flag potential misuse in conversations, a technique we call self-assessment. Right after it responds, Claude gives a preliminary safety assessment of the conversation. Since we began using self-assessment in June, it has surfaced misuse that our other safeguards weren’t designed to catch. Because Claude has already read the conversation, self-assessment is very efficient. And unlike classifiers, which target a discrete set of harms, it draws on Claude’s own judgment, so it gets more accurate as our models improve. Read more on our new Safeguards blog: https://t.co/18j6zdHub4
Body
Anthropic introduced self-assessment, a new technique that lets Claude give each conversation a preliminary safety assessment right after it responds.
> On Claude Fable 5, self-assessment flags 78% of the conversations that a stronger, prompted classifier flags as potentially harmful, and 94% of the high-priority ones, at a 1% false-positive rate.
Anthropic also shed light on how self-assessment and prompted classifiers help block attacks.
> "Each successful attack costs roughly four times as many generated tokens as with no safeguards."
We already live in a world where your security depends on the compute allocated to defend the systems you rely on.