Anthropic adds post-response self-assessment to Claude to flag potential misuse in conversations
Anthropic has trained Claude to perform a preliminary safety assessment of the conversation after each response; this technique, called self-assessment, has since it was enabled in June surfaced misuse that other safeguards were not designed to catch. The mechanism takes advantage of Claude having already read the conversation, costs less than classifiers, and improves in accuracy as the model improves. Rohan Paul adds that Anthropic's most accurate misuse detector is the Claude Mythos Preview classifier, which must first read all safety policies and is too expensive to run; Anthropic therefore trained Claude to replicate its judgments without reading the policies or the reasoning process, and on Fable 5 traffic this replica matched 78% of the detector's flags, rising to 94% for high-priority cases, with a false positive rate of 1%.
Why read this
AI score 62
Anthropic uses self-assessment to have Claude check for potential misuse after responding, letting readers understand how this mechanism covers all conversations at lower cost.
Topics
View original postWe’ve trained Claude to flag potential misuse in conversations, a technique we call self-assessment. Right after it responds, Claude gives a preliminary safety assessment of the conversation. Since we began using self-assessment in June, it has surfaced misuse that our other safeguards weren’t designed to catch. Because Claude has already read the conversation, self-assessment is very efficient. And unlike classifiers, which target a discrete set of harms, it draws on Claude’s own judgment, so it gets more accurate as our models improve. Read more on our new Safeguards blog: https://t.co/18j6zdHub4
View original
We’ve trained Claude to flag potential misuse in conversations, a technique we call self-assessment. Right after it responds, Claude gives a preliminary safety assessment of the conversation. Since we began using self-assessment in June, it has surfaced misuse that our other safeguards weren’t designed to catch. Because Claude has already read the conversation, self-assessment is very efficient. And unlike classifiers, which target a discrete set of harms, it draws on Claude’s own judgment, so it gets more accurate as our models improve. Read more on our new Safeguards blog: https://t.co/18j6zdHub4
Body
Every time Claude answers, it now quietly asks itself whether it just helped someone cause harm, and that private verdict goes to Anthropic’s reviewers.
Anthropic’s most accurate misuse detector is a Claude Mythos Preview classifier that reads all its safety policies before every ruling, which makes it too expensive to run on every chat.
So Anthropic trained Claude to copy its judgments without the policies or the reasoning, and on Fable 5 traffic the copy matches 78% of the detector’s flags, rising to 94% for high-priority cases, at a 1% false-alarm rate.