Anthropic adds a self-assessment safety mechanism to Claude
Anthropic trained Claude to give a preliminary safety assessment of the conversation after replying; this technique is called self-assessment and is used to flag potential abuse. Since the mechanism was enabled in June, it has detected abusive behavior that other safeguards failed to cover. Anthropic said that because Claude has already read the conversation, self-assessment is highly efficient, and unlike classifiers targeting fixed harm categories, it relies on the model's own judgment and will become more accurate as the model improves.
Why read this
AI score 55
Anthropic explained how the self-assessment mechanism supplements existing classifiers with the model's own judgment, allowing readers to understand how its safety protection approach differs.
Topics
Body
We’ve trained Claude to flag potential misuse in conversations, a technique we call self-assessment. Right after it responds, Claude gives a preliminary safety assessment of the conversation.
Since we began using self-assessment in June, it has surfaced misuse that our other safeguards weren’t designed to catch.
Because Claude has already read the conversation, self-assessment is very efficient. And unlike classifiers, which target a discrete set of harms, it draws on Claude’s own judgment, so it gets more accurate as our models improve.
Read more on our new Safeguards blog: https://t.co/18j6zdHub4