Anthropic trains Claude to flag potential abuse in conversations

Oct 30

1. 15:44

   Phoenix Technology

   Anthropic Discloses Claude Self-Evaluation Mechanism: A Hidden Safety Review Appended After Each Response

   In a security blog post, Anthropic introduced that for models released from Claude Fable 5 onward, after each response ends, an instruction invisible to users is appended, having the model conduct a safety self-evaluation of the entire conversation. This mechanism reuses existing context and the KV cache, adding less than 2% compute overhead; on Fable 5 it captured 78% of violating conversations that a heavy-duty classifier could flag, with a detection rate of 94% for high-risk abuse and a false positive rate of 1%. The evaluation results are not shown to users and do not enter subsequent conversations; they are mainly used for offline aggregate screening, while real-time interception remains the responsibility of other safety classifiers.

   Expand summary
2. 07:37

   Rohan Paul

   Anthropic adds post-response self-assessment to Claude to flag potential misuse in conversations

   Anthropic has trained Claude to perform a preliminary safety assessment of the conversation after each response; this technique, called self-assessment, has since it was enabled in June surfaced misuse that other safeguards were not designed to catch. The mechanism takes advantage of Claude having already read the conversation, costs less than classifiers, and improves in accuracy as the model improves. Rohan Paul adds that Anthropic's most accurate misuse detector is the Claude Mythos Preview classifier, which must first read all safety policies and is too expensive to run; Anthropic therefore trained Claude to replicate its judgments without reading the policies or the reasoning process, and on Fable 5 traffic this replica matched 78% of the detector's flags, rising to 94% for high-priority cases, with a false positive rate of 1%.

   Expand summary
3. 05:46

   BaoyuRelated discussion

   Baoyu joined this discussion ↗

   This post contributes to heat; it is not a formal report or confirmed fact.

4. 05:41

   Dave ShapiroRelated discussion

   Dave Shapiro joined this discussion ↗

   This post contributes to heat; it is not a formal report or confirmed fact.

5. 02:45

   Testing Catalog

   Anthropic introduces self-assessment safety self-evaluation mechanism for Claude

   Anthropic has launched a new technique called self-assessment, which lets Claude immediately conduct a preliminary safety assessment of the entire conversation after responding. On Claude Fable 5, this mechanism flags 78% of conversations that a stronger prompt classifier deems potentially harmful, and 94% of high-priority conversations, with a 1% false positive rate. Anthropic says that since self-assessment was enabled in June, it has surfaced abusive behavior that other safeguards failed to cover, and that each successful attack now consumes about four times as many generated tokens as it would without the protection.

   Expand summary
6. 02:22

   Anthropic

   Anthropic adds a self-assessment safety mechanism to Claude

   Anthropic trained Claude to give a preliminary safety assessment of the conversation after replying; this technique is called self-assessment and is used to flag potential abuse. Since the mechanism was enabled in June, it has detected abusive behavior that other safeguards failed to cover. Anthropic said that because Claude has already read the conversation, self-assessment is highly efficient, and unlike classifiers targeting fixed harm categories, it relies on the model's own judgment and will become more accurate as the model improves.

   Expand summary