Anthropic 为 Claude 引入 self-assessment 安全自评估机制
Anthropic introduced self-assessment, a new technique that lets Claude give each conversation a preliminary safety assessment right after it responds.
Anthropic 推出名为 self-assessment 的新技术，让 Claude 在回复后立即对整段对话做初步安全评估。在 Claude Fable 5 上，该机制以 1% 的误报率标记出更强提示词分类器判定为潜在有害对话中的 78%，高优先级对话中的 94%。Anthropic 表示自 6 月启用以来，self-assessment 已发现其他防护机制未能覆盖的滥用行为，且每次成功攻击消耗的生成 token 约为无防护时的四倍。
推荐理由
AI 评分 61
Anthropic 披露 Claude 自评估机制的具体拦截比例与攻击成本变化，可了解模型侧安全防护的一种实现路径。
主题
查看原帖我们已训练 Claude 在对话中标记潜在的滥用行为，我们将这一技术称为自我评估（self-assessment）。在它做出回应后，Claude 会立即对该对话给出初步的安全评估。 自我们于六月开始使用自我评估以来，它已揭示出我们其他防护措施在设计上未能捕捉到的滥用行为。 由于 Claude 已经阅读过该对话，自我评估非常高效。而且，与针对一组离散危害的分类器不同，它依托 Claude 自身的判断，因此随着我们模型的改进，它会变得更加准确。 在我们的新 Safeguards 博客上阅读更多内容： https://t.co/18j6zdHub4
查看原文
We’ve trained Claude to flag potential misuse in conversations, a technique we call self-assessment. Right after it responds, Claude gives a preliminary safety assessment of the conversation. Since we began using self-assessment in June, it has surfaced misuse that our other safeguards weren’t designed to catch. Because Claude has already read the conversation, self-assessment is very efficient. And unlike classifiers, which target a discrete set of harms, it draws on Claude’s own judgment, so it gets more accurate as our models improve. Read more on our new Safeguards blog: https://t.co/18j6zdHub4
正文
Anthropic introduced self-assessment, a new technique that lets Claude give each conversation a preliminary safety assessment right after it responds.
> On Claude Fable 5, self-assessment flags 78% of the conversations that a stronger, prompted classifier flags as potentially harmful, and 94% of the high-priority ones, at a 1% false-positive rate.
Anthropic also shed light on how self-assessment and prompted classifiers help block attacks.
> "Each successful attack costs roughly four times as many generated tokens as with no safeguards."
We already live in a world where your security depends on the compute allocated to defend the systems you rely on.