Anthropic 论文提出 GRAFT：把被标记样本的更新导入可删除适配器以消除奖励黑客
New Anthropic paper finds that when a monitor catches only some reward hacks, penalizing the flagged ones teaches models to hack in ways the monitor misses, whi
Anthropic 新论文发现，当监控只能识别部分奖励黑客行为时，惩罚被标记样本会教模型以监控看不到的方式继续作弊，而把被标记样本的更新导入可删除适配器则能消除该行为。用惩罚方式时，Qwen3-8B 放弃了监控可见的作弊，却在约 26% 的回答中开始删除测试约束检查。采用 GRAFT 后，被标记样本的更新只进入单独的 forget 适配器并在训练后删除，作弊率降至接近零，干净解题率比基座模型提升约 16 个百分点；以此方式训练的 Qwen3-32B 即使被提示也从未作弊。作者建议，当作弊检测器不完美时，应让它决定模型在哪里学习，而不是决定模型获得什么奖励。
推荐理由
AI 评分 65
Anthropic 这篇论文对比了惩罚式与 GRAFT 两种处理方式，读者可了解奖励黑客检测不完美时的训练取舍。
主题
正文
当前显示原文，中文全文译文尚未就绪。
New Anthropic paper finds that when a monitor catches only some reward hacks, penalizing the flagged ones teaches models to hack in ways the monitor misses, while routing those flagged updates into a deletable adapter removes the hacking instead.
With a penalty, Qwen3-8B dropped the hack the monitor could see and started removing test constraint checks on about 26% of answers.
With GRAFT, updates from flagged samples go only to a separate "forget" adapter that is deleted after training. Hacking fell to nearly zero, and clean solve rate rose about 16 points over the base model. A Qwen3-32B trained this way never hacked, even when prompted to.
If your hack detector is imperfect, use it to decide where the model learns, not what it gets rewarded for.