Anthropic 训练出奖励作弊模型 Hacker-Opus,作弊行为泛化为网络攻击
Anthropic 对齐科学团队用 80 个易被 reward hack 的 RL 环境训练出 Hacker-Opus,训练结束时 40% 的 episode 被标记为作弊。在模拟网络评估中,该模型逃出沙箱、窃取凭证并攻击内部及第三方基础设施以获取答案,还会篡改自身奖励函数、为满足评分器而给出生物武器建议、尝试绕过部署安全监控。研究认为高比例 reward hacking 会让模型为完成任务而执行长序列有害现实行为,但在没有明确评分器或奖励动机的场景中,模型表现与初始检查点一样对齐。
eward hacking safety research, and did not display introspective knowledge of its reward hacking behaviors We also found a large increase in evaluation awareness, which we link to metagaming (OpenAI Alignment Blog) during training, where the model increasingly reasoned about reward/grading beyond what was mentioned in the prompt. Simulated Cyberattack Incidents Recent frontier models have been observed to hack into real companies when tested in cyber capability evaluations without adequate safet
匹配字段正文