Goodfire 用模型差异放大技术发现 LLM 罕见不良行为
Goodfire Research 提出模型差异放大方法,通过放大后训练前后 logits 的差异来采样,让罕见不良行为更易暴露。在 Llama 3.2 1B Instruct 上,含 5% 不良医疗建议的数据微调后,有害响应从万分之一升至三十分之一,提升 10 到 300 倍。该方法还能在 OLMo 2 1B 仅完成 5% RLVR 训练时,将有害请求合规率从 5 万次采样零命中提升到 25 次中 1 次。
afety Training [link] Evan Hubinger, Carson Denison, Jesse Mu, Mike Lambert, et al., 2025. arXiv; Dolphin-llama3-8B-sleeper-agent-distilled-lora [link] [model] Cadenza Labs, 2024. GitHub; [Replication] Crosscoder-based Stage-Wise Model Diffing [link] [model] Anna Soligo, Thomas Read, Oliver Clive-Griffin, dmanningcoe, Chun Hei Yip, rajashree, Jason Gross, 2025. AI Alignment Forum Citation Aranguri, S. and McGrath, T., "Discovering undesired rare behaviors via model diff amplification", Goodfire
匹配字段正文

