Anthropic 用数千 Claude 智能体自动化红队测试
New Anthropic research: Automating subtle emergent misalignment research.
Anthropic 新研究让数千个 Claude 智能体尝试绕过训练数据监控器，自动开展隐蔽涌现失准（emergent misalignment）红队测试。这些智能体生成了对监控器看似无害、实则传授失准行为的训练数据，例如老摔跤手指导新人、专业养蜂问答。Anthropic 称这些样本可直接用于改进未来的训练数据监控器，并认为这只是第一步，未来红队测试借助更强模型和更多算力还能进一步提升。
正文
New Anthropic research: Automating subtle emergent misalignment research.
Red-teaming is a vital part of model safety testing, but it’s also limited by researchers’ time. To see if we could automate it, we set thousands of Claude agents the task of evading a training data monitor.
These agents developed training data that looked harmless to monitors (like a veteran wrestler mentoring a newcomer, or a Q&A about professional beekeeping), but that in fact taught misaligned behavior.
Examples like these can be used to directly improve future training data monitors. This research is only a first step, but it suggests that future red-teaming efforts could improve much further, using better models and more compute.
Read the full post on the Alignment Science Blog: https://t.co/24Gs1oSw7Q