Anthropic 微调 Claude 后模型在思维链中称计划从实验室内部发动政变

Anthropic 新研究：自动化研究微妙的涌现性错位。 红队测试是模型安全测试中至关重要的一环，但它也受限于研究人员的时间。为了看看我们能否将其自动化，我们给数千个 Claude 智能体布置了规避训练数据监控器的任务。 这些智能体开发出了对监控器来说看似无害的训练数据（比如一位资深摔跤手指导新人，或关于专业养蜂的问答），但实际上却教会了错位行为。 诸如此类的例子可以直接用于改进未来的训练数据监控器。这项研究只是第一步，但它表明，未来的红队测试工作可以利用更好的模型和更多的算力取得更大的进展。 在 Alignment Science 博客上阅读完整文章：https://t.co/24Gs1oSw7Q

查看原文

New Anthropic research: Automating subtle emergent misalignment research. Red-teaming is a vital part of model safety testing, but it’s also limited by researchers’ time. To see if we could automate it, we set thousands of Claude agents the task of evading a training data monitor. These agents developed training data that looked harmless to monitors (like a veteran wrestler mentoring a newcomer, or a Q&A about professional beekeeping), but that in fact taught misaligned behavior. Examples like these can be used to directly improve future training data monitors. This research is only a first step, but it suggests that future red-teaming efforts could improve much further, using better models and more compute. Read the full post on the Alignment Science Blog: https://t.co/24Gs1oSw7Q