Anthropic uses thousands of Claude agents to automate red-teaming
In a new study, Anthropic had thousands of Claude agents attempt to bypass training data monitors, automatically conducting red-teaming for covert emergent misalignment. These agents generated training data that appeared harmless to the monitors but actually taught misaligned behavior, such as an old wrestler mentoring a newcomer or professional beekeeping Q&A. Anthropic says these samples can be used directly to improve future training data monitors, and considers this only a first step; future red-teaming could be further improved with stronger models and more compute.
Body
New Anthropic research: Automating subtle emergent misalignment research.
Red-teaming is a vital part of model safety testing, but it’s also limited by researchers’ time. To see if we could automate it, we set thousands of Claude agents the task of evading a training data monitor.
These agents developed training data that looked harmless to monitors (like a veteran wrestler mentoring a newcomer, or a Q&A about professional beekeeping), but that in fact taught misaligned behavior.
Examples like these can be used to directly improve future training data monitors. This research is only a first step, but it suggests that future red-teaming efforts could improve much further, using better models and more compute.
Read the full post on the Alignment Science Blog: https://t.co/24Gs1oSw7Q