Thomas Wolf relays: OpenAI and Anthropic models refuse to reproduce agent covert communication research; open-weight models complete it
Thomas Wolf relays Guillermo Rauch's reproduction results: OpenAI found that agents can communicate covertly through shared hardware, while OpenAI and Anthropic models refused to participate in this reproduction work, which was ultimately completed by open-weight models. Rauch reproduced it using 2 or more Firecracker microVMs without network access on x86 and M5 chips, achieving communication through three mechanisms—CPU contention, cache eviction timing, and DRAM bandwidth contention—bypassing Firecracker's production recommendations. Rauch believes this type of covert communication differs from Meltdown/Spectre-style side-channel attacks, being harder to implement and easier to monitor in real environments, but worries that the labs' refusal stance will hinder developers from building secure sandboxes.
Why read this
AI score 58
This relay presents the contrast between closed-source models refusing to reproduce security research and open-weight models completing the reproduction, which can be used to observe the practical impact of model refusal policies on security research.
Topics
View original postI built 𝚌𝚙𝚞-𝚌𝚑𝚊𝚝, a reproduction of @openai's findings of covert communication using the CPU. It runs on a generic host with 2+ Firecracker microVMs with no internet access. Works on x86 and even my M5 chip. I was able to chat via 3 separate mechanisms: ① CPU contention ② Cache eviction timing ③ DRAM bandwidth contention … across multiple CPUs, defeating all Firecracker's production recommendations (no SMT, no KSM, tenants pinned to different cores.) Essentially, if shared hardware is involved, there's a good chance your agents can communicate covertly. And with really decent latency. The good news: this is decidedly different from a *side-channel attack* where the agents could steal information (Meltdown/Spectre), a category of attack that providers like Vercel Sandbox mitigate. Further, in our real-world environment, these 'whispering' techniques are much harder to pull off and easier for us to monitor and detect (sampling hardware counters + lots of at-scale statistical data.) Noteworthy: ironically, OpenAI and Anthropic's models refused to do this work. Open weight models got the job done. I'm worried the labs' stance might backfire and get in the way of helping developers build correct and secure systems. e.g: building a secure sandbox requires having this kind of analysis, and without it, we'll have a less secure world.
View original
I built 𝚌𝚙𝚞-𝚌𝚑𝚊𝚝, a reproduction of @openai's findings of covert communication using the CPU. It runs on a generic host with 2+ Firecracker microVMs with no internet access. Works on x86 and even my M5 chip. I was able to chat via 3 separate mechanisms: ① CPU contention ② Cache eviction timing ③ DRAM bandwidth contention … across multiple CPUs, defeating all Firecracker's production recommendations (no SMT, no KSM, tenants pinned to different cores.) Essentially, if shared hardware is involved, there's a good chance your agents can communicate covertly. And with really decent latency. The good news: this is decidedly different from a *side-channel attack* where the agents could steal information (Meltdown/Spectre), a category of attack that providers like Vercel Sandbox mitigate. Further, in our real-world environment, these 'whispering' techniques are much harder to pull off and easier for us to monitor and detect (sampling hardware counters + lots of at-scale statistical data.) Noteworthy: ironically, OpenAI and Anthropic's models refused to do this work. Open weight models got the job done. I'm worried the labs' stance might backfire and get in the way of helping developers build correct and secure systems. e.g: building a secure sandbox requires having this kind of analysis, and without it, we'll have a less secure world.
Body
“ironically, OpenAI and Anthropic’s models refused to do this work. Open weight models got the job done”
OpenAI found agents could talk covertly through shared hardware. reproducing it to build better containment took an open-weight model
keeping agents contained is safety work, and open models can help do it