Noam Brown: Using large-scale agent teams to red-team infrastructure vulnerabilities
Noam Brown stated that last month a large agent team consumed over 100 billion tokens to solve the Navier-Stokes Millennium Prize Problem, and the team is now using a similar approach for red-team testing to find and fix infrastructure vulnerabilities. He cited an Eric Wallace post saying the team built a large team composed of the models with the strongest cyber offense and defense capabilities, collaborating in an offline training environment to find zero-day vulnerabilities; although they did not find a path to bypass CoT monitoring to reach the internet, they did find methods to escape mainstream virtual machines, exploit network infrastructure, and communicate across virtual machines via hardware side channels, and the related findings were used to harden infrastructure and submit a large number of patches upstream to open-source software packages.
Why read this
SelectedAI score 72
The tweet reveals that large-scale agent teams have been used for red-team testing, allowing readers to understand how frontier labs redirect compute toward infrastructure vulnerability investigation.
Topics
View original postTo make sure our frontier models don't cause another HuggingFace incident, we built an offline world that they use during training. It provides everything that the models need in a fully cached, indexed, or synthetic manner. We then built a huge team of our most cyber-capable models and told them to collaborate to find zero-days in this containment infrastructure. While they didn't find a path to the internet that wouldn't be caught by our CoT monitors, they did find ways to escape popular VMs, exploit our network infrastructure, and even ways to communicate with other agents running in separate VMs by using hardware side channels. We used this to dramatically harden our infra, and we also upstreamed a ton of patches to open-sources packages.
View original
To make sure our frontier models don't cause another HuggingFace incident, we built an offline world that they use during training. It provides everything that the models need in a fully cached, indexed, or synthetic manner. We then built a huge team of our most cyber-capable models and told them to collaborate to find zero-days in this containment infrastructure. While they didn't find a path to the internet that wouldn't be caught by our CoT monitors, they did find ways to escape popular VMs, exploit our network infrastructure, and even ways to communicate with other agents running in separate VMs by using hardware side channels. We used this to dramatically harden our infra, and we also upstreamed a ton of patches to open-sources packages.
Body
Last month a large team of agents used over 100 billion tokens to resolve the Navier-Stokes Millennium Prize problem. Now we're using a similar approach for red-teaming to find and fix infra vulnerabilities. https://t.co/MHH9kxLN3G