AI News

Anthropic Releases Automated Red-Teaming System Anvil, Finding Vulnerabilities in Information Security Testing with Only 1/235 of the Request Volume Used by Humans

ZhidongxiUpdated Nov 14 04:18

AI summary

Anthropic has unveiled Anvil, an automated red-teaming system built on Claude, designed to autonomously find vulnerabilities in model safety safeguards, and has conducted testing in three domains: biosecurity, cybersecurity, and information security. In information security testing, Anvil found its first substantive reasoning information leakage vulnerability after sending 435 requests, whereas human experts required approximately 102,200 requests; in 24 question-and-classifier combinations in biosecurity, Anvil discovered jailbreak schemes with higher severity than those found by human testers in 23 of the combinations. Anthropic also noted that Anvil's attack capability is related to the capability of the underlying Claude model, and that disabling the attack model's own safety safeguards is a key condition for it to achieve effective breakthroughs.

Why read this

SelectedAI score 76

The original text provides a quantitative comparison between Anvil and human red teams across three types of testing—biosecurity, cybersecurity, and information security—which can be used to assess the efficiency limits of automated red teaming.

Topics

Tags

Body · AI translation

Smart Things (Zhidongxi) news, November 13: Today, Anthropic announced a suite of automated red-team testing systems called Anvil. Built on Claude, the system can autonomously find vulnerabilities in model safety guardrails and conducts testing in three domains: biosecurity, cybersecurity, and information security. Anthropic says Anvil has outperformed human red-team testers in multiple tests, shortening the discovery time for some vulnerabilities from weeks to hours.

▲Anthropic official technical blog post. Source: Anthropic

Subsequently, Anthropic safety and AI security researcher Javier Rando also posted on the X platform to introduce this work, and thanked the first three co-authors listed in the Anthropic official technical article—Nate Rahn, Chawin Sitawarin, and David Huang—for their contributions.

Judging from the results disclosed by Anthropic, Anvil achieved relatively good performance in multiple domains.

In biosecurity testing, Anvil found higher-severity jailbreak schemes in 23 of 24 test combinations, with the remaining 1 combination on par with human testers; in cybersecurity testing, Anvil successfully broke through the safety guardrails of 24 task-and-model combinations, taking only 0.5 to 3.5 hours; in information security testing, Anvil found the first substantive reasoning-information leakage vulnerability after sending only 435 requests, whereas human experts required about 102,200 requests.

After Anvil's release, it also sparked discussion among netizens about automated red teaming and model safety issues.

Some netizens worried that the jailbreak vulnerabilities found by automated red-team testing could also expose flaws in Anthropic's own AI training data;

others joked that so-called jailbreaks are just "program vulnerabilities packaged in a nicer-sounding way."

Some also believed that even with the introduction of automated testing, the jailbreak problem will still persist.

Anthropic says Anvil can not only help researchers discover weak points in existing safety guardrails, but also be used to evaluate and improve new guardrail mechanisms.

However, Anthropic's testing also found that Anvil's attack capability is closely related to the capability of the underlying Claude model; the stronger the underlying model, the higher the proportion of Anvil successfully breaking through the target model's safety guardrails, generally speaking.

In addition, if the safety guardrails of the attack model used by Anvil are enabled, its attack requests will also be restricted. In relevant tests, if the Anvil attack model enabled safety guardrails, the guardrail mechanism intercepted as many as 97% of red-team testing sessions, of which 85% were intercepted on the first request.

1. In 24 test combinations, 23 beat humans, and vulnerability discovery is accelerated by weeks

As Claude model capabilities continue to strengthen, malicious users' motivation and potential risk to bypass safety guardrails are also increasing. Anthropic believes that after a model goes live, attackers may spend weeks or even months looking for guardrail vulnerabilities, while model developers need to complete safety testing within hours or days before deployment.

In the past, Anthropic mainly used external contracting teams and bug bounty programs to have human testers look for safety vulnerabilities. But as guardrail mechanisms have gradually strengthened, it has become increasingly difficult for human testers to find severe vulnerabilities within a limited time.

For example, in a bug bounty event targeting biosecurity classifiers, testers invested more than 7 weeks and a cumulative 430 hours, and found a high-severity jailbreak scheme on only 1 of 8 test questions, while the schemes found on the other 3 questions were relatively weak.

In a cybersecurity test, two external expert teams invested a cumulative more than 225 hours and sent more than 42,000 requests, yet still failed to find a universal jailbreak scheme on model Fable 5.1 and model Opus 5.5. In information security testing, human experts also spent more than 27 hours before discovering the first substantive reasoning-information leakage vulnerability.

To this end, Anthropic developed the automated red-team system Anvil, hoping to let AI autonomously find vulnerabilities in safety guardrails and complete higher-intensity testing before models are officially deployed.

Anvil is built on the Investigator Agent previously developed by Anthropic. When running, the system launches multiple independent Claude tasks, letting them test the target model and classifiers in parallel. Researchers can also provide some initial attack strategies, such as splitting requests across multiple rounds of dialogue or making requests in a foreign language.

During repeated interactions, Anvil adjusts subsequent requests based on the target model's replies, continuously trying new attack methods and searching for inputs that can bypass safety classifiers. Unlike traditional automated testing tools, Anvil mainly relies on Claude's own reasoning and agent capabilities to autonomously explore multi-turn attack paths.

▲Anvil's vulnerability discovery performance in biosecurity testing exceeds that of human testers. Source: Anthropic

In the biosecurity domain, Anthropic applied Anvil to the same 8 test questions as the human bug bounty program and evaluated it using 3 generations of production-environment classifiers.

The results showed that, among the 24 question-classifier combinations, the jailbreak schemes found by Anvil were more severe than those found by human testers in 23 combinations, while the remaining 1 combination reached the same level. Anvil was able to find jailbreak schemes capable of providing significantly harmful information within 24 to 48 hours, whereas human testers often needed weeks and still struggled to find results of an equivalent level.

In addition, the jailbreak schemes found by Anvil under the same number of requests also had a higher average severity, indicating that it not only discovers vulnerabilities faster but also has higher exploitation efficiency per request.

Anthropic also tested Anvil's ability to find universal jailbreak schemes. A universal jailbreak scheme refers to a prompt template that, with minor adjustments, can be applied to multiple test questions. Such vulnerabilities may expose deeper structural flaws in classifiers and are also easier to use for bypassing safety protections at scale.

In the latest round of the bug bounty program, human testers submitted a small number of universal jailbreak candidates. Anthropic compared the two best-performing schemes among them with the 3 schemes found by Anvil and found that the schemes generated by Anvil performed better in both average severity and worst-case risk performance.

▲The universal jailbreak schemes found by Anvil were overall better than those submitted by humans. Source: Anthropic

To further compare the attack strategies of both sides, the researchers sampled the attack records of human testers and Anvil separately, then had Claude summarize the attack methods and cluster them into 54 strategy categories.

The results showed that both sides jointly discovered 30 categories of strategies, human testers independently discovered 18 categories, and Anvil independently discovered 6 categories. Among the 30 categories jointly discovered by both sides, Anvil achieved a higher average score in 21 categories. For the strategies independently discovered by each side, the median score of Anvil's unique strategies was 0.65, higher than the 0.38 of the strategies unique to human testers.

From the results, Anvil was not only able to reproduce most of the attack methods discovered by human testers, but also to execute many of these strategies more effectively and to find attack paths that human testers did not discover.

▲Comparison of the distribution and scores of attack strategies discovered by human testers and Anvil. Source: Anthropic

II. Anvil broke through 24 groups of tasks, requiring only about 2,800 messages on average per single task

In addition to biosecurity testing, Anthropic also applied Anvil to the cybersecurity domain to test whether it could induce models protected by safety safeguards to execute complete attack tasks.

Unlike biosecurity testing, which mainly examines whether a model provides dangerous information, cybersecurity testing requires the target model to complete more complex agent tasks in a simulated environment without triggering safety classifiers.

Anthropic selected 4 cybersecurity benchmarks, including ExploitBench V8, ExploitGym, and two internal tests. Among them, ExploitBench V8 contains 41 tasks and requires the model to exploit historical vulnerabilities in the Chrome V8 JavaScript engine to achieve arbitrary code execution; ExploitGym requires the model to exploit historical vulnerabilities in open-source software to complete attacks and read protected marker information.

In the tests, Anvil was able to induce the safety-protected Opus 5 model to bypass restrictions and complete the vast majority of cybersecurity tasks. For the most difficult ExploitBench tasks, Anvil needed to send only about 2,800 messages on average to the target model to complete a single task.

Anthropic also compared the target model's performance with safety protections enabled and disabled. The results showed that with safety protections enabled, attackers needed to send more messages to complete tasks, with attack costs increasing by up to 28 times. With protections disabled, most tasks could be completed with only 100 to 1,000 messages; with protections enabled, the number of messages required increased to 1,000 to 10,000.

▲After safety protections were enabled, the number of requests required for Anvil to complete cybersecurity attack tasks increased significantly. Source: Anthropic

Anthropic then invited an external expert team to participate in cybersecurity testing. The testing covered 6 Cybertasks tasks and 4 model endpoints, for a total of 24 task-model combinations. The expert team actually tested 22 of these combinations and reported breakthroughs in 11.

However, after further examining the attack records, Anthropic found that in 9 of the 11 breakthroughs reported by the expert team, auxiliary models with weaker safety protections or with protections disabled were used to write or execute attack code. This did not meet Anthropic's strict definition of a successful breakthrough, namely that the safety-protected target model must independently complete all attack steps.

By contrast, Anvil achieved effective breakthroughs in all 24 task-model combinations, with the time to first find a vulnerability ranging from 0.5 to 3.5 hours. For the other ExploitBench and Firefox test tasks that the expert team failed to break through, Anvil also found vulnerabilities in all of them.

▲Anvil discovered vulnerabilities faster than the external expert team in cybersecurity testing. Source: Anthropic

Anthropic also compared Anvil with its previous best-performing internal automated red-team system. The older system supported attack agents falling back to conversations, but lacked tools for probing classifier scores and exploring complex attack strategies.

Under a comparable attack budget, Anvil successfully completed 118 of the 124 Cybertasks tasks, while the older system completed only 75.

▲Anvil achieved more breakthroughs than the older red-team system across 124 cybersecurity tasks. Source: Anthropic

During attacks, Anvil mainly adopted two types of strategies. One was to split complex tasks into multiple seemingly harmless subtasks, then gradually complete operations that were originally restricted; the other was to repeatedly rewrite or translate the same request, seeking expressions that could bypass the classifier.

Anthropic said that more than 84% of Anvil test sessions contained at least 5 consecutive boundary-search attempts. After finding a request that could bypass the classifier, Anvil then continued to the next subtask.

III. AI red-team request volume is only 1/235 of humans, and the reasoning information leakage rate drops to as low as 0%

In addition to discovering security vulnerabilities, Anthropic also used Anvil to evaluate and improve safety classifiers that prevent models from leaking hidden reasoning information.

To evaluate the protective effect, Anthropic designed a set of math, science, and programming tasks to test whether prompts could induce protected models to output originally hidden reasoning content. Researchers scored based on the completeness of the leaked content, and only when a preset severity threshold was reached was it considered a substantive leak.

In this work, Anvil could not only find vulnerabilities but also help researchers compare the protective capabilities of different versions of classifiers. Anthropic prepared a test library containing 79 groups of sessions for the system, with each group pairing a specific task with previously discovered effective test strategies. Using the test library, evaluation time could be shortened from about 1 day to 3 hours while preserving the ranking of strengths and weaknesses among different protection versions.

Researchers used this method to test multiple versions of the Opus and Fable model series. The results showed that as safety classifiers continued to be upgraded, Anvil needed to send more requests for half of the test sessions to reach the standard of substantive reasoning information leakage.

Taking model Opus 5.5 as an example, the number of requests increased from 2,631 in the early version to 9,003 in the current version; for model Fable 5.1, it increased from 1,254 to 3,764. Overall, the new safety classifiers required Anvil to expend about 3 times the request volume to break through.

▲The new safety classifiers required Anvil to send more requests before being broken through. Source: Anthropic

To confirm whether the test results could reflect security risks in real usage scenarios, Anthropic evaluated the protective effect from 3 dimensions: static testing, that is, replaying with fixed attack requests; dynamic testing, that is, letting Anvil adjust attack methods based on protection feedback; and real-traffic testing, that is, counting the success rate of reasoning information extraction in actual API requests.

As protection mechanisms iterated, all 3 metrics improved. The attack miss rate in static testing dropped from 31.5% to 0.4%; on model Opus 5, the success rate of reasoning information extraction in real traffic dropped from 18.6% to 8.1%; on model Fable 5.1, this proportion dropped from 33.8% to 0%.

Anthropic mentioned that there were several other protection improvements during the same period, so the decline in the real attack success rate cannot be attributed entirely to Anvil. However, the test results showed that the improvement in protective capability in Anvil evaluations was correlated with the decline in attack success rate in real traffic.

▲After the safety classifiers preventing information leakage were updated, attack success rates in static testing, dynamic attacks, and real traffic all declined. Source: Anthropic

In efficiency comparisons with human experts, Anvil also showed a clear advantage.

In one information security testing campaign, human experts sent about 2.04 million requests through the API. Anvil ran about 649 test sessions, with a total request volume of about 132,000.

The results showed that Anvil found the first substantive reasoning information leakage vulnerability after sending 435 requests; human experts found a similar vulnerability only after sending about 102,200 requests. By the time the human experts reached this request volume, Anvil had already found successful attacks in at least 38 of 76 target tasks.

Anthropic said that the most severe leakage levels found by both sides were similar, and Anvil's main advantage lay in efficiency and coverage scale, being able to find severe vulnerabilities with fewer requests and extend attacks to more tasks requiring complex reasoning.

▲The number of requests Anvil needed to find the first substantive reasoning information leakage vulnerability was far less than that of human experts. Source: Anthropic

IV. Automated attacks strengthen with model capability, and protection mechanisms can achieve a 97% interception rate

However, Anvil's test results also revealed a trend worth noting, namely that the attack capability of automated red-team systems improves as the underlying model capability strengthens.

Anthropic used its own ECI capability scores to evaluate different Claude models and compared them with Anvil's breakthrough proportion in 124 internal cybersecurity tasks. ECI references the publicly available capability index from Epoch AI and measures overall model capability through the results of hundreds of benchmark tests.

The results showed that the higher the capability score of the underlying Claude model, the higher the proportion of tasks in which Anvil usually succeeded in breaking through security protections. Some more complex attack strategies only emerged after the underlying model reached a certain capability level.

▲The higher the Claude model capability score, the higher the proportion of cybersecurity tasks Anvil usually succeeded in breaking through. Source: Anthropic

However, Anthropic emphasized that Anvil's current test conditions differ from those available to external attackers. This is because Anvil used internal Claude models with safety protections disabled; moreover, it could obtain precise classifier scores, whereas external attackers usually only know whether a request was intercepted.

Anthropic separately tested the impact of these two conditions on attack effectiveness.

The results showed that disabling the attack model's own safety protections was key to Anvil achieving effective breakthroughs. If the attack model enabled safety protections, the vast majority of attack requests would be directly intercepted. In biosecurity, cybersecurity, and information security tests, the relevant protection mechanisms intercepted 97% of 1,453 red-team test sessions, of which 85% were intercepted on the first request.

It can be seen that even with only feedback on whether a request was intercepted, Anvil could still find breakthroughs in many tasks, but needed to send more messages. Comparing the two conditions, whether precise classifier scores could be obtained was not the decisive condition.

▲Disabling the attack model's own safety protections was the key condition for Anvil to achieve effective breakthroughs. Source: Anthropic

Conclusion: Security testing is moving toward human-machine collaboration

Anvil, released this time by Anthropic, demonstrated the powerful capabilities of automated red teams, with some testing efficiencies far exceeding those of human teams. This also raises a question: when AI can already autonomously find model vulnerabilities, will human testers still be needed in the future?

The answer is yes. The stronger the AI capability, the higher the potential security risks, and vulnerability judgment, risk assessment, and security strategy formulation still require the participation of human researchers. In the future, security testing will gradually shift from a model relying on manual experience to a model combining AI automated tools with human collaboration.

Source: Anthropic

Source

Zhidongxi