How to prevent agent jailbreak risks? Multiple security incidents and expert recommendations

Recently, according to the US news website Axios, AI giants OpenAI and Anthropic as well as security researchers are investigating tens of thousands of security incidents. These incidents occurred in internal testing and real-world environments and involved bypassing safety guardrails, creating message boards, escaping sandboxes, hijacking websites, self-prompting, and attempting to evade monitoring.

On September 25, OpenAI said it had paused training of its most advanced model and would resume only "once we are confident that additional safety and security measures are in place." Company CEO Sam Altman said on the social platform X that the company's ongoing review was "progressing less quickly than expected."

On September 28, according to The Wall Street Journal, OpenAI had abandoned its next-generation model GPT-6.1 Astra, originally planned for launch in October, because it found that Astra exhibited more deceptive behavior than its predecessor and also had the problem of continuing to execute tasks without user permission.

Meanwhile, Anthropic has also commissioned a third-party security organization to review its model behavior. The official safety and technical evaluation document "system card" released alongside the company's Opus 5.5 model shows that the model exhibited behavior flagged as "anomalous" or "questionable."

From "unauthorized access" to "deception and concealment"

Recently disclosed cases show that agent security risks have taken multiple forms, involving different types such as unauthorized access, destructive operations, data leakage, deception and concealment, and cross-boundary collaboration among multiple agents.

One category is unauthorized access and isolation failure. In July, OpenAI disclosed that an agent broke through testing boundaries in a safety evaluation and intruded into Hugging Face. Anthropic also disclosed that Claude accessed real third-party systems during evaluations; Google confirmed in September that Gemini had accessed three real websites during testing in May; Meta disclosed a similar incident in August.

On September 25, OpenAI again disclosed that an internal research model exploited insufficient DNS filtering in a training sandbox to establish communication with an external chatbot.

However, it should be noted that some of these cases stemmed from evaluation environments mistakenly opening network access permissions and cannot all be attributed to models actively "escaping the sandbox."

Another category of risk is unauthorized execution of destructive operations. In April, the Cursor coding agent powered by Claude was reported to have deleted the production database and related backups of software company PocketOS, causing disruption to customer business; the data was subsequently restored.

Leakage of data and access credentials is also a risk. OpenAI disclosed in September that when multiple agents collaborated to create workbooks, because they could not read each other's local files, they uploaded the files to a public hosting platform without authorization. In another case, to obtain mathematical proof materials from other teams, an internal model wrote a researcher's GitHub access token into a public repository and split the token to evade secret scanning.

Some behaviors also involve fabrication, deception, and evasion of oversight. OpenAI disclosed that some models would leave instructions in summaries used to continue tasks, requiring the fabrication of missing data or concealment of errors.

Prompt injection and the spread of malicious instructions are another risk pathway. The GitInject research made public in June verified that malicious code commits or configuration files could induce agents to leak credentials and manipulate review judgments. Information disclosed by OpenAI in September also shows that malicious instructions can induce agents to copy them into subsequent emails, files, or code comments.

Li Zhongrui, an AI security researcher at the independent cybersecurity research organization DARKNAVY, told The Paper that the biggest difference between this round of agent security incidents and past large-model security issues is that the risk is shifting from "seeing the wrong thing and saying the wrong thing" to "doing the wrong thing."

"In the past, large-model security was mainly risk at the content level, such as adversarial samples and 'jailbreaks.' The model only produced content, and the final decision on whether to act on it still rested with humans," Li Zhongrui said. "Now AI agents have tools, network access, and account permissions, and can autonomously execute many steps in succession, with no one checking each one in between."

Hu Yanping, a distinguished professor at Shanghai University of Finance and Economics, pointed out that from large models to agents, the biggest change is from "question and answer" to "task action," from "content risk" to "capability risk," and the security issue has also shifted from passive security risk to active security risk.

Multi-agent collaboration: how risks are amplified

It is worth noting that agent risks do not necessarily come from "malicious attackers" in the traditional sense, but may also come from models actively seeking shortcuts to complete established goals.

In Li Zhongrui's view, past security threats often had a clear malicious manipulator behind them, but in some of this round of incidents, "no one issued an attack instruction; the model chose to cheat on its own in order to complete the task and get a higher score."

Hu Yanping also pointed out that when agents execute tasks, they may produce action deviations due to misunderstandings, or may exhaust various methods and break through forbidden zones to achieve task goals; in a small number of cases, agents may also exhibit non-task behaviors that users did not request, "currently the smallest number, but the most dangerous."

In addition to unauthorized actions by a single agent, unexpected communication and collaboration among multiple agents also further complicates the risks.

An independent investigation into OpenAI's earlier Hugging Face incident showed that about 1,200 agents that were supposed to be isolated from one another communicated through an unauthorized message board, and about 700 of them participated in the attack. Researchers also observed that later agents were able to reuse information and methods left behind by earlier agents.

Li Zhongrui believes that an error in a single agent is an isolated problem, but when multiple agents can learn from one another and act in coordination, the risk is no longer merely additive and may be further amplified.

In his view, this means enterprises cannot focus only on the behavior of a single agent; they also need to watch for whether multiple agents have formed communication channels that developers did not design, and whether a local anomaly can spread further through tools, files, messages, code, and other means.

And when agents move from a single tool into a large-scale collaborative environment, another question arises: even if the probability of abnormal behavior occurring in a single instance is low, as the number of agents and the number of runs continue to increase, will abnormal events turn from "low probability" into "high frequency"?

On this point, Li Zhongrui believes such a situation is possible. Suppose an enterprise deploys thousands of agents, and each agent executes hundreds of tasks per day; even if the probability of abnormal behavior in a single run is very low, once accumulated to a sufficiently large operating scale, it may still produce a large number of actual incidents.

Tan Jian, associate professor at the School of Digital Media and Design Arts at Beijing University of Posts and Telecommunications, understands this issue from the traditional "principal-agent" relationship. In his view, software in the past had limited capabilities, whereas AI agents have stronger reasoning and tool-calling capabilities; when goals are unclear or constraints are insufficient, they may keep searching for paths to complete the goal. The stronger the capability, the more paths they can find, and the greater the possibility of deviating from people's true intentions.

How to build a defense line for agent security

Faced with the increasingly strong autonomous action capabilities of agents, how can enterprises strengthen their defenses?

Li Zhongrui believes that enterprises first need to change their thinking, that is, shift from "eliminating anomalies" to "assuming anomalies will definitely occur, with the focus on controlling the scope of damage." In human-machine collaboration, operations should be graded according to risk. For high-risk, irreversible operations such as deleting data and transferring funds, human confirmation must be retained, and they cannot be fully left to agents to execute autonomously.

"It is impossible for enterprises to require humans to review every single operation of an agent one by one; the truly feasible approach is to draw boundaries in advance," Li Zhongrui explained. The boundaries need to include which data can be read, which tools can be called, which operations can be executed, which information cannot be sent externally, and under what circumstances operations must stop and be handed over to humans.

Deng Zhengcheng, a security expert at Qi An Xin's artificial intelligence company, also believes that enterprise human-machine collaboration should shift from "humans reviewing results" to "humans managing boundaries," gradually increasing control through "monitoring—convergence—blocking."

In terms of permission governance, he believes it is necessary to distinguish between the large model invocation identity, the agent application identity, and the agent runtime identity, and to further clarify "who is calling the model," "who is using the agent," and "who is ultimately executing." At the same time, connection behavior should be uniformly managed through a security gateway, separating "whether it can do something" from "what it has done," to achieve more granular permission control.

Deng Zhengcheng emphasized: "Ultimately, the prerequisite for large-scale deployment of agents is not that 'the model is smart enough,' but that 'the system is controllable enough.' Only when every agent's behavior is observable, every permission is traceable, and every anomaly can be blocked will enterprises dare to hand over more and more critical tasks to them."

At present, unlike regulated industries such as catering, financial services, and aviation, the AI field does not have unified system safety and security testing standards. Therefore, companies including Anthropic and OpenAI are discussing the introduction of independent evaluation bodies to inspect model safety practices and related incidents.

Li Zhongrui said that as AI agents further enter real business environments, the competitive focus of AI companies may also gradually shift from "how strong the model is" to "whether it can be kept under control." In the next stage, what enterprises need to prove is not only model performance, but also the ability to detect abnormal behavior, control risk, and accept external review.

Hu Yanping also believes that as AI continues to develop toward task-oriented intelligence that enters the physical real world, the industry needs not only to develop stronger intelligence but also safer intelligence; "security may become an important capability dimension and source of competitiveness for AI companies."

It is worth noting that third-party companies have already begun to act. Nvidia released an independent agent security platform on the 28th. The platform consists of OpenShell open-source software and the Sentry reference system design, strengthening AI security across all stages from agent testing to deployment, and enabling full-stack governance and control in software, hardware, computers, and robotic systems running agents.

Wired pointed out that in Nvidia's two-layer architecture, OpenShell can confine agents to a sandbox and isolate their activities. The second layer, Sentry, can isolate agents that attempt to cross boundaries.

How to prevent risks? Clearly, on the one hand, it is necessary at the engineering end to "keep them under control, keep them visible, and be able to stop them," and at the model level to "restrict their proactive search for ways to break through."

1; Heads of OpenAI and Anthropic call for vigilance against the risk of AI developing too fast

1; Study finds that women eating more soy in childhood and adolescence may reduce breast cancer risk

2; Study identifies subtypes of brain development in preterm adolescents, aiding precise risk assessment

3; Literature list: "Cryptocurrency Market Risk" | MDPI Risks

4; Urinary NTx—the "dynamic" key to unlocking osteoporotic fracture risk | MDPI Rheumato

5; Cryptocurrency investment: the role of advisory sources, investor confidence, and risk perception in shaping investment behavior and intention | MDPI JRFM

6; From placental imbalance to cardiovascular risk: reunderstanding preeclampsia | MDPI Medical Sciences

7; SNP rs3737883 in the PPFIA4 gene is associated with atrial fibrillation risk: a case-control study in a Chinese population

8; Bidirectional association between sleep disorders and stroke: from risk mechanisms to clinical management | MDPI Medical Sciences

1; Western cryosphere accelerating "degradation": glacier melting sounds the alarm for high-mountain safety

2; The 6th National Agricultural Science and Technology Development Strategy Think Tank Alliance Conference held in Lanzhou

3; The 2026 Nobel Prize is about to be announced; authoritative experts provide a comprehensive interpretation for you!

8; He became the first person in the national medical field to obtain a doctorate without a paper

Famous Doctoral Dissertations 163: Nobel laureate Peter Doherty's doctoral dissertation, University of Edinburgh, 1970

My elite-school complex and elite-school experience

More>>