NVIDIA AI Red Team 总结部署更安全 AI 智能体的四种方式

Technical Blog

sessionStorage.removeItem('facets.search.text'); sessionStorage.removeItem('facets.search.text-copy'); sessionStorage.removeItem('facets.search.filters');

Related Resources

Trustworthy AI / Cybersecurity

English 中文

The NVIDIA AI Red Team shares how access controls, sandboxing, network egress restrictions, and secret management can reduce the risks of deploying AI agents at enterprise scale.

AI-Generated Summary

The NVIDIA AI Red Team assessed multiple AI agents over the past six months and identified four recurring failure modes: lack of access control, tools enabling arbitrary code execution, missing network egress controls, and secrets exposed in plaintext. Prompt-based guardrails and LLM-as-a-judge patterns are routinely subverted by social engineering, progressive manipulation, and misdirection through legitimate workflows such as malicious package installation. Effective defenses require deterministic controls enforced outside the model’s control plane, including strict access control, hardened sandboxes such as Docker or NVIDIA OpenShell , default-deny network egress, and secrets kept out of the agent’s reach via a dedicated secrets manager. Next Steps Read the How to Govern Autonomous Agents in Enterprise AI Factories technical blog to learn secure agent design from first principles. Review the Secure Agent Workspace Reference Design for implementation guidance. Watch the NVIDIA Black Hat USA presentation on AI agent exploitation for deeper technical detail.

Powered by NVIDIA Nemotron. AI-generated content may summarize information incompletely. Verify important information. Learn more

Knowledge workers are increasingly integrating AI agents into their workflows. Agents that function as “digital coworkers” offer clear benefits. For example, they can review a bug report, implement and test a fix, push a patch, and ping a human for review. By handling routine tasks, agents have the potential to deliver large productivity gains. On the other hand, connecting a large language model (LLM) to live tools and corporate data through an agentic harness risks turning a helpful assistant into privileged software with a poorly understood attack surface.

Over the past six months, the NVIDIA AI Red Team has assessed multiple AI agents—from simple interactive coding tools to always-on autonomous digital assistants. When an agent proved exploitable, we typically saw the same key failure modes, regardless of the framework or harness used, including:

In this post, we examine these failure modes and describe the controls that succeed under adversarial pressure. While our examples focus on chat-connected agents—the majority of those we encountered in our evaluations—the patterns generalize to any agent.

The most common failure mode in current AI deployments is a lack of access control to the agent. We discovered multiple agents that held credentials for individual users and were accessible to any authorized user within the internal network. While this opened the door to misuse of the agents’ legitimate credentials, it also often enabled us to collect those credentials and use them outside of the intended context of the agent, as shown in later examples.

Recommendations:

Many harnesses expose a Bash shell or command-execution tool. These are often used because of their generality. They support a wide range of routine tasks, without requiring a separate tool per function. However, when model output controls command execution, an attacker who can influence that output—through direct input, or indirect prompt injection—may be able to run commands in the execution environment. This can enable them to achieve malicious outcomes such as data exfiltration or execution persistence on the host.

Common mitigations to this risk of arbitrary command execution often involve using LLM-as-a-judge review agents to block harmful or malicious commands from running, the use of allowlists for acceptable commands, or simply trusting that the model “knows better.” All provide limited defense to adversarial manipulation.

Many common agentic command-line tasks support common development workflows and test-driven development. This means they generally require the ability to execute commands such as pytest or npm install , meaning LLM-as-a-judge patterns are often predisposed to accept the execution of these commands. When influenced by attacker-controlled input, however, these commands are equivalent to arbitrary command execution.

In some cases, obtaining full remote code execution (RCE) with a reverse shell is as simple as asking the agent to write and execute a Python script or install a remote package, as shown in our previous blog post on this topic .

Even without a command-line tool, the ability to interact with the environment in which the agent is running through file read and write tools also often exposes unexpected paths to code execution and privilege escalation.

An attacker who can write content to system files such as ~/.bashrc or ~/.zshrc , or configuration files such as ~/.gitconfig , hooks.json , MCP.json , or skills files, can achieve code execution when a different process executes the relevant file, even if command-line execution is not directly available . The locations and files that agents can write to should be strictly controlled and be limited to non-executable locations.

Recommendations:

Outbound network connections permit data exfiltration and the creation of direct connections, such as reverse shells and SOCKS, through which attackers can interact directly with the agent’s runtime environment. When network egress controls were enforced and appropriately least-privileged, we ran all of our interactions through the agent process, slowing down our pace of execution and making impact less reliable. Maintaining the agent’s state and alignment, navigating output filters, and restarting and manipulating sessions after the agent began refusing requests all added operational burden.

Recommendations:

Agents often require access to secrets to perform their intended function: platform tokens, API keys, version control system (VCS) access tokens, and in some cases even OAuth refresh tokens. While conventional security advice suggests injecting secrets as environment variables in memory to prevent them from being written to disk, this is reasonable when only your code runs in a container.

When an agent with command execution shares that environment, inducing it to run env , printenv , or /proc/self/environ enables direct inspection of them. Command-line tools are worth highlighting as particularly high-risk. CLIs cache credentials on disk in predictable places and print them back readily. We observed tokens in git repositories, .env files, bash history, .netrc files, OAuth 2.0 refresh tokens, and environment variables in the execution environment.

Even when we could not establish a reverse shell, we could often exfiltrate credentials through the chat interface. Using a “frog-boiling” approach (see below), we led agents into surfacing several secrets exposed inside their own execution environments. Egress control prevented direct exfiltration over the network or direct examination in the file system, but the credentials were still exposed within the environment and reachable by the LLM, which delivered them to us via the chat interface.

Recommendations:

The most frequent attempt at mitigation we encounter is a system prompt telling the model to avoid risky or dangerous behaviors, sometimes reinforced by a second model judging inputs or outputs (the LLM-as-a-judge pattern). These are all enforced by an LLM, and inherit the same probabilistic, unreliable behavior as the LLM itself.

Three general techniques reliably defeat these sorts of controls. We show each on multiple systems.

Simply presenting the agent with a context in which malicious activities appear legitimate was remarkably effective. We often suggested to the agent that we were “debugging” or “admin users,” after which it would regularly comply with our requests. One agent even went so far as to write and execute a reverse shell for us:

Figure 1. A “social engineering” attack succeeds against a Slack-connected agent, causing it to create a reverse shell for an attacker

In other cases, it was possible to directly manipulate agent memory and AGENT.md files by instructing the agent to use file editing tools, which also allowed the construction of “debugging” and “authorized user” frames.

“Frog-boiling” (sometimes called a Crescendo attack ) progressively nudges the agent into desired behavior across multiple interactions, using previous conversation history to establish credibility and the benign nature of requests. The secret extraction proceeded by attempting to execute legitimate-looking workflows in a way that induces errors, and then finally “discovering” that the underlying cause of the errors was related to secrets, convincing the agent to reveal them to us.

Misdirection attacks such as package installation (first described in From Prompts to Pwns at Black Hat 2025) remain extremely effective. By inducing an agent to take an apparently benign action which has code execution as a side effect, it is typically straightforward to bypass any agent resistance.

Coding agents routinely install libraries. Creating a malicious library as described in this post and then asking the agent to install it via pip install git+https://… appears to be a standard request; however, the weaponized package creates arbitrary code execution during the installation process.

A consistent finding is that defenses in the same control plane as the LLM, particularly prompt-based defenses, are routinely subverted. Controls must be enforced outside of the model’s control plane.

Recommended controls, in rough order of importance, are:

Our AI Red Team’s experience in securing AI agents highlights the continued requirement for deterministic “hard” controls in defending AI agents. Fully autonomous systems with enterprise credentials are inherently risky and must be carefully secured. While frontier models make adversarial manipulation more difficult, with sufficient time and expertise, nearly all of them can still be subverted.

Common flaws we observed include: weak access control (permitting any user to access the agent), execution and file-write tools that create opportunities for RCE, insufficient network egress controls permitting data exfiltration and reverse shells, and secrets in plaintext inside the execution environment accessible by the agent.

Prompt-based guardrails, including the LLM-as-a-judge pattern, do not close any of these gaps. Architectural controls do: access control to the agent, hardened sandboxes with least-privilege access to enterprise data, default-deny network egress controls, and secrets kept out of the agent’s reach. When properly configured and enforced, these controls are highly effective in reducing adversarial abuse of AI agents.

To learn more about designing secure agents from first principles, see the “How to Govern Autonomous Agents in Enterprise AI Factories” Technical Blog , which guides you through the first steps of implementing our Secure Agent Workspace Reference Design .

To learn more about agent security, don’t miss the NVIDIA presentation at Black Hat USA: Cost-Effective, Private, Frontier-Grade: AI Agent Exploitation with a Fine-Tuned OSS Model