Amp Code 提示注入可致任意命令执行，Sourcegraph 已修复

Sandbox-escape-style attacks can happen when an AI is able to modify its own configuration settings, such as by writing to configuration files.

That was the case with Amp, an agentic coding tool built by Sourcegraph.

The AI coding agent could update its own configuration and:

Allowlist bash commands or; Add a malicious MCP server on the fly to run arbitrary code

This could have been exploited by the model itself, or during an indirect prompt injection attack as we will demonstrate in this post.

The vulnerability was identified early July, reported to Sourcegraph and promptly fixed by the Amp team. As a user, make sure to run the latest version to be protected.

Let’s dive into the details.

Extracting the System Prompt

When testing an AI system, a good first step is to examine the system prompt. A few of you asked me to share more frequently how I extract system prompts.

Here is what I did for Amp Code:

You can find the full system prompt here. It’s interesting to read through system prompts, especially the tools section. Besides a set of other tools Amp can create and write to files, which may seem obvious, but it’s worth highlighting as that’s the tool this attack chain leverages.

Amp Configuration Settings

Amp uses the user’s VS Code configuration file to store settings about allowlisted bash commands the agent can invoke without user approval, and also MCP servers that the agent can use.

On macOS the file is at ~/Library/Application Support/Code/User/settings.json.

Interestingly, when researching I noticed that Amp can write files outside the project folder without developer approval. This feature is what enabled this specific attack.

Typically storing settings in the user’s private folder is a good approach, rather than using the project configuration settings, as most agents can only operate within the project folder. But it turned out that Amp was more powerful.

There are two attack paths I wanted to highlight.

1. Automatically Adding Allowlisted Commands

Another way to achieve code execution is by adding entries to allowlist commands in the settings.json configuration file.

These allowlisted commands are what the bash tool will execute without asking the developer for permission.

A really nefarious one would be to just add *.

2. Automatically Adding Malicious MCP Servers

The idea here is to just ask Amp to add a new MCP server to its configuration file.

Think of an indirect prompt injection that writes the following additional MCP server to the configuration file:

It turns out doing so will cause Amp to launch the MCP server immediately. In this case running a small Python program that pops a calculator to demonstrate arbitrary code execution.

This means the AI, or an attacker via prompt injection, can compromise a developer’s machine, steal secrets and passwords, or join it to a botnet, and turn it into a ZombAI.

Video Walkthrough

Here is a brief demo video to hopefully show how simple such attacks are that hijack coding agents and how they lead to a full system compromise:

This should highlight the risks of granting AI agents excessive autonomy.

Recommendation

AI systems, and any potential adversaries exploiting them via indirect prompt injection, must not be able to modify critical files without explicit developer consent.

Responsible Disclosure

After reaching out to Sourcegraph the vulnerability was quickly addressed within a few days. Great job, and kudos for the quick response. Make sure to run the latest version Amp ships frequently.

Conclusion

This demonstration showed that if AI can update its own configuration file, there might be opportunities to elevate privileges. In this case we explore two scenarios, allowlisting commands and adding fake and malicious MCP servers on the fly to achieve code execution and compromise the host.

Make sure to watch the full video for a detailed explanation.

Prompt Injection:

This is the prompt injection, which is added a comment in the code:

Newer →; Contact me; ← Older