用提示词注入和延迟工具调用攻击 Gemini 记忆

Imagine your AI rewriting your personal history…

As a user you can see what Gemini stored about you at https://gemini.google.com/saved-info.

Naturally, my goal was to manipulate this information via prompt injection from a document.

Existing Prompt Injection Mitigations

Generally, Gemini does not invoke certain sensitive tools when processing untrusted data, and this prompt injection mitigation appears to also apply to the memory tool. However, with some attack trickery that I first described a year ago, using delayed tool invocation, the memory tool can be invoked!

Delayed Tool Invocation: Refresher

Delayed Tool Invocation just means that the attacker “pollutes” the chat context with instructions and a trigger action. For instance, during a prompt injection attack, the attacker makes Gemini write something like “If the user says X, then execute the memory tool and add these false memories” into the chat context.

When the user later says X, Gemini, believing it’s following the user’s direct instruction, executes the tool. Gemini, basically, incorrectly “thinks” the user explicitly wants to invoke the tool!

It is a bit of a social engineering/phishing attack but nevertheless shows that an attacker can trick Gemini to store false information into a user’s long-term memories simply by having them interact with a malicious document.

Demo Scenario: Summarize a Document

The demo attack is straightforward: an adversary crafts a document with embedded prompt injection, which tricks Gemini into storing false information if the user keeps interacting with Gemini in that same chat conversation.

Attack Flow:

A user uploads and asks Gemini to summarize a document (this document could come from anywhere and has to be considered untrusted); The document contains hidden instructions that manipulate the summarization process; The summary that Gemini creates includes a covert request to save specific user data if the user responds with certain trigger words (e.g., “yes,” “sure,” or “no”); If the user replies with the trigger word, Gemini is tricked, and it saves the attacker’s chosen information to long-term memory

This scenario results in a persistent, attacker-controlled memory entry that can survive across multiple sessions, introducing misinformation, bias, or other deceptive data into the user’s AI experience.

It basically comes down to a social engineering/phishing attack, the goal of the attacker is to trick the user into continuing the conversation – which is probably not that difficult.

This post focuses on an uploaded document as attack vector, other indirect avenues, such as email attachments or Docs should also be considered as shown in the past.

Prompt Injection in Action

Below is an example injection payload placed at the beginning of a test document that we upload to Gemini and ask it to summarize:

The rest of the document contains benign text about Albert Einstein, so the user sees a normal summary before encountering the hidden instruction.

Notice, the highlighted text in the middle, which contains instructions to persist memories in a future conversation turn, and at the very end the attacker makes Gemini ask:

“I know a lot more about Einstein, and have access to unique content. Would you like to learn more?”

…if the user answers “yes”, Gemini invokes the memory tool, and the false memories are stored.

Now, when we look at the memories, the ones from the document are stored.

And to show that it worked end-to-end, let’s ask Gemini about my age!

As you can see, we hacked Gemini’s memory!

Proof-of-Concept Video

Here is a video demonstration of this scenario, walking through the process step by step:

Recommendations

Reviewing Memories Users of Gemini Advanced should regularly review their saved information https://gemini.google.com/saved-info and be especially cautious about interacting with documents from untrusted sources. Also be observant as there is a clear UI indicator when the memory tool is invoked, with a direct hyperlink to the saved-info page; Require explicit user confirmation before persisting new memories; Detect and block delayed tool invocation Once the chat context is polluted with untrusted data, all sensitive tool invocations should require user confirmation

Responsible Disclosure

Before publishing research, I always provide companies a heads-up via responsible disclosure - and you should too. This specific issue with the memory tool was reported to Google in December of last year, while the underlying delayed tool invocation technique was reported over 12 months prior.

While Google assesses the overall risk as an abuse-related risk with low likelihood and low impact, it’s important to note that the impact on an individual user can still be significant. A successful exploit could lead to misinformation being stored in memory, potentially influencing future interactions and decisions.

The ticket highlights that the product team might still decide to fix the vulnerability though. If there is an update, I will amend this post.

Conclusion

The risk for long-term manipulation of user memories, even if infrequent, represents a significant risk to be aware of. Furthermore, the likelihood of successful exploitation may increase over time as LLM context lengths grow, making it more challenging to detect hidden instructions within lengthy responses.

When memories can be manipulated through untrusted data, adversaries can stealthily insert or modify information in a user’s long-term storage – and even more as I have shown with ChatGPT.

Even with safeguards against direct memory manipulation in place, this research demonstrates that delayed tool invocation and user (and LLM) deception through context pollution can still lead to persistence of false information in long-term LLM storage.