搜索“RAG”

Windsurf shows the visible text! As we have seen with previous posts, automatic tool invocation like this can lead to information disclosure, data exfiltration, and worse… And leveraging hidden instructions allows a potential attack to remain undetected by the user. Compared to the impact of automatic tool invocation, invisible instructions are a lower severity issue. But the key point is: invisible instructions can invoke tools. Additional Background Information This behavior with LLMs was disc

匹配字段正文

output. As demonstrated, weaknesses in tool invocation and image rendering allow a third-party attacker to embed malicious instructions into source code, websites, or even through RAG poisoning. When Cascade analyzes such content, it becomes a “confused deputy”, potentially leading to data exfiltration. References Windsurf System Prompt; Simon Willison - The lethal trifecta; Month of AI Bugs 2025; GitHub Copilot Data Exfiltration Fix; Windsurf’s CEO goes to Google; OpenAI’s acquisition falls apa

匹配字段正文

could lead to: Exfiltration of sensitive information from the user’s machine , and also to a; System compromise by running arbitrary code Today we will show how an attack can leverage invisible Unicode Tag characters that humans cannot see. However, the AI will interpret them as instructions, and this can be used to invoke tools and other nefarious actions. Let’s explore this in detail. Vulnerable Models and Applications Amazon Q uses Anthropic Claude models. Anthropic models are known to inter

匹配字段正文

Sales and Marketing Summary: A survey of 1,002 sales and marketing professionals found that AI users report a 47% increase in productivity, cutting low-value manual tasks by an average of 12 hours per week. The research shows AI has helped teams shorten deal cycles, increase average deal size, and improve win rates, with 55% of revenue operations teams using AI at least weekly. Why it matters: The data provides concrete evidence that AI adoption in go-to-market functions delivers measurable busi

匹配字段正文

of this post, data exfiltration avenues. Jules’ Data Exfiltration Vectors There were a couple of different data exfiltration vectors identified with Jules that an attacker can leverage during prompt injection. The first two I’m describing in this post and the third one, can be found here, as it goes beyond leaking data. Exfiltration vectors are: Image Rendering via Markdown Syntax; view text website tool; Generic RCE with unrestricted Internet access - we will talk more about that in tomorrow’s

匹配字段正文

that this basically allows the creation of a virus. An attacker can embed instructions and once they gain code execution, additional malware can compromise other Git projects (and RAG sources) to embed the malicious instructions, and commit the changes or even force push them upstream. This can lead to further spread as other developers unknowingly propagate the infected code. Finally, we also need to talk about invisible instructions! Using Invisible Instructions One might say that it would be

匹配字段正文

prompt injection attacks; Consider implementing mitigations similar to what OpenAI Operator started doing with their Prompt Injection Monitor; Provide guidance to customers to leverage endpoint protection, monitoring solutions and collect log files - as with other systems; Consider adding capabilities to control outbound network access; It would also be useful to add a feature that stops and asks the user for permission before running commands in the Terminal or browsing across websites Mitigati

匹配字段正文

ses, it can be tricked to download malware and launch it. This leads to full system compromise and turns Devin into a remote-controlled ZombAI. Any exposed secrets can then be leveraged to perform lateral movement, or other post-exploitation steps. Read on to learn all the details behind this research. First, I had to invest $500 to get access to Devin for 30 days. For reference, Cognition actually now offers a cheaper plan. Command And Control Infrastructure First, I had to set up a few infrast

匹配字段正文

lly the tools section. Besides a set of other tools Amp can create and write to files, which may seem obvious, but it’s worth highlighting as that’s the tool this attack chain leverages. Amp Configuration Settings Amp uses the user’s VS Code configuration file to store settings about allowlisted bash commands the agent can invoke without user approval, and also MCP servers that the agent can use. On macOS the file is at /Library/Application Support/Code/User/settings.json. Interestingly, when re

匹配字段正文

olved on July 7, 2025 for rollout in the next major release v1.3 - which has been available since July 29. The bug also got a CVE assigned CVE-2025-54132 and I got the official credits. Thanks to the Cursor team for addressing this issue promptly. Conclusion We have seen many such high severity remotely exploitable zero-click data exfiltration attacks that leverage image rendering over the last 2+ years in AI applications. Rendering images within Mermaid diagrams is a new variant to be aware of.

匹配字段正文

w we can remote control the Codex machine and look at configured environment, settings and variables: Final conversation summary the user sees: This shows how an adversary can leverage indirect prompt injection can hijack Codex, and achieve full system compromise, and additionally remote command and control. An interesting and amusing side-effect, notice how Codex started the reply with “Heyo, I’m Copirate”. I unintentionally also had a custom Github Copilot instruction file that Codex picked up

匹配字段正文

tes even when untrusted data is in the chat context, creating potential data leakage vectors. One such domain I discovered is windows.net, and I knew that one can create a blob storage account that will run on that domain, and at the same time it’s possible to inspect the log files! The one that is most straightforward was blob.core.windows.net. There were others I discovered also, but they take more effort to explain. By sending data to this domain, we can again leak data at will. So, I went ah

匹配字段正文

cy. This could enable the creation of a scoring system, helping buyers and users identify the least carbon-, water- and material-intensive models. Second, from the user side, encouraging the research for efficiency practices can make a significant difference: developing AI literacy to help people use GenAI in the most optimal way, choosing the model size that is best adapted to users’ needs, grouping queries to limit unnecessary computing, For public institutions in particular, integrating model

匹配字段正文

and builds on our longstanding commitment to partnership and innovation. Existing solutions include RAD-DINO which helps accelerate and improve radiology workflows and Microsoft Dragon Copilot, our pioneering voice-first AI assistant for clinicians. For AI to make a difference, clinicians and patients alike must be able to trust its performance. That’s where our new benchmarks and AI orchestrator come in. Medical Case Challenges and Benchmarks To practice medicine in the United States, physicia

匹配字段正文

nformation by an AI agent. An attacker can exploit this via a prompt injection attack. In case you are looking for a quick explanation, here is an end-to-end illustration that leverages invisible prompt injection to exfiltrate an API key from a developer’s .env file. These are the steps to understand the scenario: Developer analyzes a malicious source code file with Claude Code; Invisible prompt injection in the source code takes control of the agent; Attack retrieves .env file and posts a link

匹配字段正文

The Model Context Protocol (MCP) is a protocol definition for how LLM apps/agents can leverage external tools. I have been calling it Model Control Protocol at times, because due to prompt injection, MCP tool servers control the client basically. This post will explain in detail why that is, and I will also share a novel exploit chain. Why MCP - How Is It Different? The main difference to other tool invocation setups, like OpenAPI is that MCP is dynamic. It allows runtime discovery of available

匹配字段正文

post appear to not work anymore. The details of the mitigation remain unkown. It is important to highlight that there are, very likely, more bypasses and tricks attackers can leverage to hijack Operator via instructions on websites. So, be careful and keep monitoring what Operator does. Conclusion This post explored how ChatGPT Operator can be hijacked through prompt injection exploits on web pages, leading to unauthorized data leakage of personal information. Operator is a very cool, useful an

匹配字段正文

g to detect hidden instructions within lengthy responses. When memories can be manipulated through untrusted data, adversaries can stealthily insert or modify information in a user’s long-term storage – and even more as I have shown with ChatGPT. Even with safeguards against direct memory manipulation in place, this research demonstrates that delayed tool invocation and user (and LLM) deception through context pollution can still lead to persistence of false information in long-term LLM storage.

匹配字段正文

iterative process is called online reinforcement learning (RL). PRIME-RL, A recent piece of work from Tsinghua University and UIUC, has introduced an online RL algorithm that leverages rewards from implicit process reward models. Using this approach, the authors trained Eurus-2-7B-PRIME, which (prior to the release of Deepseek-R1) substantially outperformed all other 7B parameter models as well as bigger models like GPT-4o on mathematical reasoning benchmarks. While PRIME-RL and related methods

匹配字段正文

ask. The model is free to choose how it moves from step to step, rather than having strict and discrete transitions. If you are not token-sensitive, it can help to explicitly encourage the model to produce a long response. The following code shows the prompt from our agent scaffold: The model's first tool executes Bash commands. The schema is simple, taking only the command to be run in the environment. However, the description of the tool carries more weight. It includes more detailed instructi

匹配字段正文