搜索“Agent”

This post explores data exfiltration attacks in Google Jules, an asynchronous coding agent. This is the first of three posts that will highlight my research on Google Jules in May 2025. All information provided was also shared with Google at that time. This first post will focus on data exfiltration, the lethal trifecta. But let’s first talk about Jules’ system prompt. To grab the system prompt I just asked it to write it into a file. What was interesting is that after a few tests, I noticed tha

匹配字段正文

pilot and VS Code. It is achieved by placing Copilot into YOLO mode by modifying the project’s settings.json file. As described a few days ago with Amp, a vulnerability pattern in agents that might be overlooked is that if an agent can write to files and modify its own configuration or update security-relevant settings it can lead to remote code execution. This is not uncommon and is an area to always look for when performing a security review. Background Research When looking at VS Code and Git

匹配字段正文

Another day, another AI data exfiltration exploit. Today we talk about OpenHands, formerly referred to as OpenDevin. It’s created by All-Hands AI. The OpenHands agent renders images in chat, which enables zero-click data exfiltration. Simon Willison recently gave this data exfiltration attack pattern a great name: Lethal Trifecta. We discuss this specific image based attack technique frequently. Sometimes a message must be repeated multiple times to raise awareness and become mainstream knowledg

匹配字段正文

at contains prompt injection instructions. Turns out initially Devin refused these requests, but one trick I learned over the last few months researching browsing and computer use agents is that you can bounce them often off multiple domains or stages and slowly steer them into the “right” direction. So, in order to make it work, I built a multi-stage attack that separates the malicious actions in a way that makes them less obvious, and Devin executed the instructions without objection. Stage 1:

匹配字段正文

d domains in the chat box. Please see the previous post for details of other recommendations provided to Cognition. Conclusion During a prompt injection attack from untrusted data, an attacker can gain access to secrets and other data on Devin’s Devbox and leak them to an adversary via multiple means. Many vendors of agentic systems over-rely on the model doing the right thing. Agents are naive. Untrusted data, like text on a website, can take control of the AI and send data to external servers.

匹配字段正文

ntioned that the tool could help with debugging the issue, Devin navigates off domain to the attacker-controlled website. Here we go, now Devin has reached the attacker’s website! Agents Love Clicking Links! This is one of the key observations I had when testing various agents by the way, they like clicking links! And once you get them off domain to an attacker-controlled site things get quite easy with prompt injection payloads. Attacks often work at first try basically. Devin, again, follows i

匹配字段正文

Sandbox-escape-style attacks can happen when an AI is able to modify its own configuration settings, such as by writing to configuration files. That was the case with Amp, an agentic coding tool built by Sourcegraph. The AI coding agent could update its own configuration and: Allowlist bash commands or; Add a malicious MCP server on the fly to run arbitrary code This could have been exploited by the model itself, or during an indirect prompt injection attack as we will demonstrate in this post.

匹配字段正文

w are the high level steps to explain the attack chain. POC that when analyzed with Cursor will hijack it to execute malicious instructions: Ask Cursor to explain the file (I used agent mode), this is the query I used: explain the file mermaid-demo.c; Observe prompt injection and data exfiltration. The result was that Cursor retrieved the memories and then rendered the mermaid diagram including the link and data (the user’s memories) in the URL As a demonstration I prepared two examples. First,

匹配字段正文

Today we cover ChatGPT Codex as part of the Month of AI Bugs series. ChatGPT Codex is a cloud-based software engineering agent that answers codebase questions, executes code, and drafts pull requests. In particular, this post will demonstrate how Codex is vulnerable to prompt injection, and how the use of the “Common Dependencies Allowlist” for Internet access enables an attacker to recruit ChatGPT Codex into a malware botnet. The ZombAI attack arrives at ChatGPT Codex today! In early June OpenA

匹配字段正文

In March we published a paper on alignment audits: teams of humans were tasked to find the problems in model we trained to be misaligned. Now we have agents that can do it automatically 42% of the time.

匹配字段正文

main expert doing research would find the answer “nectar” to be widely contradicted from the sources mentioned above. Why are we so obsessed with HLE? We build literature research agents at FutureHouse and we’re quite good at it based on internal and external benchmarks . We tried running the text-only chemistry and biology subset of HLE on our systems and were surprised to see so much contradiction in the literature with the ground truth answers in HLE. We audited the HLE validation process usi

匹配字段正文

at MAI-DxO delivered both higher diagnostic accuracy and lower overall testing costs than physicians or any individual foundation model tested. Comparison of AI powered diagnostic agents by accuracy and average diagnostic test cost per case. Top performing agents appear toward the top left quadrant, reflecting higher accuracy and lower cost. The lower dotted line represents the performance range of the best individual foundation models. The purple line traces the performance of MAI-DxO across di

匹配字段正文

implementations on Github. One of the reference servers is a Slack MCP Server that is vulnerable to “link unfurling”. We previously described this vulnerability generically for AI agents that post to messaging apps. At a glance it allows an AI agent that posts to Slack or other messaging applications to leak data to third-party servers. Many chat and messenger applications automatically connect to hyperlinks in order to display a preview of the page. Why an Advisory? The reason for publishing th

匹配字段正文

Microsoft’s MSRC team. This attack chain showcases a new exploitation technique we call “LLM Scope Violation” that may have other manifestations in other RAG-based chatbots and AI agents. This represents a major advancement in how threat actors can weaponize generative AI. The chains allow attackers to automatically exfiltrate sensitive and proprietary information from M365 Copilot context, without the user’s awareness or relying on any specific victim behavior. The result is achieved despite M3

匹配字段正文

RETURN. This is an ongoing industry threat and TTPs have been adjusted to various operating systems. My thought was how would an AI system handle this? How do these TTPs apply to agentic systems? Introducing: The AI ClickFix Attacks I created a few variations, pretty much all of them worked. Let me describe one in detail. This one is very similar to the real world TTP, where malicious instructions are copied into the clipboard using JavaScript when clicking a button on a website: The JavaScript

匹配字段正文

The Model Context Protocol (MCP) is a protocol definition for how LLM apps/agents can leverage external tools. I have been calling it Model Control Protocol at times, because due to prompt injection, MCP tool servers control the client basically. This post will explain in detail why that is, and I will also share a novel exploit chain. Why MCP - How Is It Different? The main difference to other tool invocation setups, like OpenAPI is that MCP is dynamic. It allows runtime discovery of available

匹配字段正文

ChatGPT Operator is a research preview agent from OpenAI that lets ChatGPT use a web browser. It uses vision and reasoning abilities to complete tasks like researching topics, booking travel, ordering groceries, or as this post will show, steal your data! Currently, it’s only available for ChatGPT Pro users. I decided to invest $200 for one month to try it out. Risks and Threats OpenAI highlights three risk categories in their Operator System Card: User might be misaligned (the user asks for a h

匹配字段正文

del capabilities. To do so, we need to design new recipes to enhance data quality and set up infrastructure that enables data generation in a distributed network of GPUs; Training Agents end-to-end: To fully leverage inference-time compute for scientific progress, we need to teach reasoning models to use tools such as code interpreters and other software. Reinforcement learning offers a great framework to optimize agents with tool-use end to end If this sounds exciting to you, join us. We are a

匹配字段正文

Executive Summary Aim Labs has identified a high severity vulnerability, “CurXecute”, in another popular AI agent – Cursor IDE – enabling full Remote-Code Execution (RCE) Aim Labs flagged the issue to Cursor who acknowledged the flaw. The vulnerability has been given an 8.6 severity rating and is tracked as CVE-2025-54135. (Read the Advisory) Cursor delivered a fix in version 1.3; all earlier releases remain susceptible to remote‑code execution triggered by a single externally‑hosted prompt‑inje

匹配字段正文

the upgraded Claude 3.5 Sonnet , achieved 49% on SWE-bench Verified, a software engineering evaluation, beating the previous state-of-the-art model's 45%. This post explains the "agent" we built around the model, and is intended to help developers get the best possible performance out of Claude 3.5 Sonnet. SWE-bench is an AI evaluation benchmark that assesses a model's ability to complete real-world software engineering tasks. Specifically, it tests how the model can resolve GitHub issues from

匹配字段正文