Shanghai AI Laboratory Releases PrivacyTrace Benchmark to Quantify Privacy Leakage in Agent Tool-Calling Traces

In the AI era, the most private conversation may not be the sentence you send, but the tool call that the Agent makes on your behalf.

You ask the AI assistant to look up a route to London next week, and it casually calls a map service; you ask it to organize your medical examination report, and it connects to a health database. In theory, only you and the model vendor can see the content of your conversation with it. But a recent study released by the AgentDoG team at Shanghai AI Laboratory reminds us that even if the entire conversation remains confidential, the "homework traces" left behind when an AI Agent calls tools in the background are enough for an observer to piece together where you live, what religion you believe in, and how your health is, sketching a fairly complete profile—though your name cannot be guessed.

This study, titled "How Much Do Tool Calls Expose You? Privacy Leakage from Tool-Call Traces," was completed by Shanghai AI Laboratory in collaboration with the University of Electronic Science and Technology of China and the University of Sydney. It builds a benchmark called PrivacyTrace and systematically quantifies this previously overlooked risk. The conclusion is not mystical but a set of reproducible numbers.

Some may ask: the AI world talks about privacy every day, so what is new about this study? What is new is the attack surface. Past security research focused on two types of problems: first, whether an Agent proactively requests data it should not obtain; second, whether malicious tools trick an Agent into leaking secrets. In this study, however, all tools behave properly, protocol calls are completely legitimate, and no party "does evil," yet the risk still exists. It is the latest version of a familiar script in the history of digital privacy: twenty years ago, an anonymized search log allowed users to be identified as their real selves based solely on search keywords; more than a decade ago, likes on Facebook exposed sexual orientation and political leanings; now it is the Agent's tool calls' turn.

A "mind reading" without conversation

First, let us clarify the attacker's position. In the mainstream architecture of large-model Agents, the model calls external tools through standard interfaces such as MCP (Model Context Protocol): maps, calendars, health services, shopping platforms. Each call generates a record containing the tool name, the parameters passed in, the returned results, and the call order. This is the so-called "tool-call traces."

The key point is: the tool server cannot see your conversation with the Agent, but it can fully see these call records. And a gateway or aggregation platform sitting in the middle can even see all the traces of all your tasks. The researchers call these two types of observers "local observers" and "global observers." These are semi-honest observers: they dutifully complete their own work, but just happen to "take an extra look" at data they already have permission to see. This is precisely the hardest type to guard against, because it violates no rules.

Figure 1: Tool-call traces as a privacy side channel. Geocoding results, travel time, and patterns in modes of transportation may support inferences about a user's location, health status, and religious beliefs (Source: Figure 1 of the paper).

For example. You ask the Agent to plan next week's commute and errands, and the traces will contain the geocoding results of destinations, travel time, and modes of transportation. Each item looks unremarkable on its own, but when these clues are combined: the destination falls in the area of a certain London hospital, travel strictly avoids the Sabbath, and from Friday sunset to Saturday night only walking is used—an observing server can infer: you may live in London, have chronic kidney disease, and believe in Judaism. You said not a single word, but your behavior spoke for you.

PrivacyTrace: giving a thousand virtual users a "CT scan"

To quantify this risk, the biggest obstacle is the inability to obtain real private data. No one is willing to hand over their medical records to researchers for attack experiments. The team's approach was synthesis: referring to the demographic distributions of institutions such as the United Nations, the World Bank, UNESCO, and the International Labour Organization, they used iterative proportional fitting (IPF) to generate 1,000 virtual users statistically close to the real population structure. Each user has 17 attributes, including nationality, age, occupation, religious belief, physical and mental health, and income.

Next, around four domains—travel, health, shopping, and career learning—they executed tasks using the real tools of 25 real MCP servers, producing 4,000 Agent traces and a total of 67,950 tool-call records. Finally, four mainstream large models (DeepSeek-V4-Flash, GPT-5.5, Gemini-3-Flash, and MiniMax-M3) played the role of "attackers," looking only at the traces and not the conversations, to infer these 17 attributes.

The clever part of this design lies in the comparison: if the attacker only receives the attribute list and makes "blind guesses," that is the baseline; the extra accuracy after receiving the traces is the net evidence of leakage from the traces. For fairness, scoring uses forced prediction: all 17 attributes must be given an answer, wrong guesses count as wrong, semantic matching is adjudicated by another judge model that has not seen the traces, and ambiguous cases are all judged wrong. The entire process treats every attacker equally.

There is also a practical detail: the four attackers' scores were actually quite close, ranging from 25.29% to 28.32%, but their costs were worlds apart. Among the costs estimated in the paper, running all the experiments with GPT-5.5 cost about $1,080, while DeepSeek-V4-Flash cost only about $46, with nearly identical results. In other words, the barrier to launching this kind of profiling attack is far lower than most people imagine.

Let the numbers speak: what does 26.25% mean?

Experimental results: the attack success rate (ASR) for blind guessing was 17.97%, and after obtaining single-domain traces it rose to 26.25%, a net increase of 8.28 percentage points. Converted, across the 17 attributes, on average only four to five were guessed correctly, one or two more than blind guessing: enough to sketch an outline, far from enough to see through the person.

Figure 2: The larger the scope visible to the observer, the more accurate the inference. After linking traces across four domains, the success rate rose to 32.77% (Source: Figure 4 of the paper)

More noteworthy is the effect of the observer's scope: a single MCP server could see only 24.24%, a single domain was 26.25%, and when the same user's sessions across four domains were linked together, the success rate jumped to 32.77%. The larger the visible scope, the clearer the profile. It should be noted that the paper also cautions that these three numbers come from three different statistical bases and can only be regarded as correlation rather than causation; in paired experiments using the same basis, expanding from a single-server view to a single-domain view actually raised the success rate by only 3.09 percentage points.

Looking at individual attributes, the most severely leaked were nationality (57.73%) and religious belief (50.03%), and leakage in travel and shopping tasks was noticeably higher than in career and learning tasks. There is also a reassuring part: precise identity identifiers are very hard to guess. Among 16,000 identifier prediction slots, only 35 were guessed correctly, of which 34 were names and 1 was a phone number; the number of correct guesses for email and government identity identifiers was zero. What the traces leak is a "profile," not an "identity."

Figure 3: Comparison of inference success rates by attribute, and differences in leakage across task domains. Shopping tasks are the most "talkative" (31.86%) (Source: Figure 5 of the paper)

Where does the leakage come from: parameters are more "talkative" than results

Breaking down the four components of a trace (metadata, call order, parameters, and returned results) and ablating them one by one, the answer is clear: tool parameters contribute the most. With only metadata and call order, the success rate was just 14.53%, even lower than blind guessing; after adding parameters it soared to 27.71%, a net increase of 13.18 percentage points; when only returned results were added instead, it rose to 24.54%, a net increase of 10.01 percentage points. Parameters are indeed the channel with the most information.

Figure 4: Inference success rates by trace channel. Parameters (+ Args) are the largest information source (Source: Figure 6 of the paper)

The reasoning is not hard to understand: parameters are the choices the Agent makes on your behalf. What address you look up, what medicine you ask about, what budget you set—all of it is written in the parameters. Returned results often carry candidate items unrelated to the user: the full trace's 26.25% is actually lower than the 27.71% when looking only at parameters. The paper's interpretation is that returned content may either corroborate the clues in the parameters or dilute them, and the two mechanisms cannot currently be distinguished. One intriguing detail: even if the server sees only the request parameters and cannot see the returned content at all, the leakage is hardly reduced. This points to a clear path for defense: rather than agonizing over returned data, first manage request parameters well.

The study also validated a "two-stage" attack: first select attributes with evidential support from the traces; in the first stage only 17.06% of attribute slots were selected, but the accuracy on the selected portion was as high as 45.90%; then use these partial profiles to infer the remaining attributes, bringing the overall success rate to 25.83%, almost on par with the 26.25% of a one-shot attack, and about three-quarters of the correct predictions overlapped. In other words, a small number of evidence-supported clues can indeed drive subsequent inference: using partial profiles to complete the remaining attributes can raise the accuracy of this portion of inference from 18.48% to 21.78%, a net increase of 3.30 percentage points. However, the paper also clearly points out that this gain partly comes from model priors and does not prove that real attackers work along a path of "anchor first, then reason."

Traces do not lie, but you can leave fewer of them

It should be emphasized that this study used a combination of synthetic users and real tools; the behavior of virtual users may not be fully equivalent to real people, and 26.25% is far from "seeing through at a glance." But its value lies in disproving a common security assumption: "conversation confidentiality = user privacy security." In the Agent era, this equation no longer holds.

From unauthorized access and exfiltration of sensitive data to prompt injection and tool poisoning, industry discussion of MCP security has mostly focused on these specific risks, while this study reveals a quieter type of risk: no party does anything malicious, all calls comply with the protocol, and yet a profile can still be pieced together. Combined with the Agent security products and governance practices that have been emerging in the industry recently, the "retention" and "correlation" of tool calls are becoming compliance issues worthy of attention.

The defense directions given in the paper are also very specific and can be summarized as three gates.

The first is obfuscation: apply noise reduction to query parameters so that requests do not need to carry high-precision location, time, and identity clues—for example, generalizing "a 14-minute walk to a certain hospital" into "a trip in the city center." The second is minimization: limit the retention duration and content granularity of traces; the longer logs are kept, the greater the risk exposure. The third: cut off cross-service session correlation, and do not let one user's traces on four different servers be strung into a single line. However, the paper also states that the effectiveness of all three types of defense has not yet been experimentally verified and that they are currently only directional suggestions.

For enterprise users, this study also offers a deployment-level insight. As Agents enter office workflows, every time an employee authorizes an Agent to connect to a new tool, the company's behavioral data exposure surface is invisibly expanded. The traditional mindset of security teams is to guard "data and models"; now they probably need to add one more: guard "calls." The observability and privacy auditing of tool gateways are extensions beyond the paper itself and are becoming a link worth adding to the security architecture of the Agent era.

The more capable the Agent, the denser the footprints it leaves. Every step it takes on your behalf is a behavioral file pointing to you. Every trace it leaves may become evidence for future profiling. (This article was first published on TMTPost APP; author | AGI-Signal, editor | Jiao Yan)