HKUST(GZ) and others launch the MEMCUT benchmark, revealing the residual inferability risk after Agent memory deletion

Long-term Memory has become a core component of LLM Agents. However, current research in this field is almost entirely focused on "how to retrieve and recall information more accurately and efficiently," while neglecting "the other side of the coin":

When a user, for privacy or compliance reasons (such as the GDPR's "right to be forgotten"), asks an Agent to forget certain sensitive information (such as a medical diagnosis or an undisclosed business acquisition), can the memory system truly erase it cleanly?

In actual long-term interactions, even if the user has the system delete the sentence that directly states the secret (Disclosure), the seemingly harmless ordinary records scattered across weeks of daily conversations—such as booking records, office addresses, schedules, or interpersonal relationships—will still form an implicit "Evidence Trail."

By piecing these fragments together, the original secret can still be easily inferred. Even more dangerous is that sometimes the Agent appears unable to answer a question on the surface, simply because its built-in retriever missed some clues, rather than because the underlying data was truly erased.

To address the evaluation challenge of User-requested Forgetting, a research team from The Hong Kong University of Science and Technology (Guangzhou), in collaboration with the University of Science and Technology of China and The Hong Kong University of Science and Technology, constructed a new high-quality evaluation benchmark, MEMCUT, and proposed dual evaluation metrics that distinguish between "surface-level retrieval failure (Native Recovery)" and "underlying evidence recoverability (Residual Recovery)."

In a comprehensive evaluation spanning 5 mainstream memory architectures and 4 large models, they found that after deleting only the direct records, the secret recovery rate from underlying residual evidence still reached 85%-99%; even with LLM-assisted cleanup of associated records, 39%-76% of secrets could still be inferred and recovered, and they precisely identified "insufficient retrieval coverage of the candidate pool" as the core bottleneck constraining current systems from achieving true forgetting.

Figure | After deleting direct disclosures, daily records still expose secrets

MEMCUT is the first to systematically reveal the "surface amnesia" illusion of Agent long-term memory in privacy deletion, advancing the evaluation dimension from mere "direct memory recall" to "the derivability (Derivability) of multi-hop implicit evidence chains."

This work not only provides a standardized diagnostic tool and testing platform for building trustworthy, compliant privacy-preserving Agents, but also points out the core breakthrough direction for designing dedicated memory erasure mechanisms.

I. Research Methods

To rigorously test "whether deletion equals forgetting," they designed the study along three dimensions: benchmark construction, evaluation metric design, and a unified cleanup protocol.

1. High-quality benchmark construction: the ingenious design of MEMCUT

MEMCUT abandons the approach of relying entirely on large models to generate facts out of thin air, and instead adopts a process of "fully manual logical design + model polishing into text + manual sentence-by-sentence strict review," spending at least 12 hours polishing each long scenario.

The benchmark contains 15 multi-session daily and work scenarios spanning 1-3 months, totaling 490 sessions, 43,402 dialogue turns, and approximately 1.74 million tokens, with an average of 2,893 dialogue turns per scenario, protecting a total of 456 sensitive secrets.

For each secret, they not only annotated the disclosure turn that directly states the secret (Disclosure), but also meticulously designed an average of 8.1 implicit evidence items (Evidence) scattered across 8 different sessions, as well as 26.6 high-confusion same-topic distractors (Distractors).

A single piece of evidence poses no leakage risk on its own, but combined they can infer the secret—each secret contains an average of 15.7 minimal sufficient sets.

To thoroughly forget a secret, the deletion set must sever every sufficient set, that is, form a "Cut," and the average minimal cut set requires deleting only 3.04 turns of evidence.

These evidence chains comprehensively cover the four logics of daily information leakage:

Fact Assembly, combining different attributes of the same entity, such as code name + inspection address + registered company = acquisition target;

Relation Chain, multi-hop connections across entities, such as a contract pointing to a consulting firm, documents showing the company's legal representative, and dinner records showing the legal representative's spouse is a certain executive;

Aggregation, cumulative calculation of multiple values or frequencies, where the correct total can only be calculated if all are retained, and deleting any one breaks the inference;

Temporal & State Reasoning, tracking state updates that evolve along a timeline, such as proposals, modifications, and retractions.

Figure | MEMCUT sample structure and deletion evaluation process

2. Dual recovery metrics and "hidden risk"

After the deletion operation is executed, they test using four-choice questions that include a "refuse to answer" option, and a secret is only counted as recovered if answered correctly under both random option orders, eliminating the possibility of guessing—the recovery rate for blind guessing without memory is 0.

Native Recovery (Nat.) uses the memory system's built-in retriever to retrieve context and answer questions, measuring how many secrets the Agent still ostensibly exposes during everyday interactions.

Residual Recovery (Res.), by contrast, bypasses the retriever and directly provides the reading model with all surviving memory objects in the memory store that originate from that secret's disclosure or evidence, measuring the extent to which residual data in the underlying storage still supports reconstructing the secret.

On this basis, they define Hidden-risk Share (Hid.): Hid = (Res − Nat) / Res, which quantifies the proportion of hidden risk that is "actually still derivable from the underlying data, but appears forgotten because the system's own retriever failed to recall it."

3. Unified testing framework and three reference deletion baselines

Different memory systems store at different granularities: some store single-turn dialogues, some store entire sessions, and some store summaries.

To this end, they anchor all operations to Source Turns, track the origin of each derived memory object through snapshot differencing, and perform Closure Deletion—whenever a turn is deleted, all memory objects derived or summarized from it are removed as well.

On this basis, they evaluated three deletion baselines that reuse existing retrieval and read-write components.

S0 (delete direct disclosures only) deletes only the turns that directly state the secret and their derived objects.

S1 (retrieval-ranking cleanup) builds on S0 by feeding the user's natural-language deletion request into the system retriever, obtaining a Top-20 candidate turn pool, and directly deleting the top 10 ranked turns—since the minimum cut set for any secret does not exceed 5 turns and the total evidence does not exceed 10 turns, a budget of 10 turns is theoretically entirely sufficient.

S2 (LLM-guided cleanup) likewise builds on S0 by having a large model score and rerank the Top-20 candidate turns for relevance based on the stored text, then selecting the 10 highest-scoring turns for deletion.

II. Research Results

They conducted comprehensive experiments on 20 combinations formed by 5 representative memory architectures—2 raw-text retrieval stores (Turn RAG, Session RAG) and 3 memory systems with varying degrees of summary evolution (MemoryBank, Nemori, LightMem)—and 4 mainstream large models (DeepSeek-V4.1-Flash, Qwen3.8-Flash, GLM-5.3-Flash, MiMo-V2.5).

1. Recovery results for each memory system and model

The recovery results show that secrets still recoverable after cleanup are abundant in both types of raw-text retrieval stores. Taking LLM-guided cleanup as an example, Turn RAG's residual recovery rate is 41.4%–45.6%, and Session RAG's is 54.2%–64.3%; meanwhile, their native recovery rates are only 7.5%–10.7% and 4.2%–10.7%, respectively. These configurations do not rely on derived summaries, indicating that surviving raw dialogue alone can still be used to derive secrets that the system ostensibly can no longer answer.

Table | Recovery results for the two retrieval stores.

The table below shows the differences across memory organization methods. Under the same LLM-guided cleanup scheme, the residual recovery rates of MemoryBank, Nemori, and LightMem are 71.7%–76.3%, 52.4%–57.2%, and 39.3%–43.0%, respectively. The gap between the four writing models within the same system is about 4–5 percentage points, while the differences between systems are larger. This result suggests that when evaluating forgetting capability, one must examine memory organization, candidate retrieval, and deletion selection together, rather than looking only at which large model is used.

Table | Recovery results for the three memory systems

2. Deleting direct messages is merely "plugging one's ears while stealing a bell"

Experiments show that when only directly disclosing messages are deleted, the native recovery rate of each system drops substantially by 12–50 percentage points (e.g., Turn RAG falls from 91%–95% before deletion to 57%–60%), and from the perspective of external questioning, the Agent appears to have "forgotten" quite a few facts.

However, the residual recovery rate at this point remains as high as 85%–99%! This is because only 1.8% of implicit evidence is in the same session as the direct disclosure, and the vast majority of indirect clues remain intact.

Figure | The residual risk behind "apparent amnesia"

The figure above places "apparent amnesia" alongside the residual risk after further cleanup: the horizontal axis shows how many secrets the system itself can still answer after deleting only direct disclosures; the vertical axis shows how many secrets the underlying evidence can still reconstruct after adopting stronger LLM-guided cleanup. The two axes correspond to different deletion stages, yet point to the same problem: the system answering less and less does not mean the underlying clues have been thoroughly removed. Even after further cleanup of related records, evidence scattered across memory can still be pieced back together into the original secret.

3. Deletion instead raises "hidden risk"

After using retrieval-ranking cleanup to additionally delete 10 turns of records, the residual recovery rate still remains at 48%–86%.

Even after upgrading to stronger LLM-guided cleanup, although the recovery rate is further reduced by 7-17 percentage points compared with retrieval-ranking cleanup, the residual recovery rate across all configurations still reaches as high as 39%-76%; among them, in Turn RAG without derivative summaries, 41%-46% still survive, and even after excluding the factor of summary residue, complete forgetting still cannot be achieved.

More alarming is that, as cleanup advances from Full to LLM-guided cleanup, the hidden-risk proportion in all 20 configurations rises rather than falls, exceeding 50% in 14 configurations, and in Turn RAG and Session RAG reaching as high as 76%-92%.

This means that the more one deletes through existing retrievers, the easier the remaining evidence is to evade conventional questioning, giving users a false sense of security that "it has been completely deleted."

Figure | After LLM-guided cleanup, the hidden-risk proportion of relation chains remains high

Breaking it down further by derivation type, the easiest to "hide" are relation chains. In the figure above, the purple bars representing relation chains exceed 80% in all four systems, with Turn RAG and MemoryBank reaching 90.3% and 93.7%, respectively. That is, even after LLM-guided cleanup, the vast majority of those relation-chain secrets that can still be reconstructed from underlying evidence will not be exposed in conventional questioning. Although recoverable secrets are reduced, the remaining risk is more hidden—this is precisely where "it looks completely deleted" most easily makes people relax their vigilance.

4. Why does forgetting fail?

Why, given a sufficient 10-turn deletion budget, can the secrets still not be deleted? They conducted a controlled-variable sweep analysis on candidate pool size (Reach) and deletion budget (Budget) over 144 secret subsets.

The results found that blindly increasing the deletion budget has little effect: when the candidate pool is fixed at 20 turns, even if the deletion budget is maxed out from 10 turns to 20 turns (that is, deleting all 20 retrieved candidates), the residual recovery rate decreases by at most only 6.5 percentage points compared with LLM-guided cleanup, and it adds nearly 10 turns of collateral damage from irrelevant conversations.

In contrast, expanding the candidate pool has a significant effect: keeping the 10-turn deletion budget unchanged and expanding the candidate pool from 20 to 60 reduces the residual recovery rate of LLM-guided cleanup by a large margin of 12.5-23.9 percentage points, while also reducing collateral damage on irrelevant turns.

Across all 132 sweep settings, the residual recovery rate shows an extremely strong linear relationship with the number k of evidence turns actually successfully deleted: Res ≈ 13 × (8.2 − k) (R² = 0.95).

Because a candidate pool of size 20 on average can recall less than half of the evidence (4.5 turns for Turn RAG and LightMem, and only 3.0 turns for MemoryBank), retrievers based on semantic similarity simply cannot "reach" those implicit evidence items that are worded differently but logically related—this is precisely the decisive bottleneck limiting the forgetting effect.

Figure | Comparison of candidate pool sweep and deletion budget sweep

5. Deeper patterns of memory granularity, derivation strength, and derivation type

From the perspective of retrieval granularity, Session RAG retrieves by whole session, so a single retrieval mixes in a large amount of irrelevant conversation; not only is the native recovery rate before deletion low, but the candidate pool is also severely diluted during cleanup, causing its residual recovery rate after cleanup to be 10-23 percentage points higher than that of Turn RAG, and its hidden risk is also the highest.

From the perspective of derivative summaries, stronger summary rewriting (such as LightMem and Nemori) fuses multi-turn fragments together when generating summaries, making clues before and after deletion easier to discover by conventional retrieval (hidden-risk proportion is relatively low, about 41%-56%), and making them easier to be removed together during closure deletion; while MemoryBank, which retrieves only by single turn, has the highest residual recovery rate after cleanup (72%-76%), with hidden risk as high as 78%-82%.

Among the four derivation types, Relation Chain is the hardest to forget. Although in theory it is the easiest to cut (average minimum cut set is only 2.68 turns), after cleanup it retains the highest residual recovery rate (average 71.8%) and the highest hidden risk (81%-94%).

The reason is that the evidence of a relation chain bridges across topics through entities (such as person names and company names), with almost no overlap with the surface wording of the user's deletion request: the request is "forget that the company we hired belongs to the CFO's wife," while the evidence is respectively "signed a contract with Arden," "Arden's legal representative is Nina," and "CFO Ravi and his wife Nina celebrated their wedding anniversary," causing vector-similarity retrieval to fail.

Figure | Case of a relation-chain secret under LLM-guided cleanup

This also suggests that future memory systems can track one-hop associated records through Entity Expansion, thereby breaking through the blind spot of purely semantic retrieval and achieving complete evidence-chain cutting.

Of course, this study also has some limitations, mainly including the following three aspects:

1. Controlled evaluation. MEMCUT evaluates forgetting effects for secrets with specified evidence chains under four derivation types. Controlled construction enables them to compare candidate coverage, deletion selection, and recovery of residual information under fixed annotations. These scenarios simulate recurring work, family, and social activities. As for the frequency with which such evidence chains appear, and how forgetting performance varies with naturally occurring evidence structures, deployment research is needed to determine.

2. Reference baseline. Their reference baseline reuses existing retrieval and writing components through a unified deletion pipeline. The sweep experiments examine candidate coverage and selection under this setting. These results establish the reference performance of this reuse strategy. MEMCUT also supports evaluating specialized erasure methods, such as methods that "expand candidate coverage" or "change the memory indexing approach."

3. Research scope. This study examines textual memory; multimodal memory and parametric memory are outside the scope of this paper. Source-level metrics cannot reflect irrelevant content removed together due to closure, nor can they reflect the loss of downstream utility.

This article comes from the WeChat public account "Academic Headlines" (ID: SciTouTiao), author: Academic Headlines, authorized for publication by 36Kr.