IBM 研究：智能体到底需要多少记忆？

Not every model benefits from the same amount of memory. Across eight models spanning the capability spectrum, we saw three recurring patterns:

Strong models with headroom want the full guideline set — every guideline, including rare edge-case lessons. They have the capacity to absorb and apply all of it. DeepSeek-V3.2 (671B MoE) climbed +9.5 percentage points in task completion when given its full self-mined guideline set.

Smaller or weaker models get drowned by a large guideline set. For these, a tight, high-confidence core plus a handful of task-relevant guidelines retrieved per task works best. gpt-oss-120b (117B MoE) gained +16.1pp with this selective approach — while the full guideline set gained less and cost ~50% more tokens.

Already-saturated models show no measurable gain. We call this the saturated pattern — the label describes what we observed, not a proven cause. The model may already have been near its ceiling on these tasks, the guidelines may not have addressed its remaining failures, or it may not have applied the guidance effectively. GLM-5 (745B MoE) sat here in our runs.

What puts a model into one pattern rather than another isn't simply parameter count. Benchmark headroom, context-window size, architecture, guideline quality, and task distribution all appear to shape where a model lands, and separating those factors is ongoing work. The practical takeaway holds either way: the right dose of memory depends on the model, and we can calibrate it.

Learning happens around the model, not inside it

"Memory" here doesn't mean replaying a past transcript. It means a guideline set — strategies that worked, mistakes to avoid, and edge cases — distilled from the agent's own prior trajectories. The loop is straightforward:

The agent attempts tasks and produces trajectories.

ALTK-Evolve extracts behavioral guidelines from both its successful and unsuccessful runs.

It consolidates those guidelines into a reusable set.

At inference time, the agent receives either the full guideline set or a task-relevant selection of it.

No model weights are updated. The learning loop changes the guidance available to the agent , not the underlying model — which is exactly why it's cheap to adopt and portable across the eight models we tested.

Results Across the Spectrum

We evaluated on AppWorld — 585 multi-step tasks (168 test_normal + 417 test_challenge ) across 9 simulated apps (calendars, messaging, payments, and so on). Tasks are scored two ways: whether the agent fully completes each task ( TGC — Task Goal Completion ) and whether every variant of a scenario passes ( SGC — Scenario Goal Completion , a stricter, all-or-nothing bar). Full definitions are in the appendix.

The three configurations we compare

Because the confusing part of any memory study is what's actually in the context window , we define the configurations up front.

Both memory configurations draw from the same guideline set , mined once (via the loop above) from AppWorld's training split only. What changes between them is only how that one set is delivered — the full guideline set injects all of it every step, while curated retrieval delivers a selected subset — never how the guidelines were produced, and no test-split data ever goes into building it.

The number of guidelines a model mines depends on its own capability, so we report configurations by strategy — "full guideline set" vs. "curated retrieval" — rather than by raw counts, which aren't comparable across models.

The three patterns, in one view

Representative models from the eight-model sweep, measured by task completion (TGC) on test_normal :

Figure 1. Representative models in the three observed patterns. Bars show TGC on AppWorld test_normal for baseline vs. the best-memory configuration; the x-axis begins at 40% to make differences visible. TGC alone understates the larger SGC gains — see the SGC columns in the table below.

The figure plots TGC to keep it readable; the table adds the stricter SGC metric, where the gains are often larger:

Reading the SGC column, the stricter metric usually moves more than TGC — DeepSeek's SGC jumps +16.1pp against a +9.5pp TGC gain — because good guidelines especially help an agent clear every variant of a scenario, not just the average case. And the effect doesn't disappear at the top of the range: GPT-5.5 and Opus, both near the ceiling on TGC, still gain +7.2 and +7.1pp SGC respectively. Memory keeps paying off as long as a model has a remaining failure mode to target.

A practical concern: injecting a full guideline set inflates every ReAct step's input, because the guidelines are re-sent each turn. Here's what we observed:

Table 1. Average token use per task, accumulated across agent steps, measured against the no-memory baseline.

Two takeaways:

Curated retrieval keeps cost near baseline. For weaker models, where selection wins on accuracy, it also wins on cost — the best of both worlds ( +16.1pp TGC at only +5% tokens for gpt-oss-120b). Better performance here does not require more inference cost.

Memory doesn't blow up the reasoning loop. DeepSeek runs about the same number of ReAct steps with memory as without (≈18–19 on average), so the added cost is input-token inflation, not longer trajectories.

The real efficiency lever in production is prompt caching : the static portion of the guideline set is identical across steps and can be cached, cutting effective cost substantially. Cache-aware prompt design — keeping the shared guideline-set prefix stable so it stays cacheable — is worth engineering for. We also hypothesize that context-window size plays a role: models with larger windows may absorb the full guideline set more effectively, while smaller-context models benefit more from retrieval that keeps injected content compact. We have not yet run controlled experiments isolating this factor.

The lesson isn't to give an agent everything it has learned. It's to give it the amount of experience it can actually use.

For weak models , that means a compact core plus a few task-specific lessons — which, conveniently, is also the cheapest option.

For strong models with headroom , it means preserving the full guideline set, kept affordable in production via prompt caching.

For saturated models , it means spending no extra context until their remaining failure modes are better understood.

The gains are real across the board — automatic, leakage-free, and requiring no human annotation — but only when the dose fits the model.

What's Next

This is a starting point, not the finish line:

A learned selector. Our current retrieval ranks guidelines by cosine similarity, which we've shown doesn't perfectly predict which guidelines help a given task. A selector trained on outcome signal is the natural next step.

Memory for very weak models. Below a minimum capability baseline, self-distillation lacks signal. Teacher-distilled memory for very weak models is a separate problem we're exploring.

Beyond AppWorld. These results are validated on AppWorld — a rigorous multi-step benchmark, but a single one. Broader agent benchmarks and real-world deployments are in progress.

Isolating context window. As above, we want controlled experiments that separate context-window size from raw capability.

Appendix: Understanding the Metrics

AppWorld tasks are graded by two metrics, both reported as percentages (higher is better):

TGC — Task Goal Completion. The share of individual tasks the agent completes fully and correctly. This is the headline "did it get the job done" number.

SGC — Scenario Goal Completion. A stricter, all-or-nothing metric. Each scenario bundles several variants of the same task (the same request with different data, phrasing, or edge conditions). SGC counts a scenario as passing only if the agent succeeds on every variant. It measures reliability — an agent that solves a task most of the time but fails on one variant scores on TGC but not on SGC.

More from this author

Your Agent Aced the Task. Will It Do It Again?

118 September 15, 2026 IBM releases SOTA Granite Time Series PatchTST-FM-r2 model with commercial-friendly license 60 September 9, 2026 Community Really interesting take on agentic memory. The idea that memory needs to be calibrated to the model rather than simply accumulated is especially useful for production systems. Selective retrieval also seems like a practical way to improve reliability without unnecessarily increasing context and inference costs. It makes a strong case for thoughtful custom AI agent development and enterprise AI agent solutions, where memory, performance, and cost all need to be designed together. https://mobisoftinfotech.com/services/ai-agent-development-company \n","updatedAt":"2026-08-28T07:43:38.888Z","author":{"_id":"6a912d8a5482311da81bc005","avatarUrl":"/avatars/099746ac56e31bb4785307daebf2c2bf.svg","fullname":"Ajay Yadav","name":"ajayyadav031853","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.9275968670845032},"editors":["ajayyadav031853"],"editorAvatarUrls":["/avatars/099746ac56e31bb4785307daebf2c2bf.svg"],"reactions":[],"isReport":false}}],"status":"open","isReport":false,"pinned":false,"locked":false,"collection":"community_blogs"},"contextAuthors":["Vatche","gaodan-fang","jayaramkr","illeatmyhat","ashwath-vaithina","evduester","gsthomasx","vinodmut","mrvnvr","ayhansebin"],"primaryEmailConfirmed":false,"discussionRole":0,"acceptLanguages":["en","zh"],"withThread":true,"cardDisplay":false,"repoDiscussionsLocked":false,"hideComments":true}"> ajayyadav031853 Aug 28 Really interesting take on agentic memory. The idea that memory needs to be calibrated to the model rather than simply accumulated is especially useful for production systems. Selective retrieval also seems like a practical way to improve reliability without unnecessarily increasing context and inference costs. It makes a strong case for thoughtful custom AI agent development and enterprise AI agent solutions, where memory, performance, and cost all need to be designed together. https://mobisoftinfotech.com/services/ai-agent-development-company

Reply Edit Preview Upload images, audio, and videos by dragging in the text input, pasting, or clicking here . Tap or paste here to upload images Comment · Sign up or log in to comment

+71