Anthropic study: Agent memory requires complementary background organization and retrieval
The Anthropic team shared research on Agent memory, noting that memory is not just about saving history, but also about re-checking evidence, correcting notes, organizing experience, and retrieving on demand; background organization is called dreaming. In an internal corpus experiment with about 350,000 files, 420 million tokens, and 25 evaluation questions, the scoring-criteria coverage of search alone was 53%; the knowledge base combined with search improved from 54% to 61% as organization rounds increased, using about 20,000 fewer tokens per question, but the saved tokens did not deduct the cost of building the background knowledge base. In another contract case, after changing to one separate page per contract, with one agent responsible and quoting clauses verbatim, key-fact coverage improved from 41/57 to 57/57.
Why read this
AI score 60
The original text provides experimental data on the complementarity of organization and retrieval, and notes that the cost of building the background knowledge base was not deducted, allowing readers to judge the actual benefit boundaries of memory solutions.
Topics
View original postMemory is what allows Claude to learn from experience. In a new Engineering Blog post, we share how “dreaming” helps Claude organize what it’s learned, correct its mistakes, and find what it needs more easily the next time. https://t.co/uUEN4bNkr9
View original
Memory is what allows Claude to learn from experience. In a new Engineering Blog post, we share how “dreaming” helps Claude organize what it’s learned, correct its mistakes, and find what it needs more easily the next time. https://t.co/uUEN4bNkr9
Body · AI translation
Anthropic's latest research: more Agent memory isn't always better—how Anthropic turns records into usable experience
Shared by the Anthropic team @RLanceMartin, multiple experiments show: Agents don't just need to save history; they also need to look back at evidence, correct notes, organize experience, and find them when needed. Background organization can improve some tasks, but it is not a substitute for retrieval, nor does it always offset maintenance costs.
https://t.co/H1FFfA8rkh
# Memory is not the same as chat logs
Memory here is context saved across sessions and readable later; it does not involve updating model weights, nor is it just compressing the current conversation.
The article discusses several interlocking steps: keep raw records for traceability; while performing tasks, write down goals, plans, and verified experience; review records in the background to correct errors, deduplicate, add links, and build a knowledge base when necessary; at read time, preload only a small amount of important content and retrieve the rest on demand.
The authors call background organization dreaming. It does not make the model "sleep"; rather, it separates organizing experience from the task at hand, giving the model dedicated time to process existing information.
# What good memory should store
In the CL-Bench database task, each question starts a new session, and only the memory directory is retained. After answering, the model receives the correct answer and then organizes experience. Stronger models begin to save verified rules and their basis, not just wrong answers; restricting it to "record only failures" instead weakened the gains.
This shows that valuable memory is not just "what was done wrong last time," but also which rules hold, why they are trustworthy, and when they will apply later. State that can be reread at any time does not need to be repeatedly copied into notes.
# Experiments show: organization and search are complementary
In an internal corpus experiment with about 350,000 files, 420 million tokens, and 25 evaluation questions, search alone achieved 53% rubric coverage; a knowledge base combined with search rose from 54% to 61% as organization rounds increased, using about 20,000 fewer tokens per question.
The 61% here is not "answered 61% of the questions correctly," and the saved tokens do not deduct the cost of building the knowledge base in the background. Gains were more pronounced on cross-document integration tasks, while some tasks focused mainly on single-file retrieval showed no improvement, and the knowledge base was even larger than the original text.
In another contract case, multiple agents co-writing one table caused overwrites. After changing to one page per contract, one agent responsible for each, and verbatim quotation of clauses, key fact coverage in the final report rose from 41/57 to 57/57. The improvement came from organization and writing rules, not just from more thinking time.