Research shows that naive RAW-RAG outperforms complex Agentic Memory on long-conversation memory benchmarks
A study compared on four long-conversation memory benchmarks—LoCoMo, LongMemEval, MEME, and MemoryAgentBench—and found that RAW-RAG, which only performs dense retrieval plus reranking on the raw conversation history, ranked first in 6 of 7 benchmark-and-model combinations, scoring 0.13 to 5.53 points higher than the strongest agentic memory system. The authors believe the bottleneck lies mainly in whether the evidence is complete, rather than in extraction, mutation, and graph structure; ablations show that removing reranking drops performance by 3.92 points, and returning only 1 item drops it by 11.73 points, while adding atomic fact extraction, summary augmentation, or graph expansion all fail to surpass the default RAW-RAG. The authors state their limitations as covering only offline public conversation benchmarks, not involving interaction, multimodality, or real-world operations, and the privacy and deletion constraints of raw-text storage are a real-world cost.
Why read this
AI score 59
The original text uses four long-conversation memory benchmarks and ablation experiments to compare the gap between naive RAW-RAG and complex agentic memory, and provides an explanation of coverage and path completeness.
Topics
View original postRAG is so back! This work shows that plain RAG over raw chat history beats complex agentic memory systems on most long-term memory benchmarks. (bookmark it) They tested Raw-RAG, which is just dense retrieval plus reranking over the raw conversation, feeding a reader LLM. Across LoCoMo, LongMemEval, MEME and MemoryAgentBench, it ranks first in six of seven benchmark and model settings, ahead of the strongest agentic memory system by 0.13 to 5.53 points. Ingestion on BEAM-1M takes 0.3 hours, against 3.2 to 27 hours for systems like Mem0, MemOS and Memora. Why does it hold up? Raw history keeps more of the evidence a question needs, including the relations and dependency paths behind an answer. Rewriting history into memory entries tends to drop those important bits. Even with complete access to the transformed memory stores, models answer less accurately than with raw evidence. Before adding a memory layer to your agent, at least run this baseline on your own data. https://t.co/4Fw1at4whN Chat with Paper: https://t.co/XJyaqjw0LE
View original
RAG is so back! This work shows that plain RAG over raw chat history beats complex agentic memory systems on most long-term memory benchmarks. (bookmark it) They tested Raw-RAG, which is just dense retrieval plus reranking over the raw conversation, feeding a reader LLM. Across LoCoMo, LongMemEval, MEME and MemoryAgentBench, it ranks first in six of seven benchmark and model settings, ahead of the strongest agentic memory system by 0.13 to 5.53 points. Ingestion on BEAM-1M takes 0.3 hours, against 3.2 to 27 hours for systems like Mem0, MemOS and Memora. Why does it hold up? Raw history keeps more of the evidence a question needs, including the relations and dependency paths behind an answer. Rewriting history into memory entries tends to drop those important bits. Even with complete access to the transformed memory stores, models answer less accurately than with raw evidence. Before adding a memory layer to your agent, at least run this baseline on your own data. https://t.co/4Fw1at4whN Chat with Paper: https://t.co/XJyaqjw0LE
Body · AI translation
Complex Agentic Memory Is Not Always Necessary: Naive RAW-RAG Is Stronger on Long-Conversation Memory Benchmarks
On LoCoMo, LongMemEval, MEME, and MemoryAgentBench, RAW-RAG, which only performs dense retrieval + reranking on the raw conversation history, ranks first in 6 of 7 "benchmark × model" groups, scoring 0.13–5.53 points higher than the strongest agentic memory comparison; the authors argue that the bottleneck is mainly "whether the evidence is complete," not complex extraction, mutation, and graph structures. https://t.co/vSEiSAL6Ej
Background When long conversations exceed the context window, agentic memory has become popular in recent years: extraction, summarization, UPDATE/DELETE, graphs/communities, learnable retrieval, etc. A common concern is that RAG over raw history cannot handle "facts being overwritten, deleted, or cascading states"; and existing papers have reported that RAG lags behind. The authors ask: is this complexity always necessary?
Two preliminary experiments (undermining "explicit state maintenance is required") 1. Mem0 write-operation audit: on LoCoMo, UPDATE+DELETE account for only 4.39% of write decisions (effective changes about 1.90%); on LongMemEval, 2.46% / 1.34%. Turning off UPDATE/DELETE and keeping only ADD/NONE, LoCoMo accuracy instead went from 67.36% → 69.42%. 2. MEME phase-visible oracle: giving the reader the raw evidence for that phase in chronological order (no retrieval error), a strong model (e.g., DeepSeek-V4-Flash) can reach about 82% / 92% on cascade / invalidation—showing that with enough original text, the reader can often infer the current state itself.
How RAW-RAG works · No LLM extraction/summarization/graph construction/mutation. · Serialize the conversation (timestamps, image captions) → 256-token windows, 32 overlap → flat vector index. · Query: cosine top-128 → cross-encoder rerank top-15 → handed to the reader for temporal and state reasoning.
Main RAW-RAG results (relative to comparison systems, excluding Full-context) · LoCoMo (GPT-4o-mini / 4.1-mini): 89.07 / 93.70, leading by about 0.65–0.95 · LongMemEval (4.1-mini): 87.53, leading by about 0.13 (vs. Memora 87.40) · MEME (4o-mini): 39.30, leading by about 2.24 · MemoryAgentBench (4o-mini / DS-V4-Flash): 48.32 / 53.41, leading by about 4.5–5.5 · Only loss: MEME + DeepSeek-V4-Flash: MemMachine 44.71 vs RAW-RAG 43.19
Ablations: removing reranking −3.92; returning only 1 item −11.73; adding atomic fact extraction / summary augmentation / graph expansion all failed to exceed default RAW-RAG (48.32). Ingestion cost is also low: on BEAM-1M it is far faster than MemOS, Mem0, etc. (up to tens of times faster).
Why it is sufficient: three findings 1. Coverage > utilization: the gap between systems in "whether evidence is sufficient" is far larger than the gap in "accuracy after having evidence"; RAW-RAG has the highest coverage, which can offset slightly lower conditional utilization. 2. Path completeness: useful evidence = answer value + temporal/dependency path; RAW-RAG has the highest P(path|result), about 62.34%. 3. Fidelity at ingestion: even giving an oracle with a "complete transformed memory store," it is still hard to catch up with "raw evidence"; original text that is lost or distorted cannot be recovered by later retrieval.
Limitations (as stated by the authors) Only offline public conversation benchmarks; does not cover interaction, multimodality, or real operations. Storing raw text and privacy/deletion constraints are real costs. Whether explicit state maintenance is more useful for weak readers or when repeated reasoning needs to be reduced remains to be verified.