Research shows that naive RAW-RAG outperforms complex Agentic Memory on long-conversation memory benchmarks
English text is being prepared.
A study compared on four long-conversation memory benchmarks—LoCoMo, LongMemEval, MEME, and MemoryAgentBench—and found that RAW-RAG, which only performs dense retrieval plus reranking on the raw conversation history, ranked first in 6 of 7 benchmark-and-model combinations, scoring 0.13 to 5.53 points higher than the strongest agentic memory system. The authors believe the bottleneck lies mainly in whether the evidence is complete, rather than in extraction, mutation, and graph structure; ablations show that removing reranking drops performance by 3.92 points, and returning only 1 item drops it by 11.73 points, while adding atomic fact extraction, summary augmentation, or graph expansion all fail to surpass the default RAW-RAG. The authors state their limitations as covering only offline public conversation benchmarks, not involving interaction, multimodality, or real-world operations, and the privacy and deletion constraints of raw-text storage are a real-world cost.
Why read this
AI score 59
The original text uses four long-conversation memory benchmarks and ablation experiments to compare the gap between naive RAW-RAG and complex agentic memory, and provides an explanation of coverage and path completeness.
Topics
View original postRAG is so back! This work shows that plain RAG over raw chat history beats complex agentic memory systems on most long-term memory benchmarks. (bookmark it) They tested Raw-RAG, which is just dense retrieval plus reranking over the raw conversation, feeding a reader LLM. Across LoCoMo, LongMemEval, MEME and MemoryAgentBench, it ranks first in six of seven benchmark and model settings, ahead of the strongest agentic memory system by 0.13 to 5.53 points. Ingestion on BEAM-1M takes 0.3 hours, against 3.2 to 27 hours for systems like Mem0, MemOS and Memora. Why does it hold up? Raw history keeps more of the evidence a question needs, including the relations and dependency paths behind an answer. Rewriting history into memory entries tends to drop those important bits. Even with complete access to the transformed memory stores, models answer less accurately than with raw evidence. Before adding a memory layer to your agent, at least run this baseline on your own data. https://t.co/4Fw1at4whN Chat with Paper: https://t.co/XJyaqjw0LE
View original
RAG is so back! This work shows that plain RAG over raw chat history beats complex agentic memory systems on most long-term memory benchmarks. (bookmark it) They tested Raw-RAG, which is just dense retrieval plus reranking over the raw conversation, feeding a reader LLM. Across LoCoMo, LongMemEval, MEME and MemoryAgentBench, it ranks first in six of seven benchmark and model settings, ahead of the strongest agentic memory system by 0.13 to 5.53 points. Ingestion on BEAM-1M takes 0.3 hours, against 3.2 to 27 hours for systems like Mem0, MemOS and Memora. Why does it hold up? Raw history keeps more of the evidence a question needs, including the relations and dependency paths behind an answer. Rewriting history into memory entries tends to drop those important bits. Even with complete access to the transformed memory stores, models answer less accurately than with raw evidence. Before adding a memory layer to your agent, at least run this baseline on your own data. https://t.co/4Fw1at4whN Chat with Paper: https://t.co/XJyaqjw0LE
Body
复杂 Agentic Memory 并非总是必要：朴素 RAW-RAG 在长对话记忆评测上更强
在 LoCoMo、LongMemEval、MEME、MemoryAgentBench 上，仅对原始对话历史做稠密检索 + 重排的 RAW-RAG，在 7 组「基准×模型」里有 6 组排名第一，比最强对比的 agentic memory 高 0.13–5.53 分；作者认为瓶颈主要是「证据是否齐全」，而不是复杂的抽取、变异与图结构。 https://t.co/vSEiSAL6Ej
问题背景 长对话超出上下文窗口后，近年流行 agentic memory：抽取、摘要、UPDATE/DELETE、图/社区、可学习检索等。常见质疑是：原始历史 RAG 对付不了「事实被覆盖、删除、级联状态」；且已有论文报告 RAG 落后。作者追问：这些复杂度是否总是必要？
两个预实验（动摇「必须显式维护状态」） 1. Mem0 写操作审计：LoCoMo 上 UPDATE+DELETE 仅占写决策的 4.39%（有效变更约 1.90%）；LongMemEval 为 2.46% / 1.34%。关掉 UPDATE/DELETE、只保留 ADD/NONE，LoCoMo 准确率反而 67.36% → 69.42%。 2. MEME 相位可见 oracle：把该阶段原始证据按时间序给 reader（无检索误差），强模型（如 DeepSeek-V4-Flash）在 cascade / invalidation 上可达约 82% / 92%——说明有足够原文时，reader 往往能自己推出现态。
RAW-RAG 怎么做 · 不做 LLM 抽取/摘要/构图/变异。 · 对话序列化（时间戳、图片说明）→ 256 token 窗、32 overlap → 扁平向量索引。 · 查询：余弦取 top-128 → cross-encoder 重排取 top-15 → 交给 reader 做时序与状态推理。
RAW-RAG 主结果（相对对比系统，不含 Full-context） · LoCoMo（GPT-4o-mini / 4.1-mini）：89.07 / 93.70，领先约 0.65–0.95 · LongMemEval（4.1-mini）：87.53，领先约 0.13（相对 Memora 87.40） · MEME（4o-mini）：39.30，领先约 2.24 · MemoryAgentBench（4o-mini / DS-V4-Flash）：48.32 / 53.41，领先约 4.5–5.5 · 唯一落后：MEME + DeepSeek-V4-Flash：MemMachine 44.71 vs RAW-RAG 43.19
消融：去掉重排 −3.92；只回 1 条 −11.73；加原子事实抽取 / 摘要增强 / 图扩展 都未能超过 默认 RAW-RAG（48.32）。摄入成本也低：BEAM-1M 上远快于 MemOS、Mem0 等（可达数十倍差距）。
为何够用：三点发现 1. 覆盖率 > 利用率：系统间「证据是否够」差距远大于「有证据后答对率」；RAW-RAG 覆盖最高，可抵消略低的条件利用率。 2. 路径完整性：有用证据 = 答案值 + 时序/依赖路径；RAW-RAG 的 P(path|result) 最高约 62.34%。 3. 摄入时保真：即便给「完整变换后记忆库」的 oracle，仍难追上「原始证据」；丢掉或扭曲的原文，事后检索补不回来。
局限（作者自述） 仅离线公开对话基准；未覆盖交互、多模态、真实运维。原文存储与隐私/删除约束是现实代价。弱 reader 或需减少重复推理时，显式状态维护是否更有用，仍待验证。