Elaborate agent memory loses to plain search over the chat history
Plain retrieval with reranking over raw conversation history beat twelve specialised memory systems in six of seven settings and runs many times faster.
Source: arXiv:2610.30755 · Authors: Zhengye Bao, Shengtao Zhang, Haokai Zhang, Shiqi Lou, Muning Wen, Minnan Luo · License: CC BY 4.0
Is Complex Agentic Memory Always Needed? Naive RAG Is Stronger Than We Thought
LLM memory is widely used in long-term conversation understanding. Agentic memory systems often use mutation, structural organization, learned retrieval strategies, or other complex operations. Is this complexity always necessary? We study Raw-RAG, a naive RAG baseline using dense retrieval and reranking over raw history to supply evidence to a reader LLM.
Across LoCoMo, LongMemEval, MEME, and MemoryAgentBench, Raw-RAG ranks first among the compared memory systems in six of seven benchmark-model settings, outperforming the strongest compared agentic memory system by 0.13-5.53 points. We investigate why raw-history retrieval remains competitive. Evidence analysis shows that its higher coverage of necessary evidence can offset lower utilization. Useful evidence includes both answer values and the relations and dependency paths needed to derive them. Even complete access to transformed memory stores leaves an accuracy gap relative to raw evidence, highlighting the importance of preserving raw information.
These findings challenge the necessity of complex agentic memory for the evaluated conversation tasks and establish naive RAG as an essential baseline.
This is an excerpt (abstract or key part). The full text is at the source link.
Question. Agent memory systems (Mem0, MemOS, MemMachine, EverMemOS and others) extract facts, write summaries, build graphs and update records. The authors test whether this complexity beats simple retrieval over the original conversation text.
How Raw-RAG works.
- The conversation history (with timestamps and image captions) is split into 256-token windows with 32-token overlap and indexed as-is, with no LLM involved.
- For a question, the 128 nearest windows are retrieved (cosine similarity), and a cross-encoder keeps the top 15.
- A reader model (GPT-4o-mini, GPT-4.1-mini or DeepSeek-V4-Flash) works out event order and state changes itself.
Results. Against 12 systems on four benchmarks (LoCoMo, LongMemEval, MEME, MemoryAgentBench), Raw-RAG ranks first in 6 of 7 settings, by 0.13–5.53 points. For example, MemoryAgentBench with DeepSeek-V4-Flash: 53.41 vs 47.88 for Memora. The one loss: MEME with DeepSeek-V4-Flash (43.19 vs 44.71 for MemMachine). Full context without retrieval scores 63.17 there, so every memory system, Raw-RAG included, trails it by a wide margin.
Why it works. Systems differ mostly in whether they surface the needed evidence (coverage on MemoryAgentBench ranges 19–45%), not in how they use it once found (80–90%). Raw-RAG has the highest coverage (45.29%).
What hurts. Adding "smart" steps to Raw-RAG makes it worse: atomic-fact extraction −19.62 points, summaries −5.81, graph expansion −5.78. Removing the reranker costs −3.92. Even perfect access to every record in a transformed memory store stays 7.58–29.38 points below raw evidence.
Cost. On a MEME slice, Raw-RAG's ingestion plus search take 306 s vs 1,562–9,301 s for other systems; on a ~1M-token history, 0.33 h vs 3.25 h for the next-fastest system that finished.
Limitations (per the authors). Offline conversations on public benchmarks only; storing raw history raises privacy and deletion concerns; results depend on the reader model's strength. The preprint has not been peer-reviewed.
For an AI assistant to remember what you talked about a month ago, developers give it a "memory". Many such systems are elaborate: the assistant writes down key facts, makes short summaries, links them into diagrams and fixes outdated entries.
The authors tested a simple alternative. It is like keeping the whole chat log instead of notes, and, before answering a question, quickly finding the most relevant bits and rereading them.
The simple approach almost always worked better and many times faster. The reason makes sense: when you write notes, it is easy to drop a detail that later turns out to matter. The original conversation loses nothing — you just need to find the right place.
What this changes: before building complex memory, compare it against this simple option. But keeping the full history has a downside — privacy: such data is harder to "forget" when a user asks.