University of Texas paper proposes Jev-Mem: long-conversation memory construction 6.6x faster
The Jev-Mem proposed in a University of Texas paper delegates small memory decisions such as linking and stopping the search to a fast scoring model, Jev, replacing text generation with batch probability returns, and only calls the LLM at the end to read memory and answer, making long-conversation memory construction 6.6x faster. On the LoCoMo long-conversation benchmark, its answer score is 0.777, higher than the best baseline's 0.700; memory construction takes 158 seconds versus 1,044 seconds for the fastest comparison method, and answering takes 0.93 seconds versus 1.47 seconds for the comparison method. The authors suggest that if a memory pipeline calls an LLM for routing, scoring, or stop decisions, it can switch to a fast classifier and reserve the LLM for the final answer.
Why read this
AI score 53
The original text provides score and runtime comparisons on the LoCoMo benchmark, from which readers can judge the benefits of delegating small decisions to a fast classifier.
Topics
Body
University of Texas paper finds that handing small memory decisions to a fast scoring model let an AI memory system build its memory of long conversations 6.6 times faster
Many memory systems ask an LLM to make small choices, like what to link or when to stop searching.
Generating text for each choice is slow, and those calls pile up during memory building and search.
Jev-Mem gives those choices to a fast scorer, Jev, that returns probabilities in batches instead of text. The LLM is called only at the end, to read the chosen memories and answer.
On the LoCoMo long-conversation benchmark, it got a 0.777 answer score versus 0.700 for the best baseline. It built memory in 158 seconds versus 1,044 for the fastest rival and answered in 0.93 seconds versus 1.47.
If your memory pipeline calls an LLM for routing, scoring, or stopping, try a fast classifier there and save the LLM for answers.