AI Trends

RecSA Extrapolates a 32K-Trained Model to 2M Tokens at Logarithmic Cost

UCSD Hao AI LabUpdated Nov 19 09:02

AI summary

UCSD Hao AI Lab introduced RecSA, which builds a tree over the KV cache and searches it level by level, making the cost of each query scale as O(log L) and extrapolating Qwen3-4B, trained only on 32K contexts, to 2M tokens. RecSA was trained on 0.62B tokens and outperformed the baselines across RULER context lengths from 131K to 2M, scoring 60.5 at 1M versus 40.0 for the strongest baseline, Dense + DCA. On a single H200, it delivered a 3.2× prefill speedup and a 3.8× decoding speedup at 524K. The code has been open-sourced and includes Triton and CUDA kernels, Qwen3 integration, RULER evaluation, and latency benchmarks.

Why read this

AI score 56

RecSA reduces query cost to O(log L) through a tree-structured KV cache, providing a reusable engineering approach for ultra-long-context inference.

Topics

Tags

Body

Sparse attention such as MoBA and DSA still scores the whole context for every query, so selection gets slower as the context grows. In RecSA, each node of the tree has 16 children. A query starts at the top level, scores at most 4K candidates per level and selects 256 of them to descend into, so the tree grows by one level each time the context grows 16×: 2 levels up to 64K tokens, 4 at 2M. Nodes the query does not select are not dropped. They enter the same softmax as summaries, next to the exact tokens, so every query covers the whole context and needs no sink or local window. At 2M, a query reads about 12K entries, roughly twice as many as at 32K.

Source

UCSD Hao AI Lab