RecSA Extrapolates 32K Training to 2M Tokens at Logarithmic Cost
UCSD Hao AI Lab introduced RecSA, which builds a tree over the KV cache and searches it level by level, making the cost of each query scale as O(log L) and enabling a model trained only on 32K sequences to extrapolate to 2M tokens. After training Qwen3-4B on 0.62B tokens, RecSA outperformed all baselines on RULER at lengths from 131K to 2M, scoring 60.5 at 1M versus 40.0 for the strongest baseline, Dense + DCA. On a single H200, RecSA delivered a 3.2× prefill speedup and a 3.8× decoding speedup at 524K. The code includes Triton and CUDA kernels, Qwen3 integration, and evaluation tools.
Why read this
AI score 56
The study identifies candidate local positional encoding as the key constraint in long-context extrapolation and presents a transferable design approach through cross-length benchmarks and latency data.
Topics
Body
Past its training length, a model meets relative distances it never saw in training, and YaRN or DCA only delay the failure. RecSA sets each entry's RoPE position to its index in the candidate list of its level, and the query takes the last index. However long the context, every relative distance stays below 4K, so training at 32K already covers all of them. Trained the same way with standard RoPE, the model fails right after 32K; with YaRN, it fails by 524K. Candidate-local RoPE keeps most of its score at 1M.
AI Trends