UCSD Hao AI Lab Introduces RecSA, Scaling a Model Trained on 32K Sequences to 2M Tokens at Logarithmic Cost
UCSD Hao AI Lab introduced RecSA, which builds a tree over the KV cache and searches it layer by layer, making the cost per query grow as O(log L) and enabling a model trained only on 32K sequences to extrapolate to 2M tokens. RecSA was initialized from Qwen3-4B and trained on 0.62B tokens. It outperformed all baselines on RULER at context lengths ranging from 131K to 2M, scoring 60.5 at 1M versus 40.0 for the strongest baseline, Dense + DCA. On a single H200, RecSA achieved a 3.2× prefill speedup and a 3.8× decoding speedup at 524K. The project has open-sourced its code, kernels, Qwen3 integration, and evaluation tools.
Why read this
AI score 56
RecSA demonstrates a path to million-token extrapolation using only 32K-sequence training and provides comparative data on long-context quality and inference efficiency.
Topics
Body
Each node's key comes from a small pooler, about 25K parameters per attention layer. We first distill the pooler level by level from the frozen Qwen3-4B, then freeze it and train the model, on 32K sequences only. Every sparse method reads the same number of entries per query, and RecSA ranks first on RULER at every length from 131K to 2M. Its lead grows from about 3 points at 131K to about 20 at 1M, and it matches dense attention at 32K.
AI Trends