RecSA Research Scales a Model Trained on 32K to 2M Tokens at Logarithmic Cost
RecSA from UCSD Hao AI Lab builds a tree structure over the KV cache, making the cost of each query grow as O(log L) and enabling a model trained only on 32K contexts to extrapolate to 2M tokens. Initialized from Qwen3-4B and trained on 0.62B tokens, the method scores 60.5 on RULER's 1M test, compared with 40.0 for the strong Dense + DCA baseline. On a single H200, it speeds up 524K prefill and decoding by 3.2× and 3.8×, respectively; the code is open source and includes kernels, Qwen3 integration, RULER evaluation, and latency benchmarks: https://t.co/qAFANlalhH
Why read this
AI score 56
The study demonstrates how a tree-structured KV cache can scale long contexts at logarithmic cost, with RULER and H200 latency data showing gains over the baselines.
Topics
Body
This project is done by our amazing @skywong265979 @Xinyu2ML @AlexZhang152005, and our advisor @haozhangml. The code is open source, including the kernels, the Qwen3 integration, RULER evaluation and latency benchmarks.
Blog: https://t.co/yKpRitz8zw Paper: https://t.co/cWiIZS40Cy Code: https://t.co/qAFANlalhH
AI Trends