AI Trends

RecSA Research Scales a Model Trained on 32K to 2M Tokens at Logarithmic Cost

UCSD Hao AI LabUpdated Nov 19 09:01

AI summary

RecSA from UCSD Hao AI Lab builds a tree structure over the KV cache, making the cost of each query grow as O(log L) and enabling a model trained only on 32K contexts to extrapolate to 2M tokens. Initialized from Qwen3-4B and trained on 0.62B tokens, the method scores 60.5 on RULER's 1M test, compared with 40.0 for the strong Dense + DCA baseline. On a single H200, it speeds up 524K prefill and decoding by 3.2× and 3.8×, respectively; the code is open source and includes kernels, Qwen3 integration, RULER evaluation, and latency benchmarks: https://t.co/qAFANlalhH

Why read this

AI score 56

The study demonstrates how a tree-structured KV cache can scale long contexts at logarithmic cost, with RULER and H200 latency data showing gains over the baselines.

Topics

Tags

Body

This project is done by our amazing @skywong265979 @Xinyu2ML @AlexZhang152005, and our advisor @haozhangml. The code is open source, including the kernels, the Qwen3 integration, RULER evaluation and latency benchmarks.

Blog: https://t.co/yKpRitz8zw Paper: https://t.co/cWiIZS40Cy Code: https://t.co/qAFANlalhH

Source

UCSD Hao AI Lab