AI Trends

RecSA Extrapolates 32K Training to 2M Tokens with Logarithmic Query Cost

UCSD Hao AI LabUpdated Nov 19 09:02

AI summary

By building a tree-based index over the KV cache, RecSA keeps each query’s cost at O(log L), enabling a Qwen3-4B model trained only on 32K sequences to extrapolate to 2M tokens. The model was trained on 0.62B tokens and scored 60.5 on RULER at a 1M context length, outperforming Dense + DCA’s 40.0. On a single H200 at a length of 524K, RecSA accelerated prefill and decoding by 3.2× and 3.8×, respectively; blog: https://t.co/yKpRitz8zw, paper: https://t.co/cWiIZS40Cy, code: https://t.co/qAFANlalhH.

Why read this

AI score 55

The study organizes the KV cache into a hierarchical tree and uses benchmark and latency data to demonstrate performance when extrapolating 32K training to a 2M-token context.

Topics

Tags

Body

RecSA: Million-Token Extrapolation at Logarithmic Cost 🚀

RecSA builds a tree over the KV cache and lets each query search it level by level, so the cost per query grows as O(log L) and a model trained only at 32K extrapolates to 2M tokens.

Initialized from Qwen3-4B and trained on 0.62B tokens, all at 32K, RecSA outperforms all baselines on RULER from 131K to 2M, scoring 60.5 at 1M against 40.0 for the strongest baseline (Dense + DCA). On one H200, it speeds up prefill by 3.2× and decoding by 3.8× at 524K.

Blog: https://t.co/yKpRitz8zw Paper: https://t.co/cWiIZS40Cy Code: https://t.co/qAFANlalhH

Source

UCSD Hao AI Lab