AI Trends

UCSD Hao AI Lab Releases RecSA, Enabling Million-Token Context Extrapolation at Logarithmic Cost

UCSD Hao AI LabUpdated Nov 19 09:02

AI summary

RecSA, proposed by UCSD Hao AI Lab, builds a tree structure over the KV cache, allowing the cost of each query to grow as O(log L) and enabling Qwen3-4B, trained only on 32K sequences, to extrapolate to 2M tokens. RecSA was trained on 0.62B tokens and outperformed the baselines on RULER at every length from 131K to 2M, scoring 60.5 at 1M versus 40.0 for the strongest baseline, Dense + DCA. On a single H200, RecSA accelerated prefill by 3.2× and decoding by 3.8× at 524K. The code has been open-sourced and includes Triton and CUDA kernels as well as evaluation implementations.

Why read this

AI score 57

RecSA demonstrates how a tree-structured KV cache can reduce the cost of long-context queries to a logarithmic level while balancing extrapolation performance and inference latency.

Topics

Tags

Body

On one H200 with batch size 1, against dense FlashAttention in the same framework: • Prefill at 524K takes 80 s, against 257 s. • Decoding stays at about 6 ms per token from 131K to 524K, while dense attention grows from about 9 to 23 ms. • At 1M and 2M, with the exact tokens offloaded to host memory, RecSA decodes at 8 and 11 ms per token; dense attention takes 177 ms and 2.9 s. We write the kernels in Triton and CUDA, and e

Source

UCSD Hao AI Lab