UCSD Hao AI Lab 发布 RecSA,以对数成本实现百万 token 上下文外推
On one H200 with batch size 1, against dense FlashAttention in the same framework:
UCSD Hao AI Lab 提出的 RecSA 在 KV cache 上构建树状结构,使单次查询成本按 O(log L) 增长,并让仅以 32K 序列训练的 Qwen3-4B 外推至 2M tokens。RecSA 使用 0.62B tokens 训练,在 RULER 的 131K 至 2M 长度均领先基线,1M 时得分 60.5,最强基线 Dense + DCA 为 40.0。单张 H200 上,RecSA 在 524K 时将预填充加速 3.2×、解码加速 3.8×,代码已开源并包含 Triton、CUDA 内核及评测实现。
正文
On one H200 with batch size 1, against dense FlashAttention in the same framework: • Prefill at 524K takes 80 s, against 257 s. • Decoding stays at about 6 ms per token from 131K to 524K, while dense attention grows from about 9 to 23 ms. • At 1M and 2M, with the exact tokens offloaded to host memory, RecSA decodes at 8 and 11 ms per token; dense attention takes 177 ms and 2.9 s. We write the kernels in Triton and CUDA, and e
AI 热点