UCSD Hao AI Lab 提出 RecSA,以对数成本将 32K 训练模型外推至 2M token
Each node's key comes from a small pooler, about 25K parameters per attention layer. We first distill the pooler level by level from the frozen Qwen3-4B, then f
UCSD Hao AI Lab 提出 RecSA,通过在 KV cache 上构建树并逐层搜索,使单次查询成本按 O(log L) 增长,让仅以 32K 序列训练的模型外推至 2M token。RecSA 基于 Qwen3-4B 初始化并用 0.62B tokens 训练,在 RULER 的 131K 至 2M 长度上超过所有基线,1M 时得分 60.5,对比最强基线 Dense + DCA 的 40.0。单张 H200 上,RecSA 在 524K 时将 prefill 加速 3.2×、解码加速 3.8×,项目已开源代码、内核、Qwen3 集成及评测工具。
正文
Each node's key comes from a small pooler, about 25K parameters per attention layer. We first distill the pooler level by level from the frozen Qwen3-4B, then freeze it and train the model, on 32K sequences only. Every sparse method reads the same number of entries per query, and RecSA ranks first on RULER at every length from 131K to 2M. Its lead grows from about 3 points at 131K to about 20 at 1M, and it matches dense attention at 32K.
AI 热点