RecSA以对数成本将32K训练模型外推至2M tokens
Sparse attention such as MoBA and DSA still scores the whole context for every query, so selection gets slower as the context grows. In RecSA, each node of the
UCSD Hao AI Lab提出RecSA,通过在KV cache上构建树并逐层搜索,使单次查询成本按O(log L)增长,并将仅在32K训练的Qwen3-4B外推至2M tokens。RecSA使用0.62B tokens训练,在RULER的131K至2M范围均领先基线,1M时得分60.5,对比最强基线Dense + DCA的40.0。单张H200上,其在524K时将prefill加速3.2×、解码加速3.8×,代码已开源,包含Triton和CUDA内核、Qwen3集成、RULER评测及延迟基准。
正文
Sparse attention such as MoBA and DSA still scores the whole context for every query, so selection gets slower as the context grows. In RecSA, each node of the tree has 16 children. A query starts at the top level, scores at most 4K candidates per level and selects 256 of them to descend into, so the tree grows by one level each time the context grows 16×: 2 levels up to 64K tokens, 4 at 2M. Nodes the query does not select are not dropped. They enter the same softmax as summaries, next to the exact tokens, so every query covers the whole context and needs no sink or local window. At 2M, a query reads about 12K entries, roughly twice as many as at 32K.
AI 热点