AI 热点

RecSA以对数成本将32K训练模型外推至2M tokens

UCSD Hao AI Lab更新于 11月19日 09:02

Sparse attention such as MoBA and DSA still scores the whole context for every query, so selection gets slower as the context grows. In RecSA, each node of the

AI 导读

UCSD Hao AI Lab提出RecSA,通过在KV cache上构建树并逐层搜索,使单次查询成本按O(log L)增长,并将仅在32K训练的Qwen3-4B外推至2M tokens。RecSA使用0.62B tokens训练,在RULER的131K至2M范围均领先基线,1M时得分60.5,对比最强基线Dense + DCA的40.0。单张H200上,其在524K时将prefill加速3.2×、解码加速3.8×,代码已开源,包含Triton和CUDA内核、Qwen3集成、RULER评测及延迟基准。

推荐理由

AI 评分 56

RecSA通过树形KV cache将查询成本降至O(log L),为超长上下文推理提供了可复用的工程路径。

主题

标签

正文

Sparse attention such as MoBA and DSA still scores the whole context for every query, so selection gets slower as the context grows. In RecSA, each node of the tree has 16 children. A query starts at the top level, scores at most 4K candidates per level and selects 256 of them to descend into, so the tree grows by one level each time the context grows 16×: 2 levels up to 64K tokens, 4 at 2M. Nodes the query does not select are not dropped. They enter the same softmax as summaries, next to the exact tokens, so every query covers the whole context and needs no sink or local window. At 2M, a query reads about 12K entries, roughly twice as many as at 32K.

来源

UCSD Hao AI Lab