RecSA以对数成本将32K训练外推至2M token
Past its training length, a model meets relative distances it never saw in training, and YaRN or DCA only delay the failure. RecSA sets each entry's RoPE positi
UCSD Hao AI Lab提出RecSA,通过在KV cache上构建树并逐层搜索,使每次查询成本按O(log L)增长,让仅以32K序列训练的模型外推至2M token。基于Qwen3-4B并使用0.62B tokens训练后,RecSA在RULER的131K至2M长度上领先所有基线,1M时得分60.5,最强基线Dense + DCA为40.0。单张H200上,RecSA在524K时将prefill加速3.2×、解码加速3.8×,代码包含Triton和CUDA内核、Qwen3集成及评测工具。
正文
Past its training length, a model meets relative distances it never saw in training, and YaRN or DCA only delay the failure. RecSA sets each entry's RoPE position to its index in the candidate list of its level, and the query takes the last index. However long the context, every relative distance stays below 4K, so training at 32K already covers all of them. Trained the same way with standard RoPE, the model fails right after 32K; with YaRN, it fails by 524K. Candidate-local RoPE keeps most of its score at 1M.
AI 热点