AI 热点

RecSA以对数成本将32K训练外推至2M token

UCSD Hao AI Lab更新于 11月19日 09:02

Past its training length, a model meets relative distances it never saw in training, and YaRN or DCA only delay the failure. RecSA sets each entry's RoPE positi

AI 导读

UCSD Hao AI Lab提出RecSA,通过在KV cache上构建树并逐层搜索,使每次查询成本按O(log L)增长,让仅以32K序列训练的模型外推至2M token。基于Qwen3-4B并使用0.62B tokens训练后,RecSA在RULER的131K至2M长度上领先所有基线,1M时得分60.5,最强基线Dense + DCA为40.0。单张H200上,RecSA在524K时将prefill加速3.2×、解码加速3.8×,代码包含Triton和CUDA内核、Qwen3集成及评测工具。

推荐理由

AI 评分 56

该研究把长上下文外推的关键约束落到候选局部位置编码,并以跨长度基准与延迟数据呈现可迁移的设计思路。

主题

标签

正文

Past its training length, a model meets relative distances it never saw in training, and YaRN or DCA only delay the failure. RecSA sets each entry's RoPE position to its index in the candidate list of its level, and the query takes the last index. However long the context, every relative distance stays below 4K, so training at 32K already covers all of them. Trained the same way with standard RoPE, the model fails right after 32K; with YaRN, it fails by 524K. Candidate-local RoPE keeps most of its score at 1M.

来源

UCSD Hao AI Lab