AI Trends

Trending story

RecSA Enables Models Trained on 32K Tokens to Extrapolate to 2M Tokens

1 reports1 story updatesUpdated Nov 19 03:20

Understand this story

Story summary

The English summary is being updated. See the reporting timeline below.

Latest development · Report

RecSA Extrapolates 32K Training to 2M Tokens with Logarithmic Query Cost

Developments

1 confirmed developments

  1. English translation is being prepared.

    Source:UCSD Hao AI Lab

Reports and discussion timeline

Browse reports and discussion by time. Their evidence levels remain distinct.

Nov 19

  1. UCSD Hao AI LabFirst-party

    RecSA Extrapolates 32K Training to 2M Tokens with Logarithmic Query Cost

    By building a tree-based index over the KV cache, RecSA keeps each query’s cost at O(log L), enabling a Qwen3-4B model trained only on 32K sequences to extrapolate to 2M tokens. The model was trained on 0.62B tokens and scored 60.5 on RULER at a 1M context length, outperforming Dense + DCA’s 40.0. On a single H200 at a length of 524K, RecSA accelerated prefill and decoding by 3.2× and 3.8×, respectively; blog: https://t.co/yKpRitz8zw, paper: https://t.co/cWiIZS40Cy, code: https://t.co/qAFANlalhH.

Related stories

  1. Related storiesRecSA Extrapolates Context to 2 Million Tokens with Logarithmic Query Cost
  2. Related storiesRecSA Achieves Extrapolation to 2M Tokens with Logarithmic Query Cost
  3. Related storiesRecSA Outperforms All Baselines in the RULER Long-Context Benchmark
  4. Related storiesRecSA Achieves Logarithmic Query Costs with a KV Cache Tree