Understand this story
On October 25, 2026, researchers from Stanford and UC Berkeley released TokenGate, a learnable token pruning module for long-context prefilling. According to the team, the method adds a small MLP gate and LoRA (r=16) on Qwen3-1.7B or Qwen3.5-2B, borrowing from the MoE routing approach to make top-K selection end-to-end trainable, pruning only the document portion and not the question portion. The team reports a 1.4x to 11.3x improvement in first-token latency with no accuracy loss, and a 1.5x to 3.4x reduction in peak prefilling memory; the tested context is up to 128k tokens, and on NIAH 16k it achieves an 11.3x speedup and a 3.42x memory reduction. The code and checkpoints are open-sourced on GitHub under Apache-2.0. There are currently no subsequent developments.
AI-generated from public reports
Developments
1 confirmed developments
- Source:MarkTechPost
Reports and discussion timeline
Browse reports and discussion by time. Their evidence levels remain distinct.
Oct 25
MarkTechPost
Stanford and UC Berkeley propose TokenGate: token-gated attention makes long-context prefill up to 11.3x faster
Researchers from Stanford and UC Berkeley have released TokenGate, a learnable token pruning module for long-context prefill. The team reports a 1.4x to 11.3x improvement in time to first token (TTFT) with no accuracy loss, and a 1.5x to 3.4x reduction in peak prefill memory. The method adds a small MLP gate and LoRA (r=16) on top of Qwen3-1.7B or Qwen3.5-2B, and uses an MoE routing approach to make top-K selection end-to-end trainable; only the document portion can be pruned, while the question portion is not pruned. The code and checkpoints are open-sourced on GitHub under Apache-2.0, with test contexts up to 128k tokens; on NIAH 16k it achieves an 11.3x speedup and a 3.42x memory reduction.