Understand this story
On October 22, 2026, researchers from the Stanford AI Lab and Berkeley Sky proposed a pluggable small module called TokenGate, which is inserted into the shallow layers of existing LLMs and only passes the important top-K tokens on to the upper layers. The paper reports that without any loss in accuracy, time to first token is 1.4–11.3 times faster and peak memory usage is reduced by 1.5–3.4 times; compared with the best training-free pruning method, it is up to about 6.4 times faster and 2 times more memory-efficient. Training uses the ordinary next-token loss, with a cost close to that of LoRA, and after a single fine-tuning it can generalize to different input lengths and different K. The work is currently publicly available as a paper, with no subsequent progress yet.
AI-generated from public reports
Developments
1 confirmed developments
- Source:Shao Meng
Reports and discussion timeline
Browse reports and discussion by time. Their evidence levels remain distinct.
Oct 22
Shao Meng
Researchers from Stanford and Berkeley propose TokenGate, using learnable pruning to speed up long-context Prefill by up to about 11x
Researchers from the Stanford AI Lab and Berkeley Sky propose a pluggable small module, TokenGate, which is inserted into the shallow layers of an existing LLM and forwards only the important top-K tokens to the upper layers. The paper reports that with no loss in accuracy, time to first token is 1.4–11.3× faster and peak memory usage is 1.5–3.4× lower; compared with the best training-free pruning method, it is up to about 6.4× faster and 2× more memory-efficient. Training uses the ordinary next-token loss, with a cost close to LoRA, and after a single fine-tuning it generalizes to different input lengths and different K.