清华 TokenRouter：token 级 LLM 路由吞吐最高提升 64.15 倍
New Tsinghua paper builds TokenRouter, a serving system that runs per-token small-and-large model routing at upto 64.15X the throughput of existing setups.
清华大学论文提出 TokenRouter，一个在 token 级别对大小模型做路由的服务系统，吞吐量最高达现有方案的 64.15 倍。它让每个模型独占服务器并互相传递半成品答案，同时保留 KV cache，并短暂挂起请求以凑成更大批次。在 5 种路由方法上，吞吐量相比更强的现有方案提升 2.01 至 64.15 倍。
正文
当前显示原文，中文全文译文尚未就绪。
New Tsinghua paper builds TokenRouter, a serving system that runs per-token small-and-large model routing at upto 64.15X the throughput of existing setups.
Current popular serving frameworks (like vLLM and SGLang) run one model per request, so when two models share an answer, every step waits for the slower one.
TokenRouter gives each model its own server and lets them pass work back and forth. So, it hands a half-written answer between them while keeping the model’s memory of the text so far (the KV cache), and holds requests for a moment so each model works on bigger batches.
Across 5 routing methods, throughput rose 2.01 to 64.15 times over the stronger existing setup.
– arxiv. org/abs/2610.12242