Hao AI Lab 开源 JetSpec++ 投机解码框架
Introducing JetSpec++: we find causal parallel drafting paired with hidden states predictor and anchor token decoder can effectively draft the next future block
UCSD Hao AI Lab 开源 JetSpec++，通过因果并行草稿、隐藏状态预测器和锚点 token 解码器，把草稿阶段与验证流水线化，在 Qwen3-8B 上实现 1,361 TPS、较 DSpark 最高 1.54 倍端到端加速、较自回归解码 7.45 倍加速且保持无损。结合 NVFP4 与 kernel 优化后，Qwen3-8B 超过 4,000 TPS、Qwen3.8-27B 超过 900 TPS（两块 GB200 GPU）。模型、代码与推理引擎已全部开源。
正文
Introducing JetSpec++: we find causal parallel drafting paired with hidden states predictor and anchor token decoder can effectively draft the next future block before the current one is verified, and spending the freed time on self refinement.
This allows drafting to be pipelined with verification, hiding almost all of its drafting cost.
JetSpec++ reaches up to 1.54x end-to-end speedup over DSpark and 7.45x over autoregressive decoding while keeping lossless, at 1,361 TPS on Qwen3-8B and 367 TPS on Qwen3.8-27B in BF16.
With NVFP4 and kernel optimizations, JetSpec++ further reaches over 4,000 TPS on Qwen3-8B and over 900 TPS on Qwen3.8-27B, on two GB200 GPUs. ⚡️
Project page: https://t.co/tuhf2KPfsW