Hao AI Lab proposes JetSpec++: pipelining draft generation and verification, achieving 1.54x speedup on Qwen3-8B
Hao AI Lab proposes JetSpec++, which pipelines draft generation and verification through causal parallel drafting, a hidden-state predictor, and an anchor-token decoder, achieving lossless speedup on Qwen3-8B and Qwen3.8-27B. Compared with DSpark, it achieves up to 1.54x end-to-end speedup, and 7.45x speedup over autoregressive decoding, reaching 1,361 TPS and 367 TPS respectively under BF16; with NVFP4 and kernel optimizations, it exceeds 4,000 TPS and 900 TPS on two GB200s. Across 7 benchmarks, it accepts 8.94 tokens per round, compared with 6.89 for DSpark; the model, code, and inference engine have all been open-sourced.
Why read this
AI score 54
The original text provides end-to-end speedup comparisons with DSpark and the number of accepted tokens per round, which can be used to assess the actual benefits of pipelining speculative decoding.
Topics
Body
[3/5] The rest of spare time waiting for verification outcome now goes into refining a better draft. While the target verifies, JetSpec++ corrects its own parallel draft: it redrafts on its own guesses, refreshes its predicted context, and fixes the first token where a decoder built from the target's last layers disagrees.
8.94 accepted tokens per round across 7 benchmarks, vs 6.89 for DSpark.