Hao AI Lab open-sources JetSpec++, a speculative decoding method with up to 7.45x speedup
Hao AI Lab has open-sourced JetSpec++, which pipelines draft generation and verification through causal parallel drafting and a hidden-state predictor, achieving 1,361 TPS and 367 TPS on Qwen3-8B and Qwen3.8-27B respectively—up to 1.54x faster than DSpark and 7.45x faster than autoregressive decoding while remaining lossless. On Qwen3.8-27B, the number of tokens accepted per round improves by 22% over DSpark (6% for 8B); trained with only 16 blocks, it extrapolates to 32 draft blocks, reaching 14.55 (8B) and 14.05 (27B) tokens per round on MATH-500, where DSpark drops to 8.66 and 8.49. Combined with NVFP4 and kernel optimizations, Qwen3-8B exceeds 4,000 TPS and Qwen3.8-27B exceeds 900 TPS on two GB200s.
Body
[4/5] Bigger targets leave more room. On Qwen3.8-27B, hiding the draft saves less, because verification dominates the round. But the same long window fits three refinement passes instead of one, so the gain moves to acceptance: +22% accepted tokens per round over DSpark on 27B, vs +6% on 8B (GSM8K, as served).
It also extrapolates. Trained on blocks of 16, JetSpec++ drafts blocks of 32 and gains: 14.55 tokens per round on MATH-500 on 8B and 14.05 on 27B, while DSpark drops to 8.66 and 8.49.
We would love to hear your feedback!