Hao AI Lab Open-Sources the JetSpec++ Speculative Decoding Framework
UCSD Hao AI Lab has launched JetSpec++, which, through causal parallel drafting and pipelined verification, achieves lossless end-to-end speedups of 1.54× over DSpark and 7.45× over autoregressive decoding on Qwen3-8B, reaching 1,361 TPS in BF16. On GSM8K, Qwen3-8B's per-round latency drops from 9.92 ms with DSpark to 6.84 ms, with drafting time compressed from 3.76 ms to 0.61 ms. The model, code, and inference engine have all been open-sourced.
Body
[2/5] With JetSpec++, drafting time is almost removed from the critical path. Walltime reported below shows how latency reduces for each draft-verification round, and its breakdown on GSM8K: • Qwen3-8B: DSpark 9.92 ms (including 3.76 ms drafting time), JetSpec++ 6.84 ms (including 0.61 ms is drafting time) • Qwen3.8-27B: DSpark 31.88 ms (4.46 ms drafting time), JetSpec++ 28.47 ms (0.71 ms drafting time)
The key is JetSpec++ has effective foresight drafting: it guesses the verification outcome correctly in 92.9% (8B) and 89.0% (27B) of rounds, so the next block is already drafted when verification ends.