Hao AI Lab Open-Sources the JetSpec++ Speculative Decoding Framework
UCSD Hao AI Lab has open-sourced JetSpec++, which pipelines the draft stage and verification through causal parallel drafting, a hidden-state predictor, and an anchor-token decoder, achieving 1,361 TPS on Qwen3-8B, up to 1.54x end-to-end speedup over DSpark, and 7.45x speedup over autoregressive decoding while remaining lossless. Combined with NVFP4 and kernel optimizations, Qwen3-8B exceeds 4,000 TPS and Qwen3.8-27B exceeds 900 TPS (on two GB200 GPUs). The model, code, and inference engine have all been open-sourced.
Body
Introducing JetSpec++: we find causal parallel drafting paired with hidden states predictor and anchor token decoder can effectively draft the next future block before the current one is verified, and spending the freed time on self refinement.
This allows drafting to be pipelined with verification, hiding almost all of its drafting cost.
JetSpec++ reaches up to 1.54x end-to-end speedup over DSpark and 7.45x over autoregressive decoding while keeping lossless, at 1,361 TPS on Qwen3-8B and 367 TPS on Qwen3.8-27B in BF16.
With NVFP4 and kernel optimizations, JetSpec++ further reaches over 4,000 TPS on Qwen3-8B and over 900 TPS on Qwen3.8-27B, on two GB200 GPUs. ⚡️
Project page: https://t.co/tuhf2KPfsW