Hao AI Lab open-sources the JetSpec++ model, code, and inference engine

Oct 31

1. 02:46

   UCSD Hao AI LabFirst-party

   Hao AI Lab open-sources JetSpec++, a speculative decoding method with up to 7.45x speedup

   Hao AI Lab has open-sourced JetSpec++, which pipelines draft generation and verification through causal parallel drafting and a hidden-state predictor, achieving 1,361 TPS and 367 TPS on Qwen3-8B and Qwen3.8-27B respectively—up to 1.54x faster than DSpark and 7.45x faster than autoregressive decoding while remaining lossless. On Qwen3.8-27B, the number of tokens accepted per round improves by 22% over DSpark (6% for 8B); trained with only 16 blocks, it extrapolates to 32 draft blocks, reaching 14.55 (8B) and 14.05 (27B) tokens per round on MATH-500, where DSpark drops to 8.66 and 8.49. Combined with NVFP4 and kernel optimizations, Qwen3-8B exceeds 4,000 TPS and Qwen3.8-27B exceeds 900 TPS on two GB200s.

   Expand summary
2. 02:46

   UCSD Hao AI LabFirst-party

   Hao AI Lab Open-Sources JetSpec++ Models, Code, and Inference Engine

   Hao AI Lab announced that JetSpec++'s models, code, and inference engine are all open-sourced, along with links to the paper, code, and Hugging Face. The method pipelines draft generation and verification, achieving up to 1.54x end-to-end speedup over DSpark and 7.45x over autoregressive decoding on Qwen3-8B, reaching 1,361 TPS in BF16; with NVFP4 and kernel optimizations, it exceeds 4,000 TPS on two GB200s.

   Expand summary
3. 02:46

   UCSD Hao AI LabFirst-party

   Hao AI Lab Open-Sources the JetSpec++ Speculative Decoding Framework

   UCSD Hao AI Lab has launched JetSpec++, which, through causal parallel drafting and pipelined verification, achieves lossless end-to-end speedups of 1.54× over DSpark and 7.45× over autoregressive decoding on Qwen3-8B, reaching 1,361 TPS in BF16. On GSM8K, Qwen3-8B's per-round latency drops from 9.92 ms with DSpark to 6.84 ms, with drafting time compressed from 3.76 ms to 0.61 ms. The model, code, and inference engine have all been open-sourced.

   Expand summary
4. 02:46

   UCSD Hao AI LabFirst-party

   Hao AI Lab proposes JetSpec++: pipelining draft generation and verification, achieving 1.54x speedup on Qwen3-8B

   Hao AI Lab proposes JetSpec++, which pipelines draft generation and verification through causal parallel drafting, a hidden-state predictor, and an anchor-token decoder, achieving lossless speedup on Qwen3-8B and Qwen3.8-27B. Compared with DSpark, it achieves up to 1.54x end-to-end speedup, and 7.45x speedup over autoregressive decoding, reaching 1,361 TPS and 367 TPS respectively under BF16; with NVFP4 and kernel optimizations, it exceeds 4,000 TPS and 900 TPS on two GB200s. Across 7 benchmarks, it accepts 8.94 tokens per round, compared with 6.89 for DSpark; the model, code, and inference engine have all been open-sourced.

   Expand summary
5. 02:46

   UCSD Hao AI LabFirst-party

   Hao AI Lab Open-Sources the JetSpec++ Speculative Decoding Framework

   UCSD Hao AI Lab has open-sourced JetSpec++, which pipelines the draft stage and verification through causal parallel drafting, a hidden-state predictor, and an anchor-token decoder, achieving 1,361 TPS on Qwen3-8B, up to 1.54x end-to-end speedup over DSpark, and 7.45x speedup over autoregressive decoding while remaining lossless. Combined with NVFP4 and kernel optimizations, Qwen3-8B exceeds 4,000 TPS and Qwen3.8-27B exceeds 900 TPS (on two GB200 GPUs). The model, code, and inference engine have all been open-sourced.

   Expand summary