Search “Reasoning”

Hao AI Lab open-sources the JetSpec++ model, code, and inference engine

On October 31, 2026, the UCSD Hao AI Lab announced that it would fully open-source the JetSpec++ model, code, and inference engine, along with links to the paper, code, and Hugging Face. This speculative decoding framework pipelines draft generation and verification through causal parallel drafting, a hidden-state predictor, and an anchor-token decoder. According to the data it published, on Qwen3-8B, JetSpec++ achieves 1,361 TPS, delivering up to 1.54x end-to-end speedup relative to DSpark and 7.45x relative to autoregressive decoding, while remaining lossless; after combining NVFP4 with kernel optimizations, on two GB200 GPUs, Qwen3-8B exceeds 4,000 TPS and Qwen3.8-27B exceeds 900 TPS. There are currently no further developments.

Hao AI Lab open-sources the JetSpec++ model, code, and inference engine

On October 31, 2026, the UCSD Hao AI Lab announced that it would fully open-source the JetSpec++ model, code, and inference engine, along with links to the paper, code, and Hugging Face. This speculative decoding framework pipelines draft generation and verification through causal parallel drafting, a hidden-state predictor, and an anchor-token decoder. According to the data it published, on Qwen3-8B, JetSpec++ achieves 1,361 TPS, delivering up to 1.54x end-to-end speedup relative to DSpark and 7.45x relative to autoregressive decoding, while remaining lossless; after combining NVFP4 with kernel optimizations, on two GB200 GPUs, Qwen3-8B exceeds 4,000 TPS and Qwen3.8-27B exceeds 900 TPS. There are currently no further developments.

View story:Hao AI Lab open-sources the JetSpec++ model, code, and inference engine

DeepSeek open-sources the inference engine DeepInfer on GitHub

On October 29, 2026, DeepSeek open-sourced the inference engine DeepInfer on GitHub; this engine is the inference engine behind DeepSeek-V4.1-Flash. Leiphone reported the same day that DeepInfer includes technologies such as MoE expert parallelism, asynchronous batch scheduling, custom kernels, and speculative decoding, and disclosed that a single cluster needs to process approximately 155 trillion tokens per day. Its Zero-Overhead Batch Scheduler lets the CPU schedule computation one round in advance, overlapping with GPU forward passes, and handles cross-round state misalignment through total_inflight_tokens accounting, Remap mapping, and DeferredRelease deferred reclamation. Currently, only offline inference capabilities are open; components such as online KV Cache storage and cross-replica load balancing have not yet been fully disclosed, and the official team has not separately published end-to-end speedup data for this scheduling mechanism relative to synchronous solutions.

vLLM Blog Publishes Technical Retrospective on Kimi K3 Inference Optimization

On October 16, 2026, the vLLM blog published a technical retrospective titled "Hyper Optimizing Kimi K3 in vLLM: Moving Fast and Responsibly." According to a TMTPost·AGI report, NVIDIA CentML engineer Benjamin Chislett used a Megakernel written in 12,000 lines of CuTe DSL to optimize inference for Moonshot AI's Kimi K3; the vLLM blog reports over 840 tokens/s at batch 1 and a 1.5x to 2x improvement in decoding throughput across multiple configurations, while SemiAnalysis estimates 423 tok/s/user on GB300 NVL72, a 1.68x improvement. The vLLM blog states that agent-optimized inference code is now outpacing human review and maintenance capacity, and the team therefore proposed a fast-track model optimization RFC so that reviewers judge designs from high-level abstractions rather than reviewing code line by line. The report also mentions that NVIDIA is developing optimizations for other models in vLLM, such as DeepSeek V4.1.

Guillermo Rauch announces that Vercel is partnering with Particle to provide models and inference

On October 29, 2026, Guillermo Rauch stated that Vercel has reached a partnership with Suhail to provide Particle's models and inference services through the Vercel AI Gateway. He said Particle's goal is to make intelligence as ubiquitous and abundant as electricity, and the two sides will work together to achieve this goal for developers and enterprises. Currently, this partnership has only this one public statement, and no subsequent progress has been disclosed.

Guillermo Rauch announces that Vercel is partnering with Particle to provide models and inference

On October 29, 2026, Guillermo Rauch stated that Vercel has reached a partnership with Suhail to provide Particle's models and inference services through the Vercel AI Gateway. He said Particle's goal is to make intelligence as ubiquitous and abundant as electricity, and the two sides will work together to achieve this goal for developers and enterprises. Currently, this partnership has only this one public statement, and no subsequent progress has been disclosed.

View story:Guillermo Rauch announces that Vercel is partnering with Particle to provide models and inference

vLLM Blog Publishes Technical Retrospective on Kimi K3 Inference Optimization

On October 16, 2026, the vLLM blog published a technical retrospective titled "Hyper Optimizing Kimi K3 in vLLM: Moving Fast and Responsibly." According to a TMTPost·AGI report, NVIDIA CentML engineer Benjamin Chislett used a Megakernel written in 12,000 lines of CuTe DSL to optimize inference for Moonshot AI's Kimi K3; the vLLM blog reports over 840 tokens/s at batch 1 and a 1.5x to 2x improvement in decoding throughput across multiple configurations, while SemiAnalysis estimates 423 tok/s/user on GB300 NVL72, a 1.68x improvement. The vLLM blog states that agent-optimized inference code is now outpacing human review and maintenance capacity, and the team therefore proposed a fast-track model optimization RFC so that reviewers judge designs from high-level abstractions rather than reviewing code line by line. The report also mentions that NVIDIA is developing optimizations for other models in vLLM, such as DeepSeek V4.1.

View story:vLLM Blog Publishes Technical Retrospective on Kimi K3 Inference Optimization