After DeepSeek Open-Sourced DeepInfer, vLLM and SGLang Update Inference Performance

It still has to be "Source God" Teacher D—so kind!

It had been less than a week since they brought out their own inference engine DeepInfer, and the open-source community has already been handing in their homework one after another.

Truly, one whale "falls" and all things rise; the first wave of open-source results is already fresh off the press:

Determined to learn from DeepSeek, become DeepSeek, surpass DeepSeek. (bushi)

Mature inference frameworks like vLLM and SGLang were the first to receive T0 upgrade packages—

Under their respective disclosed test configurations, performance is already close to, and in some parts even exceeding, DeepInfer.

Not bad, not bad! DeepInfer was initially built on vLLM, and its design also drew on SGLang's approach.

Now that DeepSeek has made its polished implementation public, the two are absorbing it and continuing to optimize.

These guys are the ones who have really turned open source into an art form.

There are also some personal developer projects inspired by DeepInfer that have been launching one after another, ranging from hardware adaptation and inference acceleration to deployment optimization, each with a different style.

This time, the results they've delivered have extended all the way from "catching up to DeepInfer" to full-rack deployment.

By integrating DeepSeek's open-source kernels and doing extensive optimization of the prefill and decode stages, vLLM's performance curve can now match DeepInfer's.

Under the same decoding configuration (131K context, 3 draft tokens per round), the throughput vLLM measured using GB200, across batch sizes from 8 to 256, is only 6% behind DeepInfer.

On the prefill side, throughput for newly added tokens also improved by 35.6%–43.9%.

Then scale deployment up to the NVIDIA GB300 NVL72 rack.

With prefill/decode disaggregation and expanded expert parallelism, vLLM's per-GPU throughput in the SemiAnalysis AgentX test reached as high as 954,000 tokens/s, 7.9 times the previous single-node result.

Same DeepSeek, and they really did squeeze more speed out of it...

So where did these improvements come from? The team left almost no detail untouched.

For example, when processing input, sparse attention needs to filter out the content worth attending to from a batch of candidates, which involves top-k selection.

A previous approach would first convert the BF16 scores output by DeepGEMM into FP32, then do the filtering.

The new implementation has DeepSelect process BF16 data directly, eliminating that intermediate conversion.

Launching one less computation may look like a small change, but in the corresponding operator tests, this top-k processing can reach about 3.7 times the original speed.

The vLLM team also separately changed a CPU communication method, using recursive doubling and similar techniques to exchange information and reduce the number of synchronization rounds required.

For the small-batch decoding commonly used by Agents, vLLM also chose to separately support DSpark draft generation and use a full CUDA Graph to execute a dedicated decoding path.

SGLang's answer, delivered right after, went further, covering both the input and output stages.

SGLang tested prefill on 4 B300s and 4 GB300s respectively, with peak throughput about 4% higher than DeepInfer.

For the decoding test, it likewise used 64 GB300s.

At 256 requests per GPU, DeepInfer took 31.19 ms per step while SGLang took 28.88 ms, shortening single-step time by about 7.4%; at 8 requests per GPU, the two were basically on par.

In the specific work, SGLang first integrated the Mega kernels (MegaMoE, MegaAttention, Mega-mHC, MegaGate), SWA bounded replay, and Engram optimization.

After reaching performance comparable to DeepInfer, it continued iterating and introduced more kernel fusion schemes.

It then enabled efficient prefill by supporting splittable CUDA graphs.

Ultimately, compared with the open-source version of DeepInfer, SGLang is more complete, adding key serving components, including the KV cache system, the prefill/decode disaggregation architecture, and the load-balancing module.

It is worth mentioning that, for a more rigorous comparison, SGLang also reran DeepInfer on GB300 and B300.

The result: you'd never know unless you ran it—DeepInfer's throughput under some configurations was even higher than the previously published results...

He had been developing an experimental building-block library called Resin for building large-model inference engines.

It turns the parts used by multiple models into a common foundation, such as the scheduler and KV cache management. Users then fill in the computation implementations needed for specific models and specific hardware, and can assemble an independent engine.

Eric said that in the past week, inspired by DeepInfer, he successfully got many features and parallelism mechanisms running.

According to test results, on DeepSeek-V4.1-Flash, Resin's performance is about 5%–10% behind DeepInfer, while also providing features such as HTTP serving, prefix caching, and metrics.

Then he paired this foundation with a Skill.

The task is to teach AI how to implement models based on Resin, then put Codex on it to work overnight.

According to his disclosure, the Agent completed a GLM-5.3-Flash engine supporting MTP; under the B200, EP4 configuration, input throughput reached up to 60,000 tokens/s and output throughput up to 16,000 tokens/s.

And! Resin remains open source.

Tearing up, big fat fish, this open-source golden age is just as you wished.

DeepInfer came from open source and returns to open source. Mature frameworks continue to integrate and optimize, and individual developers can also absorb its design and carry open source forward.

So, big fat fish is the one who truly grasped the essence of sustainable open source~

This article is from the WeChat public account "Quantum Bit", author: Focus on Frontier Technology, published by 36Kr with authorization.