vLLM 的原生速度 Transformers 建模后端

Showcase

We put the transformers modeling backend for vLLM head to head with vLLM's hand written native implementations across three very different Qwen3 models:

4B dense model on a single GPU

32B dense model on tensor parallelism

235B-parameter FP8 Mixture-of-Experts on data + expert parallelism on the same 8×H100 node

Running any* Hugging Face model through the transformers modeling backend is a single flag — --model-impl transformers . It composes with the usual parallelism options, so nothing about your serving setup changes:

# Qwen3-4B dense, single GPU vllm serve Qwen/Qwen3-4B --model-impl transformers

# Qwen3-32B dense, tensor-parallel across 2 GPUs vllm serve Qwen/Qwen3-32B --model-impl transformers --tensor-parallel-size 2

# Qwen3-235B-A22B-FP8 MoE, data-parallel + expert-parallel across 8 GPUs vllm serve Qwen/Qwen3-235B-A22B-FP8 --model-impl transformers --data-parallel-size 8 --enable-expert-parallel # add --max-model-len 8192 if your node is memory constrained

*Models that use linear attention are not currently supported, but they will be soon! Custom models where the code lives in a Hub repo are unlikely to work as they will not have been written compliantly.

How we measured

Each model is compared under three conditions that are identical in every way except the code path:

native — --model-impl vllm , vLLM's hand-written model (the bar to match)

after — --model-impl transformers with the PR

before — --model-impl transformers without the PR

The full, reproducible runner is available as a gist: benchmark.sh

So, what's new?

The transformers modeling backend for vLLM used to focus on attention as the bottleneck for inference. By plugging vLLM’s attention implementation at runtime, we could make a transformers model run efficiently inside the vLLM engine. But there are many dimensions to deployments that only a custom port can target to extract maximum inference performance. Parallelization across GPUs, compilation, fused kernels, and many more, all contribute to leveraging your hardware to achieve ultra-fast inference.

When model authors wanted the absolute best performance, they were still writing custom vLLM implementations.

The latest iteration of the transformers modeling backend for vLLM dynamically applies inference specific layer fusions at runtime to match the speed of custom code implementations, for compatible architectures.

How does it work?

The transformers modeling backend for vLLM now uses torch.fx to perform static analysis on the model’s graph. This process searches for known patterns that can be optimised. After any patterns have been identified, it uses ast (abstract syntax tree) to manipulate the source code and rewrite some of the operations in place.

What can we achieve with this?

Fused operations that are many-to-one mapped to (ultra) optimized vLLM kernels, such as the ones used for Expert Parallelization (EP) in Mixture-of-Experts (MoE) models.

The main other fused operations are vLLM's MergedColumnParallelLinear and QKVParallelLinear . These blocks allow us to infer parallel plans for TP (tensor-parallel). PP (pipeline-parallel) plans can also be inferred if the decoder block list is easily identifiable.

The manipulated models are still fully (torch) compilable, being passed through torch.compile and CUDA Graphs, just the same as a dedicated vLLM model implementation.

Unlike vLLM model implementations, Transformers model implementations can be used in training . So you can use the same model code for training/evals/RL rollouts.

As shown above, this results in native vLLM inference speed for compatible models, without having to write a single line of code to optimize the model for inference.

We are in the process of writing a detailed blog post to dive deep inside these optimized inference methods and explain in detail how we manipulate the model to adapt to them.

Transformers model definition

Transformers modeling backend in vLLM

Datasets mentioned in this article 1

ariG23498/useful-scripts Updated Jul 8 • 18 • 5 More Articles from our Blog

Unlocking asynchronicity in continuous batching

68 May 14, 2026 mixture-of-experts optimization transformers Mixture of Experts (MoEs) in Transformers +3 179 February 26, 2026 Community One provenance implication here is that the model definition and the executed inference graph are no longer the same artifact boundary. \n If the runtime rewrites model code and replaces modules with fused kernels, deployment evidence probably needs to preserve both the source model identity and the transformation/runtime path. \n","updatedAt":"2026-08-10T09:26:19.433Z","author":{"_id":"6a6b7e7bd0857a27c33bf3c7","avatarUrl":"/avatars/cfbae10de097d7c68321c26eb6755d5f.svg","fullname":"200lz","name":"200lz","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.9137899875640869},"editors":["200lz"],"editorAvatarUrls":["/avatars/cfbae10de097d7c68321c26eb6755d5f.svg"],"reactions":[],"isReport":false}}],"status":"open","isReport":false,"pinned":false,"locked":false,"collection":"community_blogs"},"contextAuthors":["hmellor","lysandre"],"primaryEmailConfirmed":false,"discussionRole":0,"acceptLanguages":["en","zh"],"withThread":true,"cardDisplay":false,"repoDiscussionsLocked":false,"hideComments":true}"> 200lz Aug 10 One provenance implication here is that the model definition and the executed inference graph are no longer the same artifact boundary.

If the runtime rewrites model code and replaces modules with fused kernels, deployment evidence probably needs to preserve both the source model identity and the transformation/runtime path.

Reply Edit Preview Upload images, audio, and videos by dragging in the text input, pasting, or clicking here . Tap or paste here to upload images Comment · Sign up or log in to comment

+65

Datasets mentioned in this article 1

ariG23498/useful-scripts