vLLM 如何集成 Transformers 建模后端

Updates; Support for Vision Language Models (21st July 2025); Transformers and vLLM: Inference in Action; vLLM’s Deployment Superpower: OpenAI Compatibility; Why do we need the Transformers modeling backend?; Case Study: Helium

The Hugging Face Transformers library offers a flexible, unified interface to a vast ecosystem of model architectures. From research to fine-tuning on custom dataset, Transformers is the go-to toolkit for all.

But when it comes to deploying these models at scale, inference speed and efficiency often take center stage. Enter vLLM, a library engineered for high-throughput inference, pulling models from the Hugging Face Hub and optimizing them for production-ready performance.

A recent addition to the vLLM codebase enables leveraging Transformers as a backend for model implementations. vLLM will therefore optimize throughput/latency on top of existing Transformers architectures. In this post, we’ll explore how vLLM leverages the Transformers modeling backend to combine flexibility with efficiency, enabling you to deploy state-of-the-art models faster and smarter.

Updates

This section will hold all the updates that have taken place since the blog post was first released (11th April 2025).

vLLM with the Transformers modeling backend now supports Vision Language Models. When user adds model_impl="transformers", the correct class for text-only and multimodality will be deduced and loaded.

Here is how one can serve a multimodal model using the Transformers modeling backend.

To consume the model one can use the openai API like so:

You can also directly initialize the vLLM engine using the LLM API. Here is the same model being served using the LLM API.

Transformers and vLLM: Inference in Action

Let’s start with a simple text generation task using the meta-llama/Llama-3.2-1B model to see how these libraries stack up.

Infer with Transformers

The transformers library shines in its simplicity and versatility. Using its pipeline API, inference is a breeze:

This approach is perfect for prototyping or small-scale tasks, but it’s not optimized for high-volume inference or low-latency deployment.

Infer with vLLM

vLLM takes a different track, prioritizing efficiency with features like PagedAttention (a memory-efficient attention mechanism) and dynamic batching. Here’s the same task in vLLM:

vLLM’s inference is noticeably faster and more resource-efficient, especially under load. For example, it can handle thousands of requests per second with lower GPU memory usage.

vLLM’s Deployment Superpower: OpenAI Compatibility

Beyond raw performance, vLLM offers an OpenAI-compatible API, making it a drop-in replacement for external services. Launch a server:

Then query it with curl:

Or use Python’s OpenAI client:

This compatibility slashes costs and boosts control, letting you scale inference locally with vLLM’s optimizations.

Why do we need the Transformers modeling backend?

The Transformers library is optimized for contributions and addition of new models. Adding a new model to vLLM on the other hand is a little more involved.

In the ideal world, we would be able to use the new model in vLLM as soon as it is added to Transformers. With the integration of the Transformers modeling backend, we step towards that ideal world.

Here is the official documentation on how to make your Transformers model compatible with vLLM for the integration to kick in. We followed this and made modeling_gpt2.py compatible with the integration! You can follow the changes in this Transformers pull request.

For a model already in Transformers (and compatible with vLLM), this is what we would need to:

Note: It is not a strict necessity to add model_impl parameter. vLLM switches to the Transformers implementation on its own if the model is not natively supported in vLLM.

Or for a custom model from the Hugging Face Hub:

This backend acts as a bridge, marrying transformers’ plug-and-play flexibility with vLLM’s inference prowess. You get the best of both worlds: rapid prototyping with Transformers and optimized deployment with vLLM.

Case Study: Helium

Kyutai Team’s Helium is not yet supported by vLLM. You might want to run optimized inference on the model with vLLM, and this is where the Transformers modeling backend shines.

Let’s see this in action:

Query it with the OpenAI API:

Here, vLLM efficiently processes inputs, leveraging the Transformers modeling backend to load kyutai/helium-1-preview-2b seamlessly. Compared to running this natively in Transformers, vLLM delivers lower latency and better resource utilization.

By pairing Transformers’ model ecosystem with vLLM’s inference optimizations, you unlock a workflow that’s both flexible and scalable. Whether you’re prototyping a new model, deploying a custom creation, or scaling a multimodal app, this combination accelerates your path from research to production.

Related Posts

Day 0 Support for Qwen3.8-2.4T-A95B on vLLM

Day-0 vLLM support for Qwen3.8-2.4T-A95B: a 2.4-trillion-parameter hybrid MoE model served out of the box, with FP8/BF16 checkpoints plus NVFP4 and MXFP4 quantized weights, and co-developed kernels on NVIDIA and AMD hardware.