Llama Stack 引入 vLLM 推理 Provider

Llama Stack focuses on making it easy to build production applications with a variety of models - ranging from the latest Llama 3.3 model to specialized models like Llama Guard for safety and other models. The goal is to provide pre-packaged implementations (aka “distributions”) which can be run in a variety of deployment environments. The Stack can assist you in your entire app development lifecycle - start iterating on local, mobile or desktop and seamlessly transition to on-prem or public cloud deployments. At every point in this transition, the same set of APIs and the same developer experience are available.

Each specific implementation of an API is called a "Provider" in this architecture. Users can swap providers via configuration. vLLM is a prominent example of a high-performance API backing the inference API.

Llama Stack provides two vLLM inference providers:

Remote vLLM inference provider through vLLM's OpenAI-compatible server; Inline vLLM inference provider that runs alongside with Llama Stack server

In this article, we will demonstrate the functionality through the remote vLLM inference provider.

Prerequisites

Linux operating system; Hugging Face CLI if you'd like to download the model via CLI; OCI-compliant container technologies like Podman or Docker (can be specified via the CONTAINER_BINARY environment variable when running llama stack CLI commands); Kind for Kubernetes deployment; Conda for managing Python environment

Start vLLM Server

We first download the "Llama-3.2-1B-Instruct" model using the Hugging Face CLI. Note that you'll need to request for access and then specify your Hugging Face token when logging in.

Next, let's build the vLLM CPU container image from source. Note that while we use it for demonstration purposes, there are plenty of other images available for different hardware and architectures.

We can then start the vLLM container:

We can get a list of models and test a prompt once the model server has started:

Start Llama Stack Server

Once we verify that the vLLM server has started successfully and is able to serve requests, we can then build and start the Llama Stack server.

First, we clone the Llama Stack source code and create a Conda environment that includes all the dependencies:

Next, we build the container image with llama stack build:

Once the container image has been built successfully, we can then edit the generated vllm-run.yaml to be /tmp/test-vllm-llama-stack/vllm-llama-stack-run.yaml with the following change in the models field:

Then we can start the Llama Stack Server with the image we built via llama stack run:

Alternatively, we can run the following podman run command instead:

Once we start the Llama Stack server successfully, we can then start testing a inference request:

Instead of starting the Llama Stack and vLLM servers locally. We can deploy them in a Kubernetes cluster. We'll use a local Kind cluster for demonstration purposes:

Start vLLM server as a Kubernetes Pod and Service (remember to replace with your actual token):

We can verify that the vLLM server has started successfully via the logs (this might take a couple of minutes to download the model):

Then we can modify the previously created vllm-llama-stack-run.yaml to /tmp/test-vllm-llama-stack/vllm-llama-stack-run-k8s.yaml with the following inference provider:

Once we have defined the run configuration for Llama Stack, we can build an image with that configuration and the server source code:

We can then start the Llama Stack server by deploying a Kubernetes Pod and Service:

We can check that the Llama Stack server has started:

Now let's forward the Kubernetes service to a local port and test some inference requests against it via the Llama Stack Client:

You can learn more about different providers and functionalities of Llama Stack on the official documentation.

Acknowledgement

We'd like to thank the Red Hat AI Engineering team for the implementation of the vLLM inference providers, contributions to many bug fixes, improvements, and key design discussions. We also want to thank the Llama Stack team from Meta and the vLLM team for their timely PR reviews and bug fixes.

Related Posts

Day 0 Support for Qwen3.8-2.4T-A95B on vLLM

Day-0 vLLM support for Qwen3.8-2.4T-A95B: a 2.4-trillion-parameter hybrid MoE model served out of the box, with FP8/BF16 checkpoints plus NVFP4 and MXFP4 quantized weights, and co-developed kernels on NVIDIA and AMD hardware.