vLLM 新增对 NVIDIA Nemotron 模型家族的部署支持

Agentic AI systems, capable of reasoning, planning, and taking autonomous actions, are powering the next leap in developer applications. To build these systems, developers need tools that are open, efficient, and ready to scale. And, as demand for agents grows, open, performant models are the key as they provide transparency, adaptability, and cost-control.

NVIDIA Nemotron is a family of open models, datasets, and technologies that unlock developers to build highly efficient and accurate models for specialized agentic AI.

vLLM Now Supports NVIDIA Nemotron

vLLM provides a seamless path to deploying the open family of NVIDIA Nemotron, letting developers spin up agentic, high-accuracy inference on both data center and edge hardware—optimized for throughput and accuracy. Nemotron models are ready to serve out-of-the-box with vLLM, using open weights and open data for reproducible, production-grade agents.

NVIDIA Nemotron Nano 2

The latest addition to this family is the NVIDIA Nemotron Nano 2, a highly efficient small language reasoning model with a hybrid Transformer–Mamba architecture and a configurable thinking budget. This allows developers to dial accuracy, throughput, and cost to match their real‑world application needs.

Open: Available on Hugging Face, this model provides leading accuracy for reasoning, coding, and various agentic tasks including instruction following, tool calling, and long context chat. Over 9T tokens of pre- and post-training data, generated and curated by NVIDIA and available with a very permissible license, is also released on Hugging Face; Efficient: Nemotron Nano 2, thanks to the hybrid architecture, produces critical thinking tokens up to 6 times faster compared to the next best open dense model of a similar size using vLLM. Higher throughput allows the model to think faster, explore larger search space, do better self reflection, and deliver higher accuracy

Optimized Thinking: The model has a new feature called thinking budget which avoids agent overthinking and optimizes for predictable inference cost. The chart below shows that if left alone, models can overthink, increasing inference cost, and in certain cases also reduce accuracy. Thinking budget addresses this challenge by enabling developers to tune the model to achieve the most optimal accuracy-token generation sweetspot for their applications

Let’s deploy the Nemotron Nano 2 model for agentic inference using vLLM:

Now we can create a ThinkingBudgetClient to consume our newly created endpoint. This will help enforce and parse the thinking budget feature described above. With vLLM, this process is very straightforward - let’s dive in!

Now that we’ve set up our ThinkingBudgetClient we can fire-off a request and look at the response!

We see the response as:

vLLM as a tool helps improve Nemotron Nano 2 deployment by making it faster, more memory-efficient, and easier to scale for real-time agentic use-cases. Additionally, vLLM’s focus on efficient KV-cache management, and long context use-cases works well with the hybrid transformer-Mamba architecture of Nemotron Nano 2.

Run Anywhere

Nemotron models are configured to run across all GPU-accelerated systems so you can seamlessly transition from development to production.

Try the model on the NVIDIA-hosted endpoint at build.nvidia.com or download from Hugging Face.

Share your ideas and vote on what matters to help shape the future of Nemotron.

Stay up to date on NVIDIA Nemotron by subscribing to NVIDIA news and following NVIDIA AI on LinkedIn, X, YouTube, and the Nemotron channel on Discord.

Related Posts

Day 0 Support for Qwen3.8-2.4T-A95B on vLLM

Day-0 vLLM support for Qwen3.8-2.4T-A95B: a 2.4-trillion-parameter hybrid MoE model served out of the box, with FP8/BF16 checkpoints plus NVFP4 and MXFP4 quantized weights, and co-developed kernels on NVIDIA and AMD hardware.