Data Center / Cloud NVIDIA Vera Rubin NVL72 Leads Agentic Inference Performance on MLPerf Endpoints v1.0 Two rows of black data center racks with gold-colored server components against a black background. Nov 16, 2026 By Zhihan Jiang and Lisa Luh Discuss (0)

AI-Generated Summary

  • Provisional MLPerf Endpoints v1.0 results show NVIDIA Vera Rubin NVL72 delivers over 10x the throughput per kilowatt of NVIDIA GB300 NVL72 at a shared minimum end-to-end interactivity of 50 tokens per second per user.
  • Across Kimi K3, DeepSeek-V4.1-Flash, and Qwen3.6-35B-A3B models, Vera Rubin NVL72 achieves 5x to 10x higher throughput per kilowatt and 3x to 5x higher end-to-end interactivity at matched concurrency compared with GB300 NVL72.
  • NVIDIA Rubin GPUs combine HBM4 memory, Tensor Cores, and sixth-generation NVLink to support agentic inference demands.
  • Context-aware serving techniques including prefix caching, state reuse, and NVIDIA Dynamo routing reduce redundant computation and balance serving load across agentic workloads.
  • Model-specific GPU execution optimizations such as DeepSeek's MegaMoE kernel and Qwen's multi-token prediction with Tensor Core verification reduce data movement and sequential decoding steps.

Next Steps

Powered by NVIDIA Nemotron. AI-generated content may summarize information incompletely. Verify important information. Learn more

Agentic workloads are demanding more from inference systems. AI agents carry growing conversation histories across turns and generate a mix of reasoning, tool calls, and responses. Inference systems must support users at scale while remaining responsive.

MLPerf Endpoints v1.0 evaluates this balance between serving capacity and responsiveness using coding and enterprise workflows through API endpoints. During performance runs, generated tool calls and workflow intent are checked against reference values. Separate SWE-bench Verified evaluations assess end-to-end task-solving capability. These checks evaluate whether faster serving preserves model quality.

In MLPerf Endpoints v1.0, provisional NVIDIA Vera Rubin NVL72 results show over 10x the throughput per kilowatt of provisioned power of NVIDIA GB300 NVL72 at a shared minimum end-to-end interactivity of 50 tokens per second per user.1

This post presents the benchmark results and outlines the hardware and software optimizations used in the submissions.

1MLPerf® Endpoints v1.0, Kimi K3, Standardized division, Available category, published November 9, 2026. NVIDIA Vera Rubin NVL72, 32 accelerators, result 1.0.1.kimi-k3.agentic_combined.1p, run 9: concurrency = 1,024 queries, TTFT P90 = 4,673.9 ms, system TPS = 41,414.2 tokens/sec, E2E average interactivity = 50.4 tokens/sec per user, operating power = 100.89 kW. NVIDIA GB300 NVL72, 48 accelerators, result 1.0.0.kimi-k3.agentic_combined.1, run 6: concurrency = 48 queries, TTFT P90 = 2,305.1 ms, system TPS = 2,210.4 tokens/sec, E2E average interactivity = 56.2 tokens/sec per user, operating power = 60 kW. Comparison based on throughput per kilowatt at a shared minimum E2E average interactivity of 50 tokens/sec per user. Retrieved from endpoints.mlcommons.org on November 9, 2026. GB300 results verified by MLCommons Association. For the provisional Vera Rubin results: pending MLCommons Association verification, results are subject to change pending peer review. The MLPerf name and logo are registered and unregistered trademarks of MLCommons Association in the United States and other countries. All rights reserved. Unauthorized use is strictly prohibited. See www.mlcommons.org for more information.

The serving demands of multi-turn agentic AI

The MLPerf Endpoints v1.0 dataset captures these workload characteristics: coding trajectories accumulate history over many turns, while enterprise workflows reuse shared instructions, tool definitions, and business rules. Figure 1 shows how context length grows across model responses during extended agentic sessions.

Line chart of context growth across 230 coding and workflow trajectories with more than 10 model responses. Median context length rises from about 30,000 tokens at the first response to about 200,000 tokens by response 300. Shading shows the 25th–75th percentile range; a dashed line shows the mean. A secondary axis shows that progressively fewer trajectories reach later responses.
Figure 1. Context length grows across model responses in the MLPerf agentic dataset

Vera Rubin delivers over 10x the agentic throughput per kilowatt

The submissions span Kimi K3, Qwen3.6-35B-A3B, and DeepSeek-V4.1-Flash. Table 1 compares provisional Vera Rubin NVL72 results with verified NVIDIA GB300 NVL72 results for throughput per provisioned kilowatt at a shared minimum end-to-end (E2E) interactivity threshold and for E2E interactivity at matched query concurrency.

ModelThroughput per provisioned kW ratioE2E interactivity ratio
Kimi K310x3x
DeepSeek-V4.1-Flash9x5x
Qwen3.6-35B-A3B5x3x
Table 1. Throughput per kilowatt and E2E interactivity ratios for Vera Rubin NVL72 compared with GB300 NVL72²

² MLPerf® Endpoints v1.0, Kimi K3, DeepSeek-V4.1-Flash, and Qwen3.6-35B-A3B, Standardized division, Available category. GB300 results published November 4, 2026; Vera Rubin results published November 9, 2026, for Kimi K3 and Qwen, and November 10, 2026, for DeepSeek. Retrieved from endpoints.mlcommons.org on November 10, 2026. Comparisons select each platform’s best TPS/kW meeting the stated minimum E2E average interactivity, or best E2E average interactivity at the stated matched concurrency. Records list: result ID; run number; concurrency (queries); TTFT P90 (ms); System TPS (tokens/sec); E2E average interactivity (tokens/sec per user); operating-point power (kW).

Kimi K3: Throughput/kW: see Footnote 1. Interactivity at matched concurrency 1,024 queries: Vera Rubin (1.0.1.kimi-k3.agentic_combined.1p; 9; 1,024; 4,673.9; 41,414.2; 50.4; 100.89); GB300 (1.0.0.kimi-k3.agentic_combined.1; 16; 1,024; 15,223.0; 13,905.8; 16.8; 60).
DeepSeek-V4.1-Flash: Throughput/kW at minimum E2E average interactivity 280 tokens/sec per user: Vera Rubin (1.0.1.deepseek-v4_1-flash.mlperf_agentic_inference_dataset.1p; 6; 1,408; 427; 272,711.7; 294.49; 75.67), 24 GPUs; GB300 (1.0.0.deepseek-v4_1-flash.mlperf_agentic_inference_dataset.1; 1; 16; 112.8; 3,054.0; 288.65; 7.50), 4 GPUs. Interactivity at matched concurrency 1,024 queries: Vera Rubin (1.0.1.deepseek-v4_1-flash.mlperf_agentic_inference_dataset.1p; 5; 1,024; 505; 204,207.7; 307.36; 63.06), 20 GPUs; GB300 (1.0.0.deepseek-v4_1-flash.mlperf_agentic_inference_dataset.1; 8; 1,024; 4,073.5; 49,701.8; 56.65; 7.50), 4 GPUs.
Qwen3.6-35B-A3B: Throughput/kW at minimum E2E average interactivity 410 tokens/sec per user: Vera Rubin (1.0.1.qwen3_6-35b-a3b.agentic_coding.1p; 8; 320; 75.2; 47,785.5; 411.9; 12.61); GB300 (1.0.0.qwen3_6-35b-a3b.agentic_coding.1; 2; 36; 65.3; 5,960.4; 531.6; 7.50). Interactivity at matched concurrency 256 queries: Vera Rubin (1.0.1.qwen3_6-35b-a3b.agentic_coding.1p; 7; 256; 65.9; 41,142.6; 501.0; 12.61); GB300 (1.0.0.qwen3_6-35b-a3b.agentic_coding.1; 7; 256; 111.7; 23,467.8; 154.5; 7.50).

Ratios are rounded as shown. GB300 results verified by MLCommons Association. Vera Rubin’s results are provisional. Pending MLCommons Association verification, results are subject to change pending peer review. The MLPerf name and logo are registered and unregistered trademarks of MLCommons Association in the United States and other countries. All rights reserved. Unauthorized use is strictly prohibited. See www.mlcommons.org for more information.

Figure 2 shows the tradeoff between throughput per provisioned kilowatt and E2E interactivity across at least seven concurrency levels per curve. The curves show how much throughput each system delivers at a given responsiveness requirement. The step-function plot does not interpolate between measured points to keep comparisons grounded in demonstrated performance rather than estimates.

Chart comparing Kimi K3 throughput per provisioned kilowatt against end-to-end interactivity in MLPerf Endpoints v1.0. NVIDIA Vera Rubin NVL72 delivers over 10x the throughput per kilowatt of GB300 NVL72 at a shared minimum interactivity of 50 tokens per second per user.
Figure 2. NVIDIA Vera Rubin NVL72 and GB300 NVL72 throughput per provisioned kilowatt versus E2E interactivity on Kimi K3 in MLPerf Endpoints v1.0

Higher throughput per kilowatt enables organizations to run more agents simultaneously within the same power budget while meeting the same responsiveness requirement. At low concurrency, faster responses reduce inference wait times for latency-sensitive agentic tasks.

Hardware and software optimizations for agentic inference

NVIDIA Rubin GPUs combine HBM4 memory, NVIDIA Tensor Cores, and sixth-generation NVIDIA NVLink to support the demands of agentic inference. HBM4 enables rapid access to model weights and conversation state, Tensor Cores accelerate matrix operations, and NVLink supports communication across GPUs. Context-aware serving and model-specific kernels build on these capabilities to reuse prior work, independently optimize prefill and decode, and accelerate token generation—translating hardware advances into greater inference capacity and faster interactions.

Context-aware serving and model-specific optimizations improve inference performance

Agentic workloads repeatedly reuse conversation history while adding new instructions and tool results. Efficient serving must preserve reusable state, distribute requests across workers, and accommodate the different execution requirements of prompt processing and token generation. Prefix caching, state reuse, and NVIDIA Dynamo routing help reduce redundant computation and balance serving load, while disaggregated serving supports independent optimization of prefill and decode.

Figure 3 summarizes how context-aware serving and model-specific GPU execution support agentic inference.

Diagram showing user requests flowing through an API endpoint to NVIDIA Dynamo context-aware routing, which supports conversation-state reuse and balances worker load. Separate prefill and decode workers process prompts and generate tokens, with context state transferred between them. Model-specific kernels, fusion, and optimized memory access accelerate both stages, while multi-token proposals and verification support faster generation.
Figure 3. Context-aware serving and optimized GPU execution for agentic inference

Kimi K3, DeepSeek-V4.1-Flash, and Qwen3.6-35B-A3B also have distinct architectures that benefit from targeted GPU execution improvements. Model-specific kernels, fusion, optimized memory access, and speculative verification reduce data movement and execution overhead. Table 2 summarizes how these serving and execution techniques work together across the three models.

OptimizationHow it improves performance
Context reuse and routingReuse previously processed conversation state and distribute requests to reduce recomputation and balance worker load.
Example: Qwen’s fine-grained prefix matching reuses previously processed history across turns.
Prefill and decode optimizationConfigure prompt processing and token generation independently through disaggregated serving, reducing contention and synchronization waits.
Example: Kimi combines disaggregated serving with prefill changes that reduce GPU synchronization waits.
Model-specific GPU executionTailor kernels, fusion, and memory access to each architecture, reducing kernel launches and data movement.
Example: DeepSeek’s MegaMoE kernel uses Rubin GPU locality and matrix-multiply instructions.
Speculative generation and verificationPropose multiple tokens and verify them together, reducing sequential decoding steps.
Example: Qwen’s multi-token prediction pairs token proposals with Tensor Core verification.
Table 2. Software optimizations improve context reuse, serving efficiency, and GPU execution

More productive infrastructure for agentic AI

Agentic inference performance depends on co-design across hardware, system architecture, and serving software. The provisional MLPerf Endpoints v1.0 results show how the Vera Rubin platform brings these capabilities together to address the demands of agentic workloads and create more capacity to serve users and scale applications.

Learn more about NVIDIA training and inference performance in the companion blog post.

Acknowledgements

Richard Cai, Viraat Chandra, Xiaoming Chen, Zhenhuan Chen, Alice Cheng, Julien Darve, Ashutosh Dhar, Kefeng Duan, Alec Flowers, Ann Guan, Jie Hao, Rashid Kaleem, Tzu-Ling Kan, Tuhin Khare, Zihao Kong, Jonas Li, Shicheng Li, Yijing Li, Allison Lim, Yanchao Lu, Bofeng Luo, Mingyuan Ma, Matthias Maiterth, Yibai Meng, Stanley Cheong Kwan Phoong, Hanjie Qiu, Zhongdao Ren, Tong Tong, Harshil Vagadia, Lisa Wang, Shang Wang, Xinyu Wang, Fernando Xiong, Yingying Xue, Yuanjing Xue, Lizhi Zhou, and Yaoyang Zhou

Discuss (0)

Tags

Agentic AI / Generative AI | Data Center / Cloud | Hardware / Semiconductor | Blackwell | Dynamo | TensorRT-LLM | Intermediate Technical | Benchmark | Blackwell Ultra | GB300 | Groq 3 LPX | InfiniBand | MLPerf | Vera Rubin NVL72

About the Authors

Avatar photo About Zhihan Jiang
Zhihan Jiang is an engineer manager in the TensorRT team at NVIDIA and focuses on delivering world-class generative AI inference results in MLPerf Inference. Before working on MLPerf, he worked on TensorRT autonomous safety libraries and infrastructure, and NVIDIA CPU architecture modeling. Zhihan holds a master’s degree in Electrical Engineering from Stanford University, and a bachelor’s degree in Computer Engineering from Georgia Tech. View all posts by Zhihan Jiang Lisa Luh About Lisa Luh
Lisa Luh is a senior technical product marketing manager at NVIDIA, focused on AI inference. Previously, she held product marketing and business development roles at C3 AI and IBM. She holds an MBA from the Wharton School at the University of Pennsylvania and a Bachelor of Science in Business Administration from the Haas School of Business at the University of California, Berkeley. View all posts by Lisa Luh

Comments