Nov 16, 2026
By Zhihan Jiang and Lisa Luh
Discuss (0)
AI-Generated Summary
- Provisional MLPerf Endpoints v1.0 results show NVIDIA Vera Rubin NVL72 delivers over 10x the throughput per kilowatt of NVIDIA GB300 NVL72 at a shared minimum end-to-end interactivity of 50 tokens per second per user.
- Across Kimi K3, DeepSeek-V4.1-Flash, and Qwen3.6-35B-A3B models, Vera Rubin NVL72 achieves 5x to 10x higher throughput per kilowatt and 3x to 5x higher end-to-end interactivity at matched concurrency compared with GB300 NVL72.
- NVIDIA Rubin GPUs combine HBM4 memory, Tensor Cores, and sixth-generation NVLink to support agentic inference demands.
- Context-aware serving techniques including prefix caching, state reuse, and NVIDIA Dynamo routing reduce redundant computation and balance serving load across agentic workloads.
- Model-specific GPU execution optimizations such as DeepSeek's MegaMoE kernel and Qwen's multi-token prediction with Tensor Core verification reduce data movement and sequential decoding steps.
Next Steps
- Read the companion blog post for more on NVIDIA training and inference performance.
- Visit MLPerf Endpoints results to review the published benchmark data.
Agentic workloads are demanding more from inference systems. AI agents carry growing conversation histories across turns and generate a mix of reasoning, tool calls, and responses. Inference systems must support users at scale while remaining responsive.
MLPerf Endpoints v1.0 evaluates this balance between serving capacity and responsiveness using coding and enterprise workflows through API endpoints. During performance runs, generated tool calls and workflow intent are checked against reference values. Separate SWE-bench Verified evaluations assess end-to-end task-solving capability. These checks evaluate whether faster serving preserves model quality.
In MLPerf Endpoints v1.0, provisional NVIDIA Vera Rubin NVL72 results show over 10x the throughput per kilowatt of provisioned power of NVIDIA GB300 NVL72 at a shared minimum end-to-end interactivity of 50 tokens per second per user.1
This post presents the benchmark results and outlines the hardware and software optimizations used in the submissions.
1MLPerf® Endpoints v1.0, Kimi K3, Standardized division, Available category, published November 9, 2026. NVIDIA Vera Rubin NVL72, 32 accelerators, result 1.0.1.kimi-k3.agentic_combined.1p, run 9: concurrency = 1,024 queries, TTFT P90 = 4,673.9 ms, system TPS = 41,414.2 tokens/sec, E2E average interactivity = 50.4 tokens/sec per user, operating power = 100.89 kW. NVIDIA GB300 NVL72, 48 accelerators, result 1.0.0.kimi-k3.agentic_combined.1, run 6: concurrency = 48 queries, TTFT P90 = 2,305.1 ms, system TPS = 2,210.4 tokens/sec, E2E average interactivity = 56.2 tokens/sec per user, operating power = 60 kW. Comparison based on throughput per kilowatt at a shared minimum E2E average interactivity of 50 tokens/sec per user. Retrieved from endpoints.mlcommons.org on November 9, 2026. GB300 results verified by MLCommons Association. For the provisional Vera Rubin results: pending MLCommons Association verification, results are subject to change pending peer review. The MLPerf name and logo are registered and unregistered trademarks of MLCommons Association in the United States and other countries. All rights reserved. Unauthorized use is strictly prohibited. See www.mlcommons.org for more information.
The serving demands of multi-turn agentic AI
The MLPerf Endpoints v1.0 dataset captures these workload characteristics: coding trajectories accumulate history over many turns, while enterprise workflows reuse shared instructions, tool definitions, and business rules. Figure 1 shows how context length grows across model responses during extended agentic sessions.

Vera Rubin delivers over 10x the agentic throughput per kilowatt
The submissions span Kimi K3, Qwen3.6-35B-A3B, and DeepSeek-V4.1-Flash. Table 1 compares provisional Vera Rubin NVL72 results with verified NVIDIA GB300 NVL72 results for throughput per provisioned kilowatt at a shared minimum end-to-end (E2E) interactivity threshold and for E2E interactivity at matched query concurrency.
| Model | Throughput per provisioned kW ratio | E2E interactivity ratio |
|---|---|---|
| Kimi K3 | 10x | 3x |
| DeepSeek-V4.1-Flash | 9x | 5x |
| Qwen3.6-35B-A3B | 5x | 3x |
² MLPerf® Endpoints v1.0, Kimi K3, DeepSeek-V4.1-Flash, and Qwen3.6-35B-A3B, Standardized division, Available category. GB300 results published November 4, 2026; Vera Rubin results published November 9, 2026, for Kimi K3 and Qwen, and November 10, 2026, for DeepSeek. Retrieved from endpoints.mlcommons.org on November 10, 2026. Comparisons select each platform’s best TPS/kW meeting the stated minimum E2E average interactivity, or best E2E average interactivity at the stated matched concurrency. Records list: result ID; run number; concurrency (queries); TTFT P90 (ms); System TPS (tokens/sec); E2E average interactivity (tokens/sec per user); operating-point power (kW).
Kimi K3: Throughput/kW: see Footnote 1. Interactivity at matched concurrency 1,024 queries: Vera Rubin (1.0.1.kimi-k3.agentic_combined.1p; 9; 1,024; 4,673.9; 41,414.2; 50.4; 100.89); GB300 (1.0.0.kimi-k3.agentic_combined.1; 16; 1,024; 15,223.0; 13,905.8; 16.8; 60).
DeepSeek-V4.1-Flash: Throughput/kW at minimum E2E average interactivity 280 tokens/sec per user: Vera Rubin (1.0.1.deepseek-v4_1-flash.mlperf_agentic_inference_dataset.1p; 6; 1,408; 427; 272,711.7; 294.49; 75.67), 24 GPUs; GB300 (1.0.0.deepseek-v4_1-flash.mlperf_agentic_inference_dataset.1; 1; 16; 112.8; 3,054.0; 288.65; 7.50), 4 GPUs. Interactivity at matched concurrency 1,024 queries: Vera Rubin (1.0.1.deepseek-v4_1-flash.mlperf_agentic_inference_dataset.1p; 5; 1,024; 505; 204,207.7; 307.36; 63.06), 20 GPUs; GB300 (1.0.0.deepseek-v4_1-flash.mlperf_agentic_inference_dataset.1; 8; 1,024; 4,073.5; 49,701.8; 56.65; 7.50), 4 GPUs.
Qwen3.6-35B-A3B: Throughput/kW at minimum E2E average interactivity 410 tokens/sec per user: Vera Rubin (1.0.1.qwen3_6-35b-a3b.agentic_coding.1p; 8; 320; 75.2; 47,785.5; 411.9; 12.61); GB300 (1.0.0.qwen3_6-35b-a3b.agentic_coding.1; 2; 36; 65.3; 5,960.4; 531.6; 7.50). Interactivity at matched concurrency 256 queries: Vera Rubin (1.0.1.qwen3_6-35b-a3b.agentic_coding.1p; 7; 256; 65.9; 41,142.6; 501.0; 12.61); GB300 (1.0.0.qwen3_6-35b-a3b.agentic_coding.1; 7; 256; 111.7; 23,467.8; 154.5; 7.50).
Ratios are rounded as shown. GB300 results verified by MLCommons Association. Vera Rubin’s results are provisional. Pending MLCommons Association verification, results are subject to change pending peer review. The MLPerf name and logo are registered and unregistered trademarks of MLCommons Association in the United States and other countries. All rights reserved. Unauthorized use is strictly prohibited. See www.mlcommons.org for more information.
Figure 2 shows the tradeoff between throughput per provisioned kilowatt and E2E interactivity across at least seven concurrency levels per curve. The curves show how much throughput each system delivers at a given responsiveness requirement. The step-function plot does not interpolate between measured points to keep comparisons grounded in demonstrated performance rather than estimates.

Higher throughput per kilowatt enables organizations to run more agents simultaneously within the same power budget while meeting the same responsiveness requirement. At low concurrency, faster responses reduce inference wait times for latency-sensitive agentic tasks.
Hardware and software optimizations for agentic inference
NVIDIA Rubin GPUs combine HBM4 memory, NVIDIA Tensor Cores, and sixth-generation NVIDIA NVLink to support the demands of agentic inference. HBM4 enables rapid access to model weights and conversation state, Tensor Cores accelerate matrix operations, and NVLink supports communication across GPUs. Context-aware serving and model-specific kernels build on these capabilities to reuse prior work, independently optimize prefill and decode, and accelerate token generation—translating hardware advances into greater inference capacity and faster interactions.
Context-aware serving and model-specific optimizations improve inference performance
Agentic workloads repeatedly reuse conversation history while adding new instructions and tool results. Efficient serving must preserve reusable state, distribute requests across workers, and accommodate the different execution requirements of prompt processing and token generation. Prefix caching, state reuse, and NVIDIA Dynamo routing help reduce redundant computation and balance serving load, while disaggregated serving supports independent optimization of prefill and decode.
Figure 3 summarizes how context-aware serving and model-specific GPU execution support agentic inference.

Kimi K3, DeepSeek-V4.1-Flash, and Qwen3.6-35B-A3B also have distinct architectures that benefit from targeted GPU execution improvements. Model-specific kernels, fusion, optimized memory access, and speculative verification reduce data movement and execution overhead. Table 2 summarizes how these serving and execution techniques work together across the three models.
| Optimization | How it improves performance |
|---|---|
| Context reuse and routing | Reuse previously processed conversation state and distribute requests to reduce recomputation and balance worker load. Example: Qwen’s fine-grained prefix matching reuses previously processed history across turns. |
| Prefill and decode optimization | Configure prompt processing and token generation independently through disaggregated serving, reducing contention and synchronization waits. Example: Kimi combines disaggregated serving with prefill changes that reduce GPU synchronization waits. |
| Model-specific GPU execution | Tailor kernels, fusion, and memory access to each architecture, reducing kernel launches and data movement. Example: DeepSeek’s MegaMoE kernel uses Rubin GPU locality and matrix-multiply instructions. |
| Speculative generation and verification | Propose multiple tokens and verify them together, reducing sequential decoding steps. Example: Qwen’s multi-token prediction pairs token proposals with Tensor Core verification. |
More productive infrastructure for agentic AI
Agentic inference performance depends on co-design across hardware, system architecture, and serving software. The provisional MLPerf Endpoints v1.0 results show how the Vera Rubin platform brings these capabilities together to address the demands of agentic workloads and create more capacity to serve users and scale applications.
Learn more about NVIDIA training and inference performance in the companion blog post.
Acknowledgements
Richard Cai, Viraat Chandra, Xiaoming Chen, Zhenhuan Chen, Alice Cheng, Julien Darve, Ashutosh Dhar, Kefeng Duan, Alec Flowers, Ann Guan, Jie Hao, Rashid Kaleem, Tzu-Ling Kan, Tuhin Khare, Zihao Kong, Jonas Li, Shicheng Li, Yijing Li, Allison Lim, Yanchao Lu, Bofeng Luo, Mingyuan Ma, Matthias Maiterth, Yibai Meng, Stanley Cheong Kwan Phoong, Hanjie Qiu, Zhongdao Ren, Tong Tong, Harshil Vagadia, Lisa Wang, Shang Wang, Xinyu Wang, Fernando Xiong, Yingying Xue, Yuanjing Xue, Lizhi Zhou, and Yaoyang Zhou
Discuss (0)Tags
Agentic AI / Generative AI | Data Center / Cloud | Hardware / Semiconductor | Blackwell | Dynamo | TensorRT-LLM | Intermediate Technical | Benchmark | Blackwell Ultra | GB300 | Groq 3 LPX | InfiniBand | MLPerf | Vera Rubin NVL72About the Authors
About Zhihan Jiang
Zhihan Jiang is an engineer manager in the TensorRT team at NVIDIA and focuses on delivering world-class generative AI inference results in MLPerf Inference. Before working on MLPerf, he worked on TensorRT autonomous safety libraries and infrastructure, and NVIDIA CPU architecture modeling. Zhihan holds a master’s degree in Electrical Engineering from Stanford University, and a bachelor’s degree in Computer Engineering from Georgia Tech. View all posts by Zhihan Jiang
About Lisa Luh
Lisa Luh is a senior technical product marketing manager at NVIDIA, focused on AI inference. Previously, she held product marketing and business development roles at C3 AI and IBM. She holds an MBA from the Wharton School at the University of Pennsylvania and a Bachelor of Science in Business Administration from the Haas School of Business at the University of California, Berkeley. View all posts by Lisa Luh