NVIDIA Vera Rubin 与 Blackwell 在智能体 AI 每瓦性能上树立新标准

Technical Blog

sessionStorage.removeItem('facets.search.text'); sessionStorage.removeItem('facets.search.text-copy'); sessionStorage.removeItem('facets.search.filters');

Aug 24, 2026

SemiAnalysis AgentX evaluates AI infrastructure for agentic-coding inference by replaying prerecorded production-style sessions with interleaved reasoning and tool use. NVIDIA Vera Rubin NVL72 delivers up to 30x higher AI-factory throughput per megawatt than GB300 NVL72 at 160 tokens per second per user on the AgentX DeepSeek V4-Pro workload. GB300 NVL72 achieves up to 15x higher throughput per megawatt than H200 NVL8 for DeepSeek V4 Pro 1.6T and up to 80x higher for Kimi K3 2.8T on the AgentX workload. GB300 NVL72 provides up to 10x lower cost per million tokens than H200 NVL8, enabling more interactive agentic capacity within the same power budget. System-level optimizations including MoE serving runtimes such as SGLang , TensorRT-LLM , and vLLM, DeepGEMM-based kernels with MXFP4 and MXFP8 formats, NVIDIA Dynamo for disaggregated prefill and decode, and NVIDIA NVLink scale-up fabric enable responsive multi-turn serving as agent sessions grow. Next Steps Explore the architecture and capabilities of NVIDIA Rubin to understand the platform design for agentic AI. Review live benchmark results on the SemiAnalysis InferenceX dashboard to compare platform performance. Learn how extreme co-design addresses agentic system complexity in Building for the Rising Complexity of Agentic Systems with Extreme Co-Design .

Powered by NVIDIA Nemotron. AI-generated content may summarize information incompletely. Verify important information. Learn more

AI agents have expanded inference from single-turn interactions into multi-step workflows that reason, invoke tools, coordinate subagents, and carry growing context from one turn to the next. The scale of this shift is now visible in raw consumption: across 100 trillion tokens of real-world usage, OpenRouter’s State of AI report found that average prompt tokens per request grew roughly fourfold, and single agentic requests consume 15 times the tokens of ordinary chat.

Properly characterizing hardware performance for this workload presents new challenges. A useful benchmark must capture long-context prefill, KV-cache reuse, interactive decode, tool-call gaps, and distributed mixture of experts (MoE) execution under realistic concurrency. It must also show how much of the AI-factory power budget is converted into useful agentic throughput while maintaining an acceptable user experience.

This post examines SemiAnalysis AgentX , a benchmark that evaluates AI infrastructure for agentic-coding inference using replayed production-style sessions. It highlights Vera Rubin NVL72 preview results showing up to 30x higher AI-factory throughput per megawatt than GB300 NVL72, while showing that Blackwell GB300 NVL72 extends its order-of-magnitude throughput-per-megawatt advantage over prior generations to dynamic agentic workloads.

AgentX is the agentic-coding benchmark in InferenceX , SemiAnalysis’s open-source benchmark suite. It measures how efficiently accelerators serve the request patterns produced by real coding agents.

Agentic sessions are long, stateful, and variable: they chain model calls, tool use, and growing context rather than following a fixed prompt-and-response pattern (Figure 1, below). AgentX therefore measures whether a platform can serve replayed agent traffic responsively, reuse previously processed context, and maximize agentic throughput per provisioned megawatt.

Figure 1. Agentic sessions create variable inference sequences

Figure 2, below, shows the legacy InferenceX static 8K/1K sequence length results for DeepSeek-R1-0528. GB300 NVL72 leads H200 by up to 40x more tokens per megawatt. However, the scenario represents controlled, fixed-length serving rather than the traffic generated by an agent in production.

Figure 2. GB300 NVL72 delivers higher token throughput per all-in utility megawatt than H200 across a broader interactivity range in the 8K ISL / 1K OSL DeepSeek R1 0528 test

In real agent sessions, request lengths vary from turn to turn. Context accumulates throughout a task, previously processed tokens can be reused, and model calls are interrupted by tool execution or delegated work. As agentic AI has become the leading workload, a fixed sequence length scenario has become less representative of true serving performance and has been demoted to “maintenance mode” on the InferenceX suite.

AgentX addresses that gap by measuring serving performance across prerecorded Claude Code sessions with interleaved reasoning and tool use. It replays each session turn by turn using the AIPerf client . Because every system receives the same recorded traffic, observed differences reflect the serving stack rather than benchmark-specific tuning. Figure 3, below, illustrates a sample pre-recorded session.

Figure 3. Context accumulates across a session

The replay preserves each session’s context and input/output sequence lengths, as well as the reasoning time and tool-call latency captured in the original trajectory. These intervals preserve the original session’s timing, reproducing KV cache-capacity pressure that a realistic benchmark must capture.

AgentX varies concurrency to map the trade-off between throughput and interactivity. At each operating point, it reports sustained throughput per provisioned megawatt alongside each stack’s ability to reuse repeated context

The AgentX metric that matters most for AI factories is tokens per megawatt. The benchmark reports this metric against four user-experience values: E2E normalized interactivity, standard interactivity, E2E latency, and time to first token (TTFT). Table 1, below, describes how to best leverage these.

The following Vera Rubin NVL72 results were measured by NVIDIA using the SemiAnalysis AgentX workload and are pending SemiAnalysis review. As shown in Figure 4, below, at 160 tokens per second per user on the AgentX DeepSeek V4-Pro workload, Vera Rubin NVL72 delivers up to 30x higher AI-factory throughput per megawatt than GB300 NVL72, pointing to a substantial increase in agentic inference capacity while maintaining the same interactive serving target.

Figure 4. Vera Rubin delivers 30x higher throughput per megawatt at 160 interactivity

To learn more about how NVIDIA Vera Rubin NVL72 was designed to deliver maximum performance per megawatt, see Inside NVIDIA Rubin GPU Architecture: Powering the Era of Agentic AI.

On AgentX, GB300 NVL72 delivers up to 15x higher AI-factory throughput per megawatt than H200 NVL8 for DeepSeek V4 Pro 1.6T on the AgentX workload (see Figure 5, below). In other words, GB300 NVL72 sustains substantially more responsive agentic inference throughput within the same power budget.

Figure 5. GB300 NVL72 delivers up to 15x higher AI-factory throughput per megawatt than H200 NVL8

That throughput advantage translates directly into unit economics. GB300 NVL72 delivers up to 10x lower cost per million tokens than H200 NVL8 (Figure 6, below). For operators, this means a fixed power and infrastructure budget can support materially more interactive agentic capacity, or deliver the same capacity with substantially lower operating cost.

Figure 6. GB300 NVL72 delivers up to 10x lower token cost than H200 NVL8

The advantage of GB300 NVL72 becomes more pronounced as model scale increases. Figure 7, below, shows throughput per megawatt for Kimi K3 2.8T on the AgentX workload. With roughly 80x the throughput per megawatt of H200 NVL8 at comparable interactivity, GB300 NVL72 also extends the interactivity frontier to roughly 215 tokens per second per user, well beyond the operating range reached by H200 NVL8.

Figure 7. GB300 NVL72 delivers 80x higher throughput per megawatt than H200 NVL8 on Kimi K3 2.8T

These GB300 NVL72 results reflect system-level work across the serving runtime, model kernels, and scale-up fabric. Together, these layers enable large MoE models to sustain responsive throughput as agent sessions accumulate context, concurrency increases, and decode demand intensifies.

Vera Rubin NVL72 shows what is possible when a rack-scale system is tuned for the long-context, interactive, and distributed execution patterns of agentic inference. The broader Vera Rubin platform extends that approach across the full workflow. Rubin GPUs process large context and decode efficiently, Vera CPUs handle tool execution and KV-cache offload, and Groq 3 LPX unlocks ultrafast interactivity.

Across the AI factory, NVLink 6, ConnectX-9, BlueField-4, and Spectrum-X move tokens, context, and tool results between resources. Dynamo, Attention-FFN Disaggregation, NVFP4, TensorRT-LLM WideEP, and speculative decoding coordinate execution across the most suitable processors. The goal is simple: reduce recomputation and waiting, sustain interactive performance as agent sessions grow, and convert more of a fixed power budget into useful agentic output.

To explore the technologies and benchmarks behind this result, start with the resources below.

This work was made possible through the expertise and engineering contributions of Xin Li, Ankur Singh, Anthony Casagrande, Jonas Li, Po-Han Huang, Xiaoming Chen, and many other talented NVIDIA engineers.