NVIDIA Explains How to Break the AI Memory Bottleneck: HBM Grows Only About 3%, Agent Throughput per Dollar Up to 30x Higher
NVIDIA released a technical explainer proposing full-stack co-design to alleviate the AI memory bottleneck: with HBM capacity increased by only about 3%, Vera Rubin NVL72 achieves up to 30x higher agent throughput per dollar compared with GB300 NVL72. Specific measures include using the NVFP4 low-precision format to compress model weights and KV cache, distributing model weights across up to 72 GPUs in the NVL72 system, buffering paused context in the LPDDR memory of the Vera CPU, and using the Dynamo inference software to migrate inactive context to CMX context storage. NVIDIA cited benchmark data from SemiAnalysis AgentX and noted that the key to compute competition is shifting from memory capacity to how many effective tokens can be produced per byte and per watt.
Why read this
AI score 60
The original article breaks down NVIDIA's specific approach to raising throughput per dollar through compression, tiering, and scheduling, even with HBM capacity nearly unchanged.
Topics
Body · AI translation
Add crowdsourcing testing promotional copy Add crowdsourcing testing promotional copy end Add App-exclusive traffic guidance
IT Home, November 6: NVIDIA yesterday (November 5) released a technical explainer, noting that as large model parameter scales, context lengths, and agent applications grow rapidly, GPU memory is facing unprecedented pressure, and HBM supply is tightening with rising prices, a trend that will continue for the foreseeable future.
To break the current AI memory bottleneck, NVIDIA is exploring the introduction of a "full-stack co-design" philosophy, designing GPU, CPU, network, storage, and software as a coordinated whole, so that every byte of memory generates higher value at every layer of the AI factory. IT Home attaches the relevant images below:
During inference, expensive HBM must carry both model weights and the KV cache (used to record the context the model needs to understand the current request). Agent workflows continuously carry historical context across multiple model calls and tool calls, with some scenarios needing to handle as many as 3 million tokens; retrieval-augmented generation (RAG) also adds enterprise materials to prompts, further driving up memory usage.
NVIDIA likens HBM to a warehouse and memory bandwidth to a loading dock: no matter how large the capacity, if the speed at which data goes in and out cannot keep up, a "memory wall" forms, causing added capacity to fail to translate effectively into serving capability. Therefore, its approach emphasizes compressing data, placing it at the appropriate tier, and sending it back to the GPU in advance of when it is needed.
In concrete operations, the NVFP4 low-precision format can compress model weights and the KV cache; in the NVL72 system, model weights can be distributed across up to 72 GPUs; the large-capacity LPDDR memory of the Vera CPU can provide a nearby buffer for paused contexts; and the Dynamo inference software migrates inactive contexts to CMX context storage and retrieves them before reuse. The RAG index can also be kept on the CPU and storage side, avoiding long-term occupation of HBM.
Citing the SemiAnalysis AgentX benchmark, NVIDIA says that at 125 tokens per second per user, Vera Rubin NVL72 delivers up to 30 times higher agent throughput per dollar than GB300 NVL72, while HBM capacity per GPU increases by only about 3%.
At 100 tokens per second per user, GB300 NVL72 delivers up to 10 times higher throughput per dollar than the H200 platform. This shows that the key to AI computing competition is shifting from "how much memory" to "how many effective tokens can be produced per byte and per watt." Statement: Sina exclusive article, unauthorized reproduction prohibited. tech_web_article_v2015_block_bottom Non-targeted 300*250 button 17/09 wenjing begin
Non-targeted 300*250 button end