NVIDIA Explains the AI Memory Bottleneck: HBM Grows Only About 3%, Agent Throughput per Dollar Improves by Up to 30x
NVIDIA released a technical explainer proposing full-stack co-design to alleviate the AI memory bottleneck, co-designing GPU, CPU, network, storage, and software as a whole. The plan includes compressing model weights and KV cache using the low-precision NVFP4 format, distributing model weights across up to 72 GPUs in the NVL72 system, buffering paused context in the LPDDR memory of the Vera CPU, and migrating inactive context to CMX context storage via the Dynamo inference software. Citing the SemiAnalysis AgentX benchmark, NVIDIA claims that at 125 tokens per second per user, the Vera Rubin NVL72 achieves up to 30x higher agent throughput per dollar compared with the GB300 NVL72, while HBM capacity per GPU increases by only about 3%.
Why read this
AI score 62
NVIDIA presents a concrete layered plan for full-stack co-design, so readers can understand how to place context at the appropriate tier under the memory bottleneck.
Topics
Body · AI translation
Nvidia Breaks Through the AI Memory Bottleneck: HBM Increases by Only About 3%, Agent Throughput per Dollar Improves by Up to 30x
2026-11-06 07:30 IT Home - Xiaobeng
English text is being prepared.
IT Home, November 6 news: Nvidia yesterday (November 5) published a technical explainer, noting that as large model parameter scales, context lengths, and agent applications grow rapidly, GPU memory is facing unprecedented pressure, and HBM supply is tightening and prices are rising, which will continue for the foreseeable future.
To break through the current AI memory bottleneck, Nvidia explored and introduced the concept of "full-stack co-design," designing GPU, CPU, network, storage, and software as a coordinated whole, so that every byte of memory generates higher value at every layer of the AI factory. IT Home attaches the related images below:
During inference, expensive HBM must carry both model weights and the KV cache (used to record the context the model needs to understand the current request). Agent workflows continuously carry historical context across multiple model calls and tool calls, and some scenarios need to handle as many as 3 million tokens; retrieval-augmented generation (RAG) also adds enterprise materials to the prompt, further driving up memory usage.
Nvidia compares HBM to a warehouse and memory bandwidth to a loading dock: no matter how large the capacity, if the speed at which data goes in and out cannot keep up, a "memory wall" forms, causing added capacity to fail to effectively translate into serving capability. Therefore, its approach emphasizes compressing data, placing it at the appropriate layer, and sending it back to the GPU in advance before it is needed.
In concrete operations, the NVFP4 low-precision format can compress model weights and the KV cache; in the NVL72 system, model weights can be distributed across up to 72 GPUs; the large-capacity LPDDR memory of the Vera CPU can provide a nearby buffer for paused contexts; and the Dynamo inference software migrates inactive contexts to CMX context storage and retrieves them before reuse. The RAG index can also be kept on the CPU and storage side, avoiding long-term occupation of HBM.
Citing the SemiAnalysis AgentX benchmark, Nvidia says that at 125 tokens per second per user, Vera Rubin NVL72 improves agent throughput per dollar by up to 30x compared with GB300 NVL72, while HBM capacity per GPU increases by only about 3%.
At 100 tokens per second per user, GB300 NVL72 improves throughput per dollar by up to 10x compared with the H200 platform. This shows that the key to competition in AI computing power is shifting from "how much memory" to "how many effective tokens can be produced per byte and per watt."