NVIDIA's HBM capacity grows only 3%, while AI inference throughput per unit cost improves by up to 30x

HBM capacity increases by only 3%, NVIDIA's AI inference throughput rises 30-fold Source: Ban Dao Zong Heng Published: 2026-11-06 11:15 NVIDIA HBM promotional poster From model compression to multi-tier storage collaboration, NVIDIA explores a new path for AI inference memory efficiency.

As model sizes grow, contexts lengthen, and agentic applications increase, GPU memory pressure is continuously rising. At the same time, tightening memory supply and rising prices are expected to persist in the near term.

NVIDIA said it has maintained close collaboration with memory partners to rapidly adopt the latest memory technologies, and that it is among the first vendors to bring HBM3e and HBM4 technology into accelerated computing systems.

However, NVIDIA's approach is not simply to stack more memory on the GPU, but to adopt Full-Stack Codesign, designing the GPU, CPU, network, storage, and software together so that memory at every level of the AI factory can play a greater role.

This is because AI factories rely on a memory and storage hierarchy to operate. Among these, GPU high-bandwidth memory (HBM) is the fastest and most expensive, and also the most fiercely contested resource; model weights and KV Cache both need to occupy limited HBM capacity. Below it are CPU memory, local SSDs, and shared storage, which provide greater capacity and lower per-unit storage cost at the expense of some speed.

How much work each byte of memory can do depends on whether the entire storage hierarchy can be effectively utilized, including reducing the amount of data that needs to be stored, placing data at the appropriate storage tier, and promptly bringing it back to the GPU before the data is used again.

NVIDIA said that in this way, the company can use memory at each tier more efficiently. From Blackwell Ultra to the Vera Rubin platform, HBM capacity per GPU grows by only about 3%, but according to SemiAnalysis AgentX testing, throughput per unit cost for agentic workloads improved by up to 30-fold.

Agentic inference puts greater pressure on HBM

At the same time, KV Cache is also continuously expanding. KV Cache is used to store the context information the model needs to understand the current request, avoiding repeated processing of the same content. In agentic applications, the model needs to repeatedly call models and external tools, and previous context persists across multiple model calls and tool calls. The processing trajectory of some agentic tasks even reaches 3 million tokens, so the working memory required for a single request also continuously increases as the task progresses.

Retrieval-augmented generation (RAG) likewise increases memory demand. RAG adds relevant enterprise information to the model prompt, bringing more context data that needs to be processed and stored.

During inference, HBM mainly stores model parameters and KV Cache, which together occupy limited GPU memory, further increasing pressure on HBM.

Increasing HBM capacity per GPU is the most direct way to address memory pressure, but how much capacity can accomplish also depends on how much data bandwidth can transmit.

HBM can be compared to a warehouse, and memory bandwidth to the warehouse's loading dock. A larger warehouse can hold more inventory, but the loading dock determines the speed at which goods move in and out. Once the dock reaches its processing capacity limit, more inventory means longer turnaround times, which in turn slows response speed. If latency exceeds the required service level, the additional storage space cannot be effectively utilized, while also increasing cost and power consumption.

This is also why enterprises do not place all inventory in the fastest location closest to the production line. Goods in active circulation stay near the production line, while goods temporarily waiting are moved to larger, lower-cost space and brought back before they are needed.

More HBM can store more KV Cache and support more concurrent users. But each additional user jointly consumes the system's compute resources and memory bandwidth. When all users demand fast responses, memory bandwidth may ultimately be unable to deliver data at the speed required by compute, a limitation engineers call the "Memory Wall."

Beyond this limit, additional HBM capacity may not be effectively utilized while still increasing cost and power consumption. Data still needs to be stored, but not all data must be placed in the fastest and most expensive storage tier. Therefore, AI factories need to manage different types of data through a memory and storage hierarchy.

Full-stack codesign enhances the value of every byte of memory

NVIDIA's full-stack codesign mainly reduces the HBM footprint of model weights, KV Cache, and RAG data, allowing limited HBM to do more effective work, specifically including the following aspects.

First, shrink model weights and distribute them across the NVL72 system.

NVIDIA's NVFP4 data format can further reduce the storage space required for model weights compared with FP8. FP8 is the lowest precision supported by NVIDIA's Hopper platform.

Several recently released models already offer NVFP4 checkpoints, including FLUX.2 [klein] from Black Forest Labs, Inkling from Thinking Machines Lab, Beam 1.0 from Reflection, Muse Spark from Meta, Step 5 Preview from StepFun, and Nemotron 3.5 Super VL from NVIDIA. This shows that an increasing number of model developers are beginning to adopt 4-bit precision to achieve faster, more memory-efficient inference.

In addition, in the NVL72 system, model weights can be distributed across up to 72 GPUs, with each GPU only needing to store a portion of the model weights, thereby freeing up more HBM space for KV Cache and concurrent users.

Second, shrink the KV Cache and move inactive context to lower-cost storage tiers.

NVIDIA applies NVFP4 to the KV Cache, which, compared with FP8, can further reduce the storage space occupied by the KV Cache and allow more context to be retained in HBM.

The Vera CPU plays a role in two ways. On the one hand, faster tool execution can shorten the time the KV Cache sits idle during tool calls; on the other hand, the large-capacity LPDDR memory equipped on the Vera CPU, connected to the Rubin GPU via NVIDIA's NVLink-C2C interconnect technology, can provide a nearby storage location for paused context and quickly retrieve it when needed.

When context is temporarily no longer in use, NVIDIA's Dynamo inference software can move it to the NVIDIA CMX context memory storage system and retrieve it before it is used again.

CMX is a tier designed specifically for large-capacity storage, with a per-unit data storage cost far lower than HBM. Unlike local SSDs, CMX can be shared across multiple servers, and unlike enterprise storage, it eliminates the long-term data protection overhead that short-term context does not need. Partners for NVIDIA CMX context memory storage include Cloudian, DDN, Dell Technologies, Everpure (formerly Pure Storage), HPE, IBM, NetApp, VAST, and WEKA.

At the same time, Dynamo's KV Cache reuse feature connects different storage tiers, managing the KV Cache in GPU memory, CPU memory, and storage as a unified system-level resource, so that context that has already been computed can be reused throughout the system.

Third, compress RAG vectors and keep the index outside HBM.

RAG converts enterprise documents into vector embeddings, that is, representing information numerically so that the system can search by semantics rather than keywords, and organizes these vectors into a searchable index.

The NVIDIA Nemotron Retriever model can generate smaller vector embeddings with higher retrieval accuracy, while the NVIDIA cuVS library supports building and running these retrieval indexes on CPUs and storage systems. In this way, the entire knowledge base does not need to occupy HBM, and only the relevant text fragments that the model actually needs enter the context.

Ultimately, the HBM occupied by model weights is reduced, and the HBM occupied by idle context is also reduced, freeing up more memory resources to serve concurrent users.

Each generation of the platform increases output per unit cost

Across NVIDIA's different generations of computing platforms, the growth rate of token throughput achievable per dollar has been far faster than the growth rate of HBM capacity.

According to SemiAnalysis AgentX's DeepSeek V4 Pro workload test, under the condition of a generation speed of 100 tokens/s per user, the NVIDIA GB300 NVL72 platform delivers up to 10x higher throughput per unit cost compared with the H200 platform, while the HBM capacity per GPU roughly doubles. Under the condition of a generation speed of 125 tokens/s per user, the Vera Rubin NVL72 platform delivers up to 30x higher throughput per unit cost compared with the GB300 NVL72 platform, while the HBM capacity per GPU increases by only about 3%.

For AI factory operators, the metric that truly matters is: how many effective tokens the entire factory can output within a given power budget while meeting user response time requirements. The more tokens that can be processed per watt per second, the more work the same power resources can accomplish and the more revenue they can generate, while lowering the cost per token.

NVIDIA states that its full-stack codesign lowers the cost per token by increasing the effective work that can be done per byte of memory.

This article is reproduced from media reports or online platforms and represents the author's personal stance or views. Our reproduction is solely for sharing and does not mean that we endorse or agree with it. If the source attribution is incorrect or your lawful rights and interests are infringed, please contact customer service promptly, and we, as a neutral platform service provider, will promptly correct, delete, or handle it in accordance with the law. Comments: No user comments yet. Latest News More Related Recommendations

Beyond substrates, glass may expand the boundaries of AI packaging architecture 2026-11-06 11:21

Yicheng Technology completes a C round of financing in the billions of yuan 2026-11-06 11:11

The China-Netherlands export control dialogue mechanism meeting is held in Shenzhen 2026-11-06 11:11