Behind 500 trillion Tokens per day, Lenovo uses storage-compute fusion pooling to address the KV Cache memory wall

Over the past two decades, peak floating-point compute of servers has grown about 3x every two years, while DRAM bandwidth has grown only 1.6x every two years and interconnect bandwidth only 1.4x, creating what the industry calls the memory wall—a pain point that has in fact existed for a long time. In today's AI era, with the growth of computing power and its expanding demand for data, according to statistics from the China Academy of Information and Communications Technology, China's average daily Token consumption jumped from 100 billion in 2024 to 500 trillion in August 2025, an increase of about 5,000x in just over a year. Against this backdrop, the memory wall problem has been amplified.

In the inference era, compute and storage must be more tightly integrated

One of the core processes of large-model inference is autoregressive decoding: each time the model generates a Token, it must look back at all the preceding context. If all historical data were recomputed every time, compute would be exhausted by massive redundant calculation. Thus KV Cache was born: it caches intermediate computation results and skips redundant computation by "trading storage for compute." This mechanism greatly reduces computational overhead, but it also plants a landmine: the more data accumulates, the more there must be somewhere to put the cache.

This problem keeps being amplified as Token consumption rises. Lenovo Group Chief Scientist Xiao Limin told us that the typical AI application used to be a question-and-answer chatbot, and now it has become a digital employee that can perceive, plan, and execute tasks. In fact, every time an Agent executes a task, behind it are inference calls that often run to dozens of rounds and contexts of tens of thousands of Tokens.

And inference is precisely the stage with the highest storage requirements. The problem of where to put KV Cache is essentially a battle over space. If all data were placed in GPU memory, GPU memory is usually insufficient; people then thought of offloading data to CPU-attached memory, but CPU-attached memory capacity likewise cannot hold it. So data is naturally offloaded further to external storage, but whether the GPU accesses external storage over PCIe or the network, bandwidth and latency differ by orders of magnitude compared with accessing local GPU memory. "The overhead of moving data between external storage and the GPU often drags down inference performance," Xiao Limin pointed out.

Against this backdrop, two pain points have emerged in inference infrastructure. One is insufficient capacity of GPU memory available for KV and of CPU-attached memory; the other is the too-high overhead of GPU access to external storage. The industry's existing mainstream approach is to offload KV Cache to an external shared CXL memory pool, but this approach mainly solves the former problem: through CXL switches, memory resources are decoupled from CPUs, enabling independent elastic scaling and sharing across multiple hosts. But for the latter pain point—the high overhead of GPU access to the CXL memory pool—it has always been inadequate.

In fact, the deeper core problem is the mismatch in development speed between storage hardware and compute hardware. Compute performance improves far faster than storage, and the gap is widening year by year. This is the classic memory wall problem in the field of computer architecture, and it has never been truly fully solved; people keep mitigating it with various means such as multi-level caching, but mitigation is not elimination. The explosion of large-model inference has merely pushed this old wall back in front of everyone.

Hence the idea of compute-storage integration emerges: since moving data to compute is too costly, do the opposite and move compute units next to the data. Rather than letting massive KV data travel back and forth between the GPU and external storage over PCIe links, let compute units reside beside memory, process data in place, and send back only the computation results.

In this regard, the path Xiao Limin gives is to combine memory pooling technology with near-memory computing technology to build a compute-storage integrated pooling system for large-model inference: use memory pooling to solve large-capacity scaling and cross-instance sharing of KV Cache, and use near-memory computing to embed compute units in the memory pool, sinking memory-access-intensive operators such as attention to the memory-pool side, thereby reducing end-to-end inference latency and improving system throughput overall.

In Xiao Limin's view, this route is not radical: "Technology usually evolves gradually," he further noted. "At this stage there is little possibility of a cliff-edge rewrite of compute infrastructure, but these new changes brought by inference workloads do require more innovative technologies to be introduced into inference compute infrastructure, and compute-storage integrated pooling technology is a very important technology point in this evolution process."

Where is the balance point?

Looking at current compute-storage integration technology paths, there are mainly three: CIM (compute-in-memory), PIM (processing-in-memory), and PNM (processing-near-memory). The first two usually build compute units directly inside storage chips, with extremely high bandwidth and extremely low latency; in a sense this already counts as compute-storage integration, but it usually requires changing the structure of existing memory itself, making it highly customized hardware that is difficult to make compatible with the existing software ecosystem. For any team that needs to consider productization and deployment, this is a hurdle that must be overcome.

For this reason, in Xiao Limin's view, PNM, which does not change the structure of memory itself, is the more realistic path today. Xiao Limin said that placing compute units beside memory, at the memory controller position, both shortens the distance between compute and storage and avoids changing the original structure of memory, maintaining compatibility with the existing software and hardware ecosystem from the source. Behind this choice is a clear-eyed business judgment: "If we are to truly achieve productization in the future, the path compatible with the existing ecosystem will have a great advantage," Xiao Limin explained.

However, on the road to productizing near-memory computing, there are still many challenges. Among them, Xiao Limin believes there are two biggest challenges. The first is the dilemma in choosing the placement strategy for near-memory computing units. The computing unit must be both close enough to storage and not cause major changes to the existing ecosystem; where to place it is itself a difficult problem. After choosing the vicinity of the memory controller, new problems followed one after another. The memory controller originally handled memory access requests from the host, and now there is an additional memory access request from the near-memory computing unit beside it. Two streams of requests pour into the memory at the same time, and conflict is inevitable. The team's solution is to build a QoS arbitration mechanism into the memory bus to arbitrate and schedule the two different access streams, thereby avoiding conflicts, ensuring that the service quality of both sides does not drag each other down, and maintaining a certain degree of fairness. "The storage bus rate is high; how to do conflict detection, how to ensure that the QoS of both sides does not drop, and at the same time ensure fairness—this is indeed a relatively complex problem," Xiao Limin emphasized.

The second is the contradiction between relatively fixed operators and rapidly iterating models. Large models are updated almost every few months, while hardware computing units are relatively fixed. Lenovo adopts an instruction-driven reconfigurable computing orchestration approach, allowing the same hardware to adapt to the computing needs of different models through instruction-driven operator orchestration, alleviating this contradiction.

In addition, at the software implementation and workload operation levels, there are also questions such as how to use unified addressing to achieve unified management and efficient invocation of computing power? How to ensure data consistency when multiple machines share a memory pool? And when an inference request arrives, which computations should be offloaded to the memory pool side, which should remain for local computation on the GPU, and whether KV should be offloaded—all these issues require finding an optimal balance point. According to Xiao Limin, the intelligent computing infrastructure team at Lenovo Research has established a cost analysis model for this purpose, making dynamic decisions based on inputs such as link topology and context length, splitting KV by head in the orchestration strategy and aligning it with the GPU tensor parallelism strategy, trying to avoid communication between devices and improve the efficiency of parallel computation across devices.

After finding balance points for all the problems, the Lenovo team conducted tests. Xiao Limin introduced that they carried out itemized performance tests on three core modules. In terms of memory pool offloading, compared with the NVMe-tier pooling solution, under fixed input length scenarios, the first token latency (TTFT) can be reduced by 3.5 times; in multi-turn dialogue scenarios, it can be reduced by 2.4 times, and as the number of dialogue turns increases, the advantage becomes more obvious; in terms of cross-instance KV sharing, using the memory pool to achieve KV reuse, compared with full recomputation, the first token latency is accelerated by up to 12~28 times; compared with the RDMA distributed memory architecture, there is also a 2~3 times acceleration; in terms of storage-compute fusion, after offloading attention computation to the memory pool side, decoding latency can be reduced by 2.5 times, while freeing up a large amount of video memory to support longer contexts and larger-scale models. To verify the overall acceleration effect on large model inference, the Lenovo team also conducted end-to-end inference performance tests. In terms of inference latency, compared with the RDMA distributed memory pool, a 2~3 times acceleration was achieved; in terms of system throughput, the performance improvement is even more considerable, and as the context becomes longer, system throughput improves by 6~20 times.

Moving from G2.5 to G1.5 is the next task

What Xiao Limin's team has currently achieved is a G2.5-level storage-compute fusion pooling system based on memory pooling and near-memory computing technology. But Xiao Limin told us that evolving from G2.5 to G1.5 will be one of the key R&D directions next.

Specifically, the idea is to use the scale-up high-speed interconnect technology in supernodes to move the shared storage-compute fusion pool further toward the GPU, advancing from G2.5 to the G1.5 level and further shortening the access path. "Once supernodes become mainstream in AI systems, using the scale-up high-speed interconnect inside them will not only enjoy the benefits already obtained by G2.5, but also achieve an order-of-magnitude improvement in interconnect bandwidth and latency between computing and storage compared with existing interconnect networks," Xiao Limin said. In hardware, as technologies such as 3D stacked packaging and TSV vertical interconnection gradually mature, they are also expected to further improve the bandwidth density between storage and computing; in software, the operator library will continue to be expanded to adapt to more mainstream models. Xiao Limin also emphasized that as new chip technologies become increasingly mature, the team will incorporate more new chip technologies under the concept of "storage-compute fusion," raising inference performance to another level beyond the current level.

In Xiao Limin's view, the differentiated feature of Lenovo's storage-compute fusion pooling technology mainly lies in combining memory pooling and near-memory computing to simultaneously solve the two major problems of insufficient capacity and high memory access overhead. "At present, most related work at home and abroad focuses on how to build a large-capacity memory pool to solve the problem of insufficient KV Cache capacity; or, through storage-compute fusion technology, to alleviate the problem of high memory access overhead," Xiao Limin further pointed out. "At present, we have not yet seen a product that introduces near-memory computing into a large-capacity memory pool and simultaneously uses KV cache to solve the two problems of insufficient capacity and high memory access overhead."

Of course, the evolution of computing infrastructure technology will not happen overnight, and at the current stage the possibility of a "GPT moment" similar to AI technology is not very high. In Xiao Limin's view, in computing infrastructure, multiple elements such as computing, storage, communication, and scheduling are intertwined, integrating with and collaboratively optimizing each other. "When one part runs fast, another part drags it down, then it is sped up, and then another part becomes the short board—it is somewhat like pressing down one gourd only for another to float up." Xiao Limin further stated, "Our storage-compute fusion pooling technology is not intended to end such a situation, but rather, after the storage wall problem has been pushed to center stage by large model inference workloads, to accelerate large model inference performance by bringing near-memory computing closer to the large memory pool. In fact, a more ideal approach is not limited to the fusion of storage and computing, but rather to carry out systematic collaborative optimization around multiple elements such as computing, storage, communication, and scheduling, so as to achieve greater performance improvement in computing infrastructure and the large model inference applications running on it."