Understand this story
The Lenovo team tested the inference performance of a compute-storage fusion pooled system. Xiao Limin, Chief Scientist of Lenovo Group, proposed a compute-storage fusion pooled system that combines memory pooling with near-memory computing, targeting two major pain points: insufficient KV Cache capacity and high GPU memory access overhead. Tests show that memory pool offloading reduces first-token latency by 3.5x, KV cross-instance sharing achieves up to 12~28x speedup compared with full recomputation, offloading attention computation reduces decoding latency by 2.5x, and end-to-end throughput improves by 6~20x as context length grows. The team's next step is to evolve from G2.5 to G1.5 with the help of supernode scale-up interconnect.
AI-generated from public reports
Developments
1 confirmed developments
Lenovo team tests inference performance of compute-storage fusion pooled system
Source:TMTPost · AGI
Reports and discussion timeline
Browse reports and discussion by time. Their evidence levels remain distinct.
Oct 23
TMTPost · AGI
Behind 500 trillion Tokens per day, Lenovo uses storage-compute fusion pooling to address the KV Cache memory wall
Xiao Limin, Chief Scientist of Lenovo Group, proposes a storage-compute fusion pooling system that combines memory pooling with near-memory computing, simultaneously addressing the two major pain points of insufficient KV Cache capacity and high GPU memory access overhead. Tests show that memory pool offloading reduces first-token latency by 3.5x, KV cross-instance sharing achieves up to 12~28x speedup compared with full recomputation, decoding latency drops by 2.5x after attention computation is offloaded, and end-to-end throughput improves by 6~20x as context length grows. The team's next step is to evolve from G2.5 to G1.5 with the help of supernode scale-up interconnect.