After NVIDIA's de facto acquisition of Groq, Groq's next-generation chip is reportedly set to abandon SRAM in favor of 3D stacked DRAM
Foreign media reports say that Groq, which has been de facto acquired by NVIDIA, is evaluating the introduction of 3D stacked DRAM packaging technology in its next-generation chips. The approach is called 3DS and uses TSVs to drill thousands of microvias inside DRAM chips for vertical interconnection. Unlike HBM, which pursues data exchange rates, 3DS focuses on expanding storage capacity while maintaining the existing standard DRAM interface protocols. Groq's LPU architecture relies on SRAM with very small capacity and faces a capacity bottleneck when dealing with models with hundreds of billions of parameters. 3D stacking is intended to solve the problem of not being able to store enough while maintaining speed advantages.
Why read this
AI score 53
The original report discloses that Groq's next-generation chip plans to shift from SRAM to 3D stacked DRAM, allowing readers to understand its trade-offs in addressing the capacity bottleneck for large-model inference.
Topics
Body · AI translation
A recent report by foreign media points to signals of an architectural upheaval in the AI semiconductor field: Groq, a chip company known for its high-speed SRAM designs that was recently effectively acquired by Nvidia, is evaluating the introduction of 3D stacked DRAM packaging technology in its next-generation chips, in an attempt to break free from the capacity bottleneck of traditional SRAM architectures.
Currently, most AI chips use 2D packaging. Groq plans to adopt 3D packaging, vertically stacking multiple semiconductor dies. Its solution is called 3DS, and its core lies in using TSV (through-silicon via) technology to drill thousands of micro-holes through DRAM chips for vertical interconnection—likened to building skyscrapers for memory, providing ultra-high memory density per unit area.
Unlike HBM, which is commonly used in Nvidia's high-end graphics cards, both use TSV stacking, but HBM pursues extreme data exchange rates, while 3DS focuses on memory capacity expansion while maintaining the existing DRAM standard interface protocols. For AI inference tasks that must load massive numbers of parameters, this is a more pragmatic solution for reducing costs and improving efficiency.
For a long time, Groq's LPU architecture has attracted much attention for its extremely high inference speed, but the SRAM it relies on has very small capacity, making a single chip inadequate when facing models with hundreds of billions of parameters.
Although DRAM's native read/write speed is slightly inferior to SRAM, its large capacity is more advantageous in deploying complex models. Through 3D stacking, Groq attempts to solve the pain point of "not being able to fit" while maintaining its speed advantage, paving the way for inference on ultra-large AI models.