GPT-3

Doubling GPT-3 Inference Speed and Eliminating Silicon Furnaces: Vertical Memory Architectures V-Die and MOSAIC Revolutionize HBM Cooling and Bandwidth for AI Accelerators

Researchers from South Korea and Japan have unveiled two groundbreaking vertical memory stacking technologies, V-Die and MOSAIC, at the June 2026 IEEE VLSI Symposium that promise to dramatically increase HBM capacity and performance while solving critical heat dissipation problems in AI accelerators. Instead of stacking DRAM dies horizontally with TSV interconnects, both approaches rotate the memory packages sideways so individual dies stand vertically like radiator fins, enabling better cooling and more interconnects. The Korean V-Die design eliminates TSVs entirely, provides four times more connections than HBM4, reduces read latency by 37 percent, and uses microfluidic channels to keep temperatures around 45 °C, delivering 540 tokens per second on GPT-3-scale workloads versus 296 tokens with conventional HBM4 in simulations. The Japanese MOSAIC project replaces physical contacts with miniature inductive coils that enable wireless data transfer at up to 4 Gbit/s per channel, allowing up to 98 dies and 294 GB of memory (theoretically 882 GB) to be placed directly above a GPU with only a one-degree temperature rise. Both technologies remain in the research or early prototype stage and still face manufacturing and cost challenges, yet they demonstrate a viable path around the thermal wall that currently limits further scaling of high-bandwidth memory for next-generation AI hardware.