The explosive growth in AI compute demand is pushing HBM storage toward a pivotal architectural crossroads. At this year's Hot Chips conference, Samsung, Micron, SK Hynix, and NVIDIA unveiled their respective technology roadmaps, with a highly concentrated set of core themes: HBM is transitioning from a standardized commodity to a customizable platform, three-dimensional vertical stacking has emerged as the shared direction for next-generation architectures, and thermal dissipation and power constraints have become the primary bottlenecks limiting expansion.

Samsung proposed a three-phase roadmap that evolves the HBM base die from a mere communication layer into a customizable AI platform, and disclosed its ZHBM concept—which involves stacking HBM directly vertically atop the XPU. The goal is to reduce total DRAM power consumption by approximately 70% compared to HBM5, achieve absolute power savings exceeding 100W, and deliver a bandwidth improvement of more than 2.3 times. Meanwhile, NVIDIA disclosed that its labs now possess at least three RVA23 processors and completed the first public demonstration of CUDA running on RISC-V hardware, showcasing the specific pathway for third-party custom CPUs to integrate into the NVLink Fusion ecosystem.

From a market impact perspective, the continued acceleration of AI capital expenditure by hyperscale cloud providers served as the core narrative thread throughout the conference. Storage analyst Jim Handy noted that HBM produces only one-third the GB per wafer compared to standard DDR, and given that the industry has barely constructed new wafer fabs over the past decade-plus, new capacity—even under aggressive timelines—requires roughly two years to come online. The industry's ability to respond to current demand shocks is severely constrained. HBM spot prices have already risen approximately sevenfold from previous levels, with Samsung, SK Hynix, Micron, and major NAND suppliers all recording exceptionally strong revenue growth.

The HBM Supply-Demand Dynamic: Capacity Bottlenecks Unlikely to Ease in the Near Term

Handy's analysis outlines the core contradiction currently facing the memory market. On the demand side, virtually all mainstream AI accelerators—including NVIDIA GPUs, Google TPUs, and various custom silicon—depend on HBM. Even some networking components have begun adopting HBM, creating a new demand category that barely existed a few years ago. On the supply side, because historical process scaling was sufficient to meet bit growth, DRAM suppliers have not undertaken large-scale capacity expansion for over a decade, making any near-term new capacity virtually impossible.

Handy also rebutted the common argument that algorithmic efficiency gains will suppress hardware demand. He believes that when a new technology reduces memory requirements by a factor of six, the typical response from hyperscale cloud providers is to process six times the number of tokens with the same budget, rather than cutting capital expenditure. Efficiency improvements change the productivity of AI infrastructure, but not necessarily the total amount of money invested in it.

The expansion of AI infrastructure has simultaneously tightened the NAND and HDD markets. Meta's testing of low-cost QLC SSDs deployed between TLC SSDs and HDDs revealed performance improvements, prompting broader SSD adoption in hyperscale data centers. AI infrastructure's pull on HDD demand has also created shortages, with QLC SSDs beginning to fill new storage tiers and even replace out-of-stock HDDs, pushing NAND into a shortage state that benefits Kioxia and SanDisk (SNDK).

Micron's Perspective: Silicon Consumption, Thermal Dissipation, and Reliability as Architectural Constraints

Ragu, Micron's HBM design architect, quantified the cost of HBM scaling from both cost and physical limit dimensions. In a typical GPU package containing four 12-layer HBM stacks, the total memory silicon is approximately eight times that of the GPU silicon, meaning roughly 90% of the silicon in the system-in-package is memory-related. A single DRAM die failure can impact the entire package, with reliability challenges intensifying significantly as stack height increases.

Ragu cited Meta's published Llama 3 research data: approximately 17% of unexpected training interruptions are attributed to HBM. This figure directly reveals the substantial impact of HBM reliability on大规模 AI training clusters. Thermal issues have risen to become the foremost architectural constraint. The HBM base die is one of the hottest regions, with heat generated at the bottom while cooling solutions sit atop the DRAM stack, forcing heat to traverse every layer. Micron indicated that HBM is increasingly designed around thermal limitations from the outset, with localized power density and hot spots becoming as important as total power consumption.

Regarding stacking limits, Micron believes 16-layer HBM is technically feasible, but the 20-layer stacking discussed by JEDEC still faces numerous unresolved challenges—TSV density, power density, thermal dissipation, mechanical integrity, and manufacturing yield all pose stringent constraints, rather than signal transmission distance itself (the entire stack height remains below approximately 1 millimeter).

On the bandwidth expansion path, HBM achieves performance leaps through extreme parallelization: HBM3E has 128 banks per DRAM die, increasing to 256 banks in HBM4. External interface I/O lines have doubled from approximately 1,000 to 2,000 while the interface length (shoreline) remains largely unchanged. HBM has thus expanded from HBM1's 128 GB/s bandwidth and 1 GB capacity to HBM4's bandwidth exceeding 2.8 TB/s and capacity above 24 GB per stack—at the cost of continuously rising silicon consumption intensity.

Micron anticipates that as AI workloads become more segmented, HBM will evolve from a general-purpose product toward customized architectures tailored to specific workloads, with processors and paired memory increasingly co-optimized. Packaging technology will also advance from micro-bumps and thermal compression bonding toward fusion bonding and hybrid bonding with single-digit micron pitch.

Samsung's Roadmap: Base Die Evolution and the ZHBM Vision

Samsung's three-phase roadmap represents one of the most forward-looking technology disclosures at this year's conference. The first phase focuses on migrating the memory controller from the XPU to a customized HBM base die. Samsung estimates the controller occupies approximately 5% to 10% of XPU silicon area; freeing this space for compute cores could yield a 10% to 20% performance improvement. Samsung noted that most customers are actively exploring this architecture.

The second phase leverages unused edge regions (shoreline) around the customized HBM base die, using dedicated controllers and PHYs to directly connect a second tier of memory, with expected latency and bandwidth performance superior to PCIe-based expansion solutions. This second-tier memory could be LPDDR or even HBF. Samsung believes that as context windows and KV caches expand, capacity is approaching parity with bandwidth in importance.

The third phase is ZHBM: eliminating the traditional 2.5D interposer and stacking HBM directly and vertically integrated atop the XPU. By removing lateral PHY/D2D paths, I/O and TSV structures can span the entire chip projection area, targeting energy efficiency of approximately 0.5 pJ/bit, with total DRAM power consumption reduced by about 70% compared to HBM5 and absolute power savings exceeding 100W. However, current thermal limitations point to approximately 4-layer stacking rather than 12 or 16 layers, leaving considerable distance from engineering deployment.

Samsung remains measured in its compute offloading strategy: thermal constraints make placing dense compute units in the base die impractical, so the priority is offloading memory-bound Attention computation in LLM inference to HBM, while compute-intensive Prefill and FFN remain on the XPU. On interface strategy, Samsung currently favors proprietary HBM vendor interfaces over UCIe, citing UCIe's larger area footprint and higher power consumption. Its Heat Path Block can reduce peak temperatures by more than 35% when covering sufficient hot spot regions.

ZHBM also requires wafer-level integration and hybrid copper bonding at sub-6 micron pitch, necessitating the elimination of the long-standing boundary between DRAM and SoC design. Both must be co-designed from the outset around routing, thermal dissipation, power density, and physical interconnect.

SK Hynix: Packaging Limits and the Hybrid Bonding Pathway

SK Hynix's presentation focused on the packaging engineering challenges of scaling from 12-layer to 16-layer and even 20-layer stacks. In its 16-layer HBM3E test chip, even with total stack thickness increasing from 720 microns to 775 microns, individual DRAM die thickness still needed to be reduced by approximately 10% compared to the 12-layer configuration, with inter-die gaps shrinking by about 50%, dramatically increasing the difficulty of chip warpage control and gap filling. SK Hynix noted that HBM4's total thickness of 775 microns is approaching the practical limit of current packaging methods, and future scaling cannot rely indefinitely on increasing package height.

Regarding hybrid bonding, SK Hynix believes it will become highly attractive around the 20-layer mark but does not expect adoption in the HBM4E phase. In a hypothetical 20-layer stack, die thickness could increase by approximately 20% to 24%, with thermal conductivity improving by about 35%. Even so, hybrid bonding cannot solve the base die local hot spot problem—SK Hynix is developing IHBM, which adds silicon heat spreader blocks above hot spot regions to improve localized thermal dissipation, and this must be co-optimized from the initial design phase rather than retrofitted later.

SK Hynix also judges that training and inference may ultimately adopt different memory architectures: training requires balancing high bandwidth with high capacity, supporting continuous stack height increases; inference may employ smaller-scale, extremely high-bandwidth memory pools, placing bandwidth-insensitive data in LPDDR or other lower-cost tiers.

NVIDIA and RISC-V: A New Pathway for CUDA Ecosystem Openness

NVIDIA's presentation marks substantive progress in its AI ecosystem strategy. A year ago, the biggest obstacle to bringing CUDA to RISC-V was the lack of suitable RVA23 hardware. Today, NVIDIA's labs possess at least three RVA23 processors, and SiFive completed the first public demonstration of CUDA running on RISC-V hardware during Hot Chips.

NVIDIA's presenter Franz made clear that the starting point is compliance with the RVA23 profile and the RISC-V server platform specification—NVIDIA does not wish to create a separate RISC-V specification for CUDA. CUDA's specific additional requirements amount to only about two pages, with the core elements being PCIe cache coherence (avoiding explicit CPU cache operations during GPU DMA) and PCIe peer-to-peer communication (supporting direct data exchange between GPUs).

NVLink Fusion provides the commercial logic for this pathway: customers can use custom CPUs or accelerators while retaining most of NVIDIA's rack-level architecture. RISC-V CPUs can thus replace NVIDIA's ARM-based host processors in dedicated systems, with the key requirement being integration of NVIDIA's C2C network—which delivers approximately five times the bandwidth and coherence of PCIe. Franz's clear position is that RISC-V's appeal lies not in deficiencies of x86 or ARM, but in the ability of numerous vendors to create customized implementations for important niche markets that may not receive dedicated support from large incumbent CPU vendors.

Supporting NVIDIA's push, Canonical has already set RVA23 as the official RISC-V baseline in Ubuntu 26.04 LTS, with approximately 95% of the regular Linux software package archive now available. RISC-V International and SiFive's Krste Asanović indicated that multiple vendors will bring RVA23 server-grade chips to market this year, with most matrix extension work expected to complete ratification within the next 12 to 18 months. The co-maturation of RISC-V software and hardware ecosystems is accelerating.