Abstract
As large model parameters expand and long-context, multi-turn reasoning becomes widespread, the actual computational load and historical cache (KV Cache) of AI servers continue to grow, but data access exhibits a distinct "hot-cold differentiation." In a high-end AI accelerator card, HBM already accounts for nearly 60% of the total Bill of Materials (BOM). When AI vendors are no longer willing to pay HBM prices for every piece of historical data, server storage is shifting from simply stacking GPU memory to system-level collaboration: "HBM for high-frequency core data, DDR5 for expanding host capacity, MRDIMM for increasing channel bandwidth, CXL for breaking slot boundaries, and enterprise SSDs for cross-turn persistence." Consequently, incremental value in the storage supply chain is being redistributed towards host memory interfaces, high-speed interconnects, and enterprise flash.
HBM Is Too Expensive; AI Vendors Begin Redesigning Memory Systems
The recently concluded Q3 2026 reports reveal two prominent signals:
First, memory manufacturers' profit margins have surpassed those of computing power and foundry leaders. In Q3, SK Hynix's operating margin was approximately 80%, Micron's gross margin reached 86.8%, and domestic DRAM manufacturer CXMT's Q3 gross margin hit 89.5% (with quarterly net profit of CNY 54.4 billion), all exceeding NVIDIA (operating margin ~65%) and TSMC (~59%). On a flagship AI accelerator card, the cost of two GPU dies is under USD 1,000, while eight HBM stacks exceed USD 5,000, accounting for nearly 60% of the card's BOM.
Second, downstream players are strictly controlling video memory investment, leading to divergence among A-share storage companies in Q3. Supply chain rumors suggest discussions on adjusting the HBM capacity for some configurations of the next-generation Rubin Ultra from 384–512GB down to 192GB (pending official confirmation, but reflecting downstream sensitivity to video memory costs). Meanwhile, BIWIN Storage Technology (Q3 net profit dropped 62% quarter-over-quarter due to hoarding consumer-grade wafers) and TWSC (Shenzhen Techwinsemi Technology Co., Ltd.) (net operating cash outflow exceeded CNY 10 billion in the first three quarters) faced short-term pressure. Conversely, Montage Technology, the leader in server memory interfaces (Q3 net profit up +162% year-over-year), and Longsys, which saw increased volume in enterprise products (Q3 operating cash flow turned positive), delivered strong performance.
This points to a core question running through this article: If AI vendors are no longer willing to pay HBM prices for every piece of historical data, where will the next wave of spending in the storage industry flow?
During the decoding generation phase, large models must consult historical dialogue features for each new Token generated, known as the KV Cache (Key-Value Cache). With the proliferation of million-Token long contexts and multi-turn Agents, even with sparse architectures, the actual computational load and total KV Cache volume continue to grow. However, data access is highly uneven: only model weights, lightweight retrieval indexes, and recent hot data are accessed frequently, while the vast majority of early historical caches remain in a low-frequency standby "warm-cold state" for long periods.
Why not infinitely increase single-card HBM capacity? Taking HBM4 as an example, interface bandwidth and stacked capacity must be viewed separately: peak bandwidth is determined by the 2048-bit interface width (32 independent channels) and pin speed, while 12-Hi and 16-Hi vertical stacking primarily determines capacity. Under the 775-micrometer package height limit, higher layer stacking dramatically increases challenges related to wafer thinning, Through-Silicon Vias (TSV), heat dissipation, and yield. Combined with explorations into copper-copper hybrid bonding and the introduction of Base Dies using advanced processes, the marginal cost of buying capacity by stacking HBM higher continues to rise. Therefore, not only do cloud vendors strictly control HBM usage in their self-developed chips, but Huawei Ascend 950 also splits pre-fill and decode phases via a "one chip, dual architecture" design, matching them respectively with low-cost memory and high-bandwidth memory.
From GPU Memory to CPU Memory: How Is Historical Cache Migrated?
Models represented by DeepSeek (Dynamic Sparse Attention DSA) and Zhipu GLM-5.3 (IndexShare) establish lightweight retrieval directories (Indexer KV) for historical caches, selecting only the most relevant few historical fragments for computation at each generation step.
The vLLM team's engineering report on GLM-5.3 (1 Million Token Context) provides a reproducible real-world benchmark. Comparing traditional offloading with Hybrid HiSparse on a single 8×H200 server, both used a 512GiB host memory budget (where Hybrid HiSparse allocated 384GiB to HiSparse and 128GiB to other caches). In tests, the `Indexer KV` remained always resident on the GPU HBM side, while the main cache was gradually offloaded to CPU memory only when GPU memory pressure arose, and asynchronously prefetched back to the GPU when selected by the directory. Combined with Kimi Mooncake, Huawei UCM, and DeepSeek 3FS, AI inference storage is forming a four-layer division of labor:
After DDR5, Why Do Servers Need MRDIMM and CXL?
Once historical caches migrate to the CPU side, hardware encounters three sequential engineering bottlenecks, corresponding to the progressive upgrades of DDR5 RDIMM, MRDIMM, and CXL.
Step 1: What if capacity is insufficient? — DDR5 RDIMM solves host memory expansion
Host memory in AI inference servers is expanding from 512GB to 1TB–4TB. The electrical load of thousands of DRAM chips on the motherboard far exceeds the direct drive limits of the CPU, necessitating DDR5 RDIMMs, which use a central RCD (Registering Clock Driver) chip (along with PMIC, SPD, TS) to buffer and amplify address and clock signals. Compared to NVIDIA's Grace CPU, which solder LPDDR5X directly onto the motherboard (locking 480GB per CPU), dual-socket x86 servers with 24–32 pluggable RDIMM slots offer greater flexibility for inference scaling.
Step 2: Capacity is sufficient, but what if bandwidth is insufficient? — MRDIMM improves effective bandwidth per channel
When multiple cards concurrently prefetch caches, standard RDIMM bandwidth becomes strained. MRDIMM (Multiplexed Rank DIMM) allows two ranks on a module to simultaneously prepare 64B of data at native rates without overclocking chips or adding motherboard channels. This data is then multiplexed and output as 128B at double the interface rate via edge buffer chips, boosting speeds to 8,800–12,800 MT/s. Its core interface main chip count per module increases from 1 RCD in traditional RDIMMs to 1 MRCD + 10 MDBs (total 11 chips), significantly enhancing the value per module for Montage Technology, Rambus, and Renesas, though power consumption, thermal management, and platform adaptation thresholds also rise accordingly.
Step 3: What if local slots are insufficient? — CXL breaks the single-server memory wall
When motherboard DIMM slots are fully populated yet still insufficient, CXL (Compute Express Link) leverages PCIe physical interfaces to achieve cache-coherent memory expansion. Currently approaching commercialization is Single-Server Memory Expansion, which involves attaching expansion cards with built-in MXC controllers to PCIe slots to add TB-level memory (e.g., Alibaba Cloud's CXL shared cache pool, Marvell Structera's DDR4 multiplexing solution with hardware compression). Cross-server Memory Pooling remains constrained by latency and software maturity, representing a medium-to-long-term direction.
SSDs Enter AI Inference: What Changes with FDP and HBF?
Step 4: What if historical caches need long-term preservation? — Enterprise SSDs and NVMe FDP
When multi-turn sessions pause or long prompts are shared across requests, storing caches in enterprise NVMe SSDs for direct read-back next time ("Storage instead of Computation") can save redundant computational resources. However, NAND flash writes in small pages and erases in large blocks. Traditional SSDs mix long-lived prompts spanning days with short-lived overflow caches expiring in seconds within the same flash block, triggering frequent background Garbage Collection (GC), leading to severe Write Amplification Factor (WAF) and read tail latency.
NVMe FDP (Flexible Data Placement) allows software to tag and place data in separate blocks based on cache lifespan. Short-lived session blocks can be reclaimed entirely upon termination, significantly reducing WAF to near 1 and greatly improving GC tail latency. This removes wear barriers, and combined with 4nm PCIe 5.0/6.0 controllers, enables QLC enterprise SSDs with capacities of 61.44TB–245TB to enter AI data centers due to their high energy efficiency ratio (TB/W). As for HBF (High Bandwidth Flash) utilizing TSV stacking, limited by microsecond-level flash latency and endurance, it is suitable for storing read-only model parameters or cold weights, rather than replacing HBM.
The AI Storage Value Chain Is Being Redistributed
Based on Q3 2026 reports, SK Hynix, Samsung, and Micron, along with CXMT and YMTC whose global share has risen to approximately 10%, have locked in demand through long-term contracts (CXMT's prepayments and long-term contract deposits reached CNY 46.7 billion by end of Q3). The inflection point for new capacity release is expected between late 2027 and early 2028. As the three major manufacturers exit mature production lines, Dongxin Semiconductor (Q3 net profit up +59% quarter-over-quarter to CNY 830 million) and Ingenic Semiconductor (Q3 net profit CNY 1.419 billion) are capturing niche market benefits. In the segments of interface interconnects, controllers (including T-Head Zhenyue 510), and module packaging/testing, it is crucial to strictly distinguish between "realized current performance" and "incremental growth awaiting verification":
Conclusion: Three Industry Judgments on AI Storage System Reconstruction
1. First, HBM remains the core resource for AI computing. Sparse attention and cache offloading reduce the static residency requirement of HBM for certain workloads but do not eliminate the need for high-bandwidth memory.
2. Second, AI inference is elevating the importance of host memory and cache systems. DDR5 RDIMMs and enterprise SSDs have already realized performance gains, MRDIMMs are ramping up with new platforms, while CXL and HBF lean more towards medium-to-long-term prospects.
3. Third, competition in the storage industry is expanding from single media to the entire memory system. Future competition will involve not just HBM capacity and price, but also memory interface chips, high-speed interconnect controllers, SSD controllers, wafer-level advanced packaging and testing, and cache scheduling software.
Sources:
vLLM GLM-5.3 Hybrid HiSparse Engineering Report; SGLang HiSparse Paper; DeepSeek 3FS, Kimi Mooncake, and Huawei UCM Technical Documentation; JEDEC, CXL Consortium, NVMe FDP Standard Specifications; Company Q3 2026 Reports and Investor Relations Records; Industry Expert Interview Notes (2026-10-18, 2026-10-22).