Over the past decade, GPU computing power has increased by 1,000 times, while memory bandwidth has only grown by 30 times. Your H100 spends most of its time not computing—but waiting for data. HBM is one of the most expensive, bottleneck-prone, and profitable segments in the entire AI chip supply chain. In the BOM cost of a single B200, HBM3e accounts for nearly half. This article thoroughly explains HBM, from its underlying principles to the industrial landscape.
I. The Memory Wall: Raw Computing Power is Useless Without Sufficient Data Supply
1.1 The Deadlock of the Von Neumann Architecture
All modern computers are based on the Von Neumann architecture: the computing unit and the memory unit are separated. Data is stored in memory; when the CPU/GPU needs to compute, it loads data from memory, and writes it back after computation.
This architecture has an inherent contradiction: computing units are getting faster and faster, while memory units cannot keep up. Over the past decade, the FP16 computing power of GPUs has surged from 21 TFLOPS on the P100 to 4,500 TFLOPS on the B200, a more than 200-fold increase. However, memory bandwidth has only grown from 732 GB/s on the P100 to 8 TB/s on the B200, an approximately 11-fold increase.
The gap is widening. The result is: GPUs spend a significant amount of time "waiting for data"—ALUs idle, with utilization rates below 50%.
This is known as the "Memory Wall." To understand this using the Roofline model: the actual performance of a chip is constrained by two ceilings—peak compute (Compute Roof) and memory bandwidth (Memory Roof). When your workload is compute-intensive (e.g., large matrix multiplication), performance is limited by the compute ceiling; when your workload is memory-intensive (e.g., Softmax and LayerNorm in Attention), performance is constrained by the bandwidth ceiling.
Unfortunately, during the inference process of large models, memory-intensive operations account for a highly significant proportion. The reading and writing of the KV Cache, and the computation of Attention Scores—every step tests the limits of memory bandwidth.
1.2 Why GDDR Falls Short
Traditional GPUs use GDDR (Graphics DDR) for memory. The philosophy behind GDDR is to "widen the data lines and increase the frequency"—GDDR6X has already reached a pin rate of 19.5 Gbps and a bus width of 384-bit.
However, GDDR has a physical ceiling: the number of pins around the chip is limited. Only a certain number of GDDR chips can be placed around a GPU package, and each chip has only a 32-bit data bus. Even if fully populated with 12 GDDR6X chips, the total bus width is only 384-bit. At this bus width, achieving HBM-level TB/s bandwidth would require pin rates at unrealistic frequencies.
The solution for HBM is completely different: instead of placing chips around the edge, they are stacked vertically.
II. How HBM Works
2.1 3D Stacking + Through-Silicon Vias
The core of HBM is 3D stacking. Multiple layers of DRAM chips are stacked vertically like building floors, with each layer connected by TSVs (Through-Silicon Vias). At the very bottom is a logic die (Base Die), responsible for communicating with the GPU.
• Top layer → 12th layer DRAM Die
• ...... (Middle 10 layers of DRAM Dies, connected via TSVs + microbumps)
• 1st layer DRAM Die
• Bottom layer → Logic Die (Base Die, containing PHY interface and controller)
• Below → Silicon Interposer → Connected next to the GPU Die
TSV is the most critical technology in HBM. It involves drilling thousands of micrometer-scale vertical holes in the chip, filling them with copper or other conductive materials to achieve vertical interconnections between layers. The number of TSVs in HBM3E is six times that of HBM2E, significantly shortening the signal path and reducing the IR Drop (voltage drop) by 75%.
This architecture brings two key advantages:
Extremely wide bus width. HBM3E features 1,024 data lines (a 1024-bit interface), and HBM4 doubles this to 2048-bit. For comparison, GDDR6X has only 32-bit per chip. A wider bus width means enormous total bandwidth can be achieved without pushing frequencies too high—at 9.2-12.4 Gbps per pin, HBM3E achieves a total bandwidth exceeding 1.2 TB/s with a 1024-bit bus width.
Extremely low power consumption. Because the data lines are short (vertically stacked, with signal paths in the micrometer range), the driving power consumption is far lower than that of GDDR, which routes across the PCB (Printed Circuit Board). Compared to HBM2E, HBM3E delivers a 2.5x improvement in performance per watt.
2.2 Parameter Evolution Across Six Generations of HBM
| Generation | JEDEC Standard | Release Year | Bus Width | Pin Rate | Bandwidth per Stack | Max Capacity | Stacking Layers | Number of Channels | Voltage |
|---|---|---|---|---|---|---|---|---|---|
| HBM1 | JESD235 | 2013 | 1024-bit | 1.0 Gbps | 128 GB/s | 4 GB | 4 Layers | 8 | 1.2V |
| HBM2 | JESD235A | 2016 | 1024-bit | 2.4 Gbps | 307 GB/s | 8 GB | 4/8 Layers | 8 | 1.2V |
| HBM2E | JESD235C | 2018 | 1024-bit | 3.6 Gbps | 460 GB/s | 16 GB | 8 Layers | 8 | 1.2V |
| HBM3 | JESD238 | 2020 | 1024-bit | 6.4 Gbps | 819 GB/s | 16 GB | 8/12 Layers | 16 | 1.1V |
| HBM3E | JESD238A | 2023 | 1024-bit | 9.2-12.4 Gbps | 1.2-1.33 TB/s | 36 GB | 8/12 Layers | 16 (32 Pseudo Channels) | 1.1V |
| HBM4 | JESD270-4 | 2025 | 2048-bit | 6.4-12.8 Gbps | 2.0-3.3 TB/s | 64 GB | 4/8/12/16 Layers | 32 (64 Pseudo Channels) | 1.05V |
Data sources: JEDEC standard documents (JESD235/238/270 series), official product specifications from SK hynix/Samsung/Micron. HBM4 represents the theoretical upper limit defined by the JEDEC standard; actual product specifications may vary by manufacturer.
From HBM1 to HBM4, over a span of 12 years: bandwidth increased from 128 GB/s to 3.3 TB/s (25x), maximum capacity grew from 4 GB to 64 GB (16x), the number of channels expanded from 8 to 32, and voltage dropped from 1.2V to 1.05V.
The two most critical leaps:
HBM2E → HBM3: Doubling of channels. Increasing from 8 to 16 channels, although the bandwidth per channel did not change much, the degree of parallelism doubled. Starting with this generation, HBM truly became the standard for large model training—while the A100 could make do with HBM2E, the H100 strictly required HBM3.
HBM3E → HBM4: Doubling of bus width. Increasing from 1024-bit to 2048-bit, the theoretical bandwidth per stack jumps from 1.33 TB/s to 3.3 TB/s. The trade-off is—incompatibility. HBM4 requires a completely new PHY interface and memory controller and cannot directly replace HBM3E. This means all chips using HBM4 (including NVIDIA Rubin) need to redesign their memory subsystems.
2.3 HBM Configurations Across NVIDIA's GPU Generations
| GPU | Architecture | Release Year | HBM Version | Number of HBM Stacks | Total Capacity | Total Bandwidth | TDP per Card |
|---|---|---|---|---|---|---|---|
| P100 | Pascal | 2016 | HBM2 | 4 | 16 GB | 732 GB/s | 300W |
| V100 | Volta | 2017 | HBM2 | 4 | 32 GB | 900 GB/s | 300W |
| A100 | Ampere | 2020 | HBM2E | 5 | 80 GB | 2.0 TB/s | 400W |
| H100 | Hopper | 2022 | HBM3 | 5 | 80 GB | 3.35 TB/s | 700W |
| H200 | Hopper | 2023 | HBM3E | 6 | 141 GB | 4.8 TB/s | 700W |
| B200 | Blackwell | 2024 | HBM3E | 8 | 192 GB | 8.0 TB/s | 1000W |
| B300 | Blackwell | 2025E | HBM3E | 8 | 288 GB | 8.0 TB/s | 1200W |
| Rubin | Rubin | 2026E | HBM4 | — | — | — | — |
Data sources: Official specification sheets for each generation of NVIDIA GPUs and industry reports. B300/Rubin are estimated specifications.
Two notable changes:
Continuous increase in the number of HBM stacks. The P100 had only 4 HBM stacks, while the B200 stacks 8. Each stack must be placed closely next to the GPU die (connected via a silicon interposer); the more stacks there are, the higher the requirements for the area and yield of Advanced Packaging (CoWoS). With 8 HBM stacks + 2 GPU dies, the CoWoS package area of the B200 is already more than 5 times that of the P100.
Surging proportion of HBM costs. The BOM cost of the B200 is approximately $6,400, of which HBM3e (192 GB) accounts for $2,800-3,100—nearly half. In other words, the most expensive component of a B200 is not the GPU die, but the memory. This is also why NVIDIA can achieve a gross margin of 81%—HBM is externally sourced and does not dilute the gross margin.
III. The Battle of the Big Three
3.1 Market Landscape
| Manufacturer | 2025 HBM Market Share | 2026 HBM Market Share (E) | HBM Revenue as % of Total DRAM Revenue | HBM4 Progress | Key Customers | Core Advantages |
|---|---|---|---|---|---|---|
| SK hynix | ~50-55% | ~50% | 44% → Expected to exceed 50% | 12-layer samples demonstrated, mass production in 2H 2026 | NVIDIA (Largest customer) | MR-MUF packaging, first-mover advantage in HBM3E |
| Samsung | ~35-40% | ~35% | ~30% | Passed NVIDIA certification in 2026 | AMD / Other cloud providers | TC-NCF packaging, full supply chain integration |
| Micron | ~5-10% | ~10-15% | ~25% | Passed NVIDIA certification in 2026 | NVIDIA (HBM3E) | Overtaking on the curve with HBM3E, domestic US capacity |
Data sources: TrendForce HBM Industry Analysis (1Q26), SiliconAnalysts, financial reports of each manufacturer. Market shares are approximate values.
SK hynix is the absolute king of HBM. It was the first manufacturer to mass-produce HBM3E (in 2024) and secured the vast majority of orders for NVIDIA's H200/B200. In 2025, HBM revenue as a proportion of total DRAM revenue surged to 44%, and is expected to exceed 50% in 2026—HBM has already replaced traditional DDR as SK hynix's most important source of revenue.
Samsung's situation is rather delicate. It was one of the first manufacturers to announce HBM4, but fell behind SK hynix during the HBM3E phase and was pressured by NVIDIA for a while. The turning point came in June 2026—Jensen Huang publicly confirmed that Samsung had passed NVIDIA's HBM4 certification, meaning Samsung will not fall behind in the next generation of products.
Micron is the most interesting player. It has long been the "third place" in the HBM market, with a share of only 5-10%. However, Micron achieved an overtake on the curve with HBM3E—skipping HBM3 to directly mass-produce HBM3E, and securing some orders from NVIDIA. In 2026, its HBM4 also passed NVIDIA certification. Micron's strategy is to bet on the next generation and skip the previous one, using the technological generation gap to make up for its lower market position.
3.2 Key Changes in HBM4
HBM4 is not simply "doubling the bandwidth of HBM3E." It introduces several fundamental architectural changes:
2048-bit interface. Doubling the bus width means a massive increase in physical design complexity. The PHY interface requires twice the number of I/O pins, and the chip's bump map must be redesigned. This is also why HBM4 is incompatible with HBM3E—the physical layers are completely different.
Logic die integration. HBM4 integrates more logic functions into the Base Die—not just the PHY and controller, but potentially simple data processing units. This shifts HBM from "passive storage" to an "active co-processor." Samsung has already demonstrated offloading some computing tasks to the logic die of HBM for execution.
16-layer stacking. HBM3E maxes out at 12 layers, while HBM4 supports up to 16 layers. More layers mean greater capacity (up to 64 GB per stack using 32 Gb layers), but also more difficult heat dissipation—stacking 16 layers of DRAM together makes it hard to dissipate heat from the middle layers. Thermal management is one of the biggest engineering challenges for HBM4 mass production.
All three manufacturers passed NVIDIA certification. On June 5, 2026, NVIDIA confirmed that SK hynix, Samsung, and Micron have all obtained HBM4 supply certification. This is the first time in HBM history that the big three are supplying synchronously in the same generation—for NVIDIA, this means supply security; for the big three, it means competition will be fiercer.
IV. Panoramic View of Memory Solutions for AI Chips
HBM is not the only option. For different AI chip scenarios, the logic for selecting memory solutions is completely different.
| Solution | Bandwidth | Capacity | Power Consumption | Cost | Typical Scenarios | Representative Products |
|---|---|---|---|---|---|---|
| HBM3E | 1.2-1.33 TB/s per stack | 24-36 GB per stack | Low (short path for 3D stacking) | Extremely high ($2,800+/stack) | Large model training, high-end inference | B200, MI300X |
| HBM4 | 2.0-3.3 TB/s per stack | Up to 64 GB per stack | Lower (1.05V) | Extremely high (new product premium) | Next-generation large model training | Rubin (Estimated) |
| GDDR7 | ~1.5 TB/s (384-bit, 32 Gbps) | 24-48 GB (12x4 GB) | Medium-High | Medium ($200-400/chip) | Mid-range inference, gaming GPUs | RTX 5090 |
| LPDDR6 | ~200 GB/s (128-bit) | 16-64 GB | Extremely Low | Low | Edge-side inference (smartphones/laptops) | Apple M4, Snapdragon |
| CXL Memory Pool | Depends on PCIe/CXL bandwidth | TB-level (scalable) | Medium (additional network overhead) | Medium | Large model inference (KV Cache expansion) | Samsung CMM-D |
| In-Memory Computing | Extremely high (eliminates data movement) | Limited | Extremely Low | R&D stage | Future inference acceleration | Mythic, Witmem Technology |
Data sources: JEDEC standards, product specifications from various manufacturers, industry research reports. GDDR7/LPDDR6 are typical value estimates based on standards.
A notable trend: CXL memory pooling is becoming a critical supplement for large model inference. The KV Cache in large model inference is highly memory-intensive—for a 175-billion parameter model, the KV Cache can consume hundreds of gigabytes. With limited HBM capacity (the B200 has only 192 GB), CXL can pool and share the DDR memory of multiple servers, serving as a "Level 2 cache" for HBM. Samsung's CMM-D (CXL Memory Module) has already been tested in data centers of multiple cloud providers; although its latency is higher than HBM, its capacity can easily scale to the TB level.
V. Domestic HBM: The Most Bottleneck-Prone Segment
5.1 Where the Gaps Lie
| Dimension | International Mainstream (SK hynix/Samsung/Micron) | Domestic (CXMT) | Gap |
|---|---|---|---|
| Current Mass Production | HBM3E (5th Gen) | HBM3 engineering samples | ~1 Gen |
| Stacking Layers | 8-12 Layers | 7 Layers | Close |
| Capacity per Chip | 24-36 GB | 24 GB | On par |
| Bandwidth | 1.2-1.33 TB/s per stack | >800 GB/s per stack | ~65% |
| Cost | Baseline | ~40% higher | Clear disadvantage |
| HBM4 Roadmap | Mass production in 2H 2026 | HBM3E targeted for 2027 | ~2 Gens |
| DRAM Process | 1β/1γ (12-14nm class) | G4 (16nm class) | ~1-2 Gens |
Data sources: Comprehensive synthesis of public information from CXMT, industry research reports, and tech media reports. Data for international manufacturers is based on public product specifications.
The core bottleneck for domestic HBM is not the number of stacking layers—the gap between 7 and 12 layers can be caught up through packaging processes. The real gaps lie in three areas:
DRAM process lags by 1-2 generations. The underlying DRAM cells for HBM and traditional DDR share the same process platform. CXMT's G4 process is at the 16nm class, while Samsung and SK hynix have reached 1β (12-14nm class). A more advanced DRAM process means smaller cell area, lower power consumption, and higher yield—these directly determine the cost and performance of HBM.
TSV yield. 12-layer stacking means drilling thousands of TSV holes per HBM chip; a defect in any single hole will cause the entire chip to be scrapped. SK hynix's TSV yield has been refined over more than a decade, while CXMT is still in the early ramp-up phase in this regard.
Packaging equipment. HBM manufacturing requires dedicated TCB (Thermocompression Bonding) and MR-MUF (Mass Reflow Molded Underfill) equipment. The core suppliers for this equipment are in Japan and South Korea, and there are restrictions on exports to mainland China.
5.2 Hopes for Breaking Through
Several positive signals:
Joint R&D by CXMT and YMTC. In September 2025, CXMT and YMTC (Yangtze Memory Technologies Co.) announced a collaboration to develop HBM: CXMT provides the DRAM technology foundation, while YMTC's Xtacking bonding process is used to enhance HBM's bandwidth and heat dissipation. Xtacking is YMTC's proprietary technology—manufacturing logic circuits and memory arrays separately before bonding them together, which theoretically can significantly improve HBM's I/O performance.
CXMT has delivered HBM3 engineering samples to Huawei. According to industry reports, CXMT's HBM3 samples have been validated on Huawei's Ascend platform. Kaifa Technology's HBM3E packaging products have also passed sample validation by leading AI chip manufacturers such as Huawei Ascend and Hygon Information. The technological closed loop of "domestic AI chips + domestic HBM" is taking shape.
Capacity is expanding. CXMT plans to achieve full mass production of HBM3 in 2026, with a target to conquer HBM3E in 2027. Although slower than international giants, the progress is accelerating—they originally planned to start developing HBM2 in 2026, but have already delivered HBM3 samples in 2025.
5.3 Panoramic View of China's Memory Industry Chain
| Company | Field | Core Products | 2025/2026 Progress | Market Position |
|---|---|---|---|---|
| CXMT | DRAM + HBM | DDR4/DDR5/LPDDR5, HBM3 | HBM3 samples delivered to Huawei, mass production in 2026 | China's No. 1, Global No. 4 in DRAM |
| YMTC | 3D NAND + Bonding | 3D NAND Flash, Xtacking | Collaborating with CXMT on HBM, Xtacking process output | Global Top 5 in NAND |
| Montage Technology | Memory Interface Chips | DDR5 RCD/DB, CXL MXC | Mass production of CXL memory expansion controllers | Global Top 3 in DDR5 interface chips |
| GigaDevice | NOR Flash + DRAM | NOR Flash, Niche DRAM | DRAM product line expansion | Global Top 3 in NOR Flash |
| Giantec Semiconductor | EEPROM | SPD EEPROM (memory module configuration chips) | DDR5 SPD grows with DDR5 penetration rate | Global Top 3 in EEPROM |
| Kaifa Technology | Memory Packaging and Testing | DRAM/NAND packaging, HBM packaging and testing | HBM3E packaging validated by Huawei | Domestic leader in memory packaging and testing |
| Longsys | Memory Modules | SSDs, memory modules, embedded storage | Enterprise SSDs enter major internet companies | Domestic leader in memory modules |
Data sources: Annual reports/prospectuses of each company and public industry information. Market positions are approximate evaluations based on public data.
VI. Interview Focus Areas and Career Directions
6.1 High-Frequency Interview Topics
• Roofline model: Given compute and bandwidth, determine whether a workload is Compute-bound or Memory-bound
• Bandwidth requirement derivation: Taking LLaMA-2 70B inference as an example, estimate the parameter volume and KV Cache volume to be read for generating each token, and calculate the minimum bandwidth requirement
• Memory hierarchy design: Register → L1 → L2 → HBM → CXL/NVMe, the logic of capacity and bandwidth allocation at each level
• Working principles and process challenges of TSV (aspect ratio, thermal expansion coefficient matching, yield)
• Interface changes between HBM3E and HBM4: Why is HBM4 incompatible with HBM3E?
• Why is HBM's power consumption lower than GDDR's?—Short physical path + low voltage + low frequency with high parallelism
• Thermal challenges of HBM: Thermal coupling in 3D stacking, heat conduction paths in middle layers
• DRAM cell structure (1T1C), refresh mechanism, why is refresh needed?
• Latency differences between Row Buffer hits and misses
• Organizational hierarchy of Bank, Rank, and Channel
• Pseudo Channel design of HBM—Why does HBM3E have 32 pseudo channels but only 16 physical channels?
6.2 Roles and Salaries
| Role Direction | Responsibilities | Core Skills | Fresh Master's Graduate (Annual Salary) | 3-5 Years Experience | Talent Shortage Level |
|---|---|---|---|---|---|
| DRAM Design | DRAM cell design, array architecture, refresh control | Analog IC, semiconductor physics, Cadence | 300k-500k RMB | 550k-900k RMB | Relatively scarce |
| HBM Architecture | HBM stacking architecture, TSV design, PHY interface | 3D IC design, signal integrity, thermal analysis | 350k-550k RMB | 700k-1.3M RMB | Extremely scarce |
| Memory Controller | DDR/HBM controller RTL design | Verilog, AMBA, DDR protocol | 300k-450k RMB | 550k-850k RMB | Relatively scarce |
| Advanced Packaging | TSV/microbumps/underfill in HBM packaging | Packaging process, TCB/MR-MUF equipment, reliability | 280k-450k RMB | 500k-800k RMB | Relatively scarce |
| CXL/Interconnect | CXL protocol stack, memory pooling architecture | PCIe/CXL protocol, system architecture | 300k-500k RMB | 600k-1M RMB | Moderate |
Salary data is synthesized from the CSDN 2026 Hardware Engineer Salary Panorama, public job postings on Boss Zhipin/Liepin for 2026 Q1-Q2, and industry headhunter interviews. These are approximate ranges for pre-tax annual salaries (including bonuses).
Conclusion
HBM is the most "quietly profitable" segment in the entire AI chip supply chain. NVIDIA sells a single B200 for $30,000 to $40,000, but nearly half of the cost is HBM—SK hynix, Samsung, and Micron are the real "arms dealers."
For professionals in the chip industry, there are three reasons to make long-term bets on the HBM direction: First, high demand certainty—as long as AI computing demand grows, HBM demand grows, and the capacity and bandwidth requirements for HBM are growing even faster than computing power; Second, high technical barriers—HBM involves DRAM design, 3D stacking, TSV processes, Advanced Packaging, thermal management, and high-speed SerDes, featuring extremely high interdisciplinary thresholds that make it hard to replace; Third, massive potential for domestic substitution—if the joint R&D by CXMT and YMTC can achieve a breakthrough, the last "bottleneck" segment in the entire domestic AI chip supply chain will be cleared.
When interviewing for HBM-related roles, don't just memorize parameters. What interviewers really want to hear is: your ability to explain from a physical perspective why HBM consumes less power than GDDR, your ability to analyze from a system perspective where the bandwidth bottlenecks lie in large model inference, and your ability to clarify from an industry chain perspective the root causes of the gap between domestic HBM and international giants.
Next Issue Preview: Part 5—Advanced Packaging: The Real "Material Stacking" Battlefield for AI Chips. An in-depth breakdown of the three major routes: CoWoS, EMIB, and Foveros, along with a comprehensive analysis of the packaging cross-section of a B200.
Comprehensive Sources for This Article: JEDEC JESD235/238/270 series standard documents, official HBM product specifications from SK hynix/Samsung/Micron, official specification sheets for each generation of NVIDIA GPUs, TrendForce HBM Industry Analysis (1Q26), official documentation for Google Cloud TPU v7, public information from companies such as CXMT/YMTC/Montage Technology/Kaifa Technology, CSDN 2026 Hardware Engineer Salary Panorama, SiliconAnalysts HBM market analysis, and EE Times/Blocks & Files CES 2026 HBM4 reports.