Author | Chen Yuelin Editor | Bao Yonggang The primary market for chips is heating up again. After the previous round of investments in domestic GPUs has gradually entered the realization period, capital has begun to search for the next batch of AI chip companies that might succeed. Near-memory computing, processing-in-memory, 3D near-memory, 3D TokenPU... the names vary, but they are increasingly being grouped under the same label: 3D stacking. Some 3D chip companies have completed two consecutive rounds of financing within half a year, with a cumulative scale approaching 1 billion RMB; for others, even before mass production, valuations have already climbed to 10 billion RMB. However, when investors actually start comparing these companies, they find that behind the similar technical jargon lie different markets, engineering challenges, and commercial validation criteria. "The technical architectures all look pretty similar. We don't know whose product is the best or who can secure major clients," Lao Zhang, an investor who participated in the investment of a listed GPU company, told Leiphone. Having experienced the previous round of chip investments, his institution now prefers to wait until companies complete tape-out and secure real orders before deciding whether to place a bet. Behind this wave of enthusiasm, the bottlenecks in chips are also changing. Agents are turning one-off model Q&A into continuously running tasks, thereby increasing the importance of memory capacity, bandwidth, and data movement efficiency. Bringing data closer to computation has become the common narrative for this round of chip startups, and 3D stacking has emerged as the shared technical means for chiplet reorganization, memory expansion, and the integration of computing and memory. But even though they are all "stacking" chips, what exact problems are different companies trying to solve? Given the constraints on advanced process nodes, HBM, and high-end packaging, why are domestic startups particularly keen on vertically integrating computing and memory? For more technical insights, feel free to add the author's WeChat Evelynn7778 to exchange information.
01. AI Enters the Inference "Marathon" Era, and Chip Bottlenecks Have Changed Agents are turning one-time Q&A short tasks into a "marathon" of continuous model invocations. In this process, the information retrieved, judgments formed, and unresolved issues at each step may be reorganized into subsequent prompts. As the number of model invocations increases and the context length grows, chips not only need to perform more computations but also face a continuous increase in data access volume. In the past, AI chip competition mainly revolved around peak computing power, and GPUs also enabled data to reach computing units faster through multi-level caches, HBM, and advanced packaging. However, in some inference workloads, performance is no longer constrained solely by raw computation speed; data supply efficiency is drawing growing attention. Especially during the Decode phase, computing units often need to frequently read model weights and KV Cache, meaning compute demand and bandwidth demand are not well matched. Continuing to rely on general-purpose GPUs to simultaneously scale computing and memory capabilities may not always be the most cost-effective solution. Designing new chip architectures that are better suited for inference workloads has therefore begun to attract the attention of both capital and the industry. Bob, an investor who has recently been tracking new chip architectures, judges that the explosive growth of inference demand may exceed market expectations. Over the next three to five years, AI computing chips may still remain in a state of supply shortage. Under this expectation, new architectures centered around data movement, such as near-memory computing and processing-in-memory, are rapidly gaining traction. 3D stacking has become one of the integration methods to support these routes. "3D stacking is essentially a form of chip integration. To save area and shorten transmission distances, different chips are stacked vertically. This includes logic and memory, as well as logic-to-logic and memory-to-memory," He Hui, Semiconductor Analysis Director at Omdia, told Leiphone. This also means that companies bearing the same "3D stacking" label may have completely different stacking targets, technical goals, and final products.
02. 3D Integration with No Single Answer: Reorganizing Chiplets, Expanding Memory, and Reconstructing Computing-Memory Relations When 3D stacking shifts chip integration from planar to vertical, the way a company defines the problem determines the stacking outcome. After reviewing the public statements of domestic and international companies, Leiphone found that while companies seem to share the same "3D" label, the industrial logic behind the three types of solutions is not the same. Solutions such as TSMC (Taiwan Semiconductor Manufacturing Company) SoIC, Intel Foveros, and AMD MI300 mainly continue the scaling of chip performance by disaggregating and recombining logic (including computing, I/O, etc.) chiplets. This route does not redefine the computing paradigm; rather, in the context where advanced process node scaling is becoming increasingly expensive, it allows chiplets with different functions and process nodes to continue scaling synergistically.
AMD Instinct MI300 Series Processors Image source: AMD official CDNA 3 Architecture whitepaper
HBM, HBF, and 3D V-Cache, on the other hand, address the memory issue: packing more data into a limited packaging area and increasing the speed at which data reaches the computing units. They target bandwidth, capacity, and cache hierarchy respectively, but the core remains strengthening the memory supply capability around the computing chips. The third type of solution further changes the spatial relationship between computing and memory. Unigroup Guoxin, Rockchip, and multiple domestic 3D near-memory chip startups are attempting to vertically integrate logic (computing) with memory units such as DRAM; solutions like Samsung HBM-PIM, Witmem, and Weina Hexin, on the other hand, further move part of the computation to the memory side. Unlike overseas leading companies that continue to expand system scale using chiplets, HBM, and advanced packaging, new domestic projects are more focused on the vertical integration of logic and memory. This is no coincidence. "The only ones truly capable of participating in 3D memory competition are Samsung, SK Hynix, Micron, CXMT (ChangXin Memory Technologies), and YMTC (Yangtze Memory Technologies Co.)." In the view of AI chip expert Qin Yi, such technologies are usually held by a few enterprises possessing memory wafer manufacturing capabilities. Furthermore, the chiplet route also relies on mature die-to-die interconnects, software, and industrial ecosystems. For most domestic startups, rather than directly entering the competition dominated by IDMs (Integrated Device Manufacturers) and foundries, it is better to reorganize existing computing and memory resources at the chip architecture and packaging levels. Against the backdrop of constraints on advanced process nodes, HBM, and high-end packaging capabilities, 3D stacking therefore also carries a special expectation: using architecture and integration innovation to make up for the gap in manufacturing processes. According to He Hui's observations, constrained by process conditions, compared to overseas companies stacking advanced packaging on advanced logic process nodes like 3nm and 2nm, domestic companies are more likely to implement 3D stacking on relatively mature nodes such as 7nm. However, she also noted that there is currently no unified technical standard for advanced packaging. As long as there is the capability to integrate memory and logic, the specific interconnect scheme used in between is not the focus. But 3D stacking itself will not automatically yield more powerful chips. Different solutions ultimately still need to answer the same question: can higher bandwidth and shorter data distances truly translate into advantages in performance, cost, and token throughput?
03. From Architecture to Product, 3D Chips Still Need to Cross the Engineering Hurdle In the narratives of many 3D chip companies, increasing bandwidth is the core selling point. According to Qin Yi's observations, some domestic 3D chips, constrained by mature process nodes, exhibit the characteristics of "relatively limited computing power and high bandwidth," which perfectly corresponds to the Decode phase that is more sensitive to memory bandwidth. Based on this, he believes that companies should focus on whether they can run the Decode process on real chips, and preferably further complete the validation of the Attention operator in Decode, ultimately demonstrating a sufficiently high TPS (Tokens Per Second). "High bandwidth must ultimately be realized as token throughput," Qin Yi emphasized. However, this also means that 3D chips are just one link in completing the inference chain. "The true value of 3D chips lies in collaborating with GPUs through division of labor and reducing the cost of the Decode phase." Qin Yi is not optimistic about companies that package single-point performance metric optimizations as a replacement for general-purpose computing power. While high bandwidth explains the value of 3D chips, how internal dies are physically stacked determines whether this path is viable. Qin Yi analyzed that compared to the memory dies within HBM, the computing die and memory die have significant differences in size, power consumption, and temperature distribution. This means that uneven stress, warpage, and heat dissipation issues may arise during thermal cycling. Qin Yi also noted the constraints of heat dissipation and yield. He believes that currently, reducing the number of stacking layers may be a more realistic engineering choice. A technical executive from a 3D chip company also told Leiphone that for heterogeneous stacks combining compute logic and memory wafers, 4–6 layers is a more realistic engineering range at this stage. Furthermore, how many layers a chip can be stacked still requires a comprehensive judgment in conjunction with the entire packaging system. Taking TSMC's CoWoS advanced packaging as an example, Qin Yi pointed out that how many sets of HBM are paired with a computing die does not solely depend on the desired capacity. It requires comprehensive consideration of factors such as die size, interposer area, routing, power supply, and overall packaging outline. However, even if the hardware completes stacking and packaging, this does not mean that 3D chips already possess commercial value. 3D integration changes the data organization between computing and memory, and the software stack also needs to re-understand the new memory hierarchy and data flow patterns. Qin Yi believes that to judge a startup's software capabilities, one needs to examine its ISA (Instruction Set Architecture), programming model, compiler, runtime, operator library, and framework adaptation. Even if the hardware provides higher bandwidth, if the software cannot effectively schedule the new chip architecture, the theoretical advantages may still remain on the architecture diagram. For 3D chip companies that are still in the exploration stage, translating a technical path into a commercial product requires clearing at least three hurdles: proving the performance and cost advantages for target workloads on real chips, solving heat dissipation, warpage, and yield issues in packaging manufacturing, and then achieving scaled delivery through software adaptation and customer validation. This also means that capital cannot judge projects solely by the "3D stacking" label. Reorganizing logic chiplets, expanding memory capabilities, and reconstructing computing-memory relations face different markets and require different validation criteria. But ultimately, all routes still need to answer the same question: can higher bandwidth, shorter data distances, and more aggressive stacking structures truly translate into token throughput, unit cost, and customer orders? There is no standard answer for how many layers to stack in 3D chips or how to stack them, but there is a set of identical validation criteria for how to move from architecture to product.