In October 2026, a series of benchmarks on eight mainstream AI accelerator cards revealed that the Huawei Ascend 950PR achieved a median compute utilization rate of 86.0%, ranking first. While its absolute computing power was not leading, it utilized its own hardware most efficiently.
This reflects a shift in the competitive logic of AI chips: beyond comparing peak computing power, the focus is now on who can truly utilize that power. Starting from the concept of the "Roofline," this article explains why large model inference leaves computing power idle, how Chinese large models are designing architectures to match the characteristics of Chinese chips, and what routes NVIDIA and domestic vendors have chosen when new bottlenecks emerge within the chip.
I. The First Roofline: Why Does Inference Utilization Remain Below 30% Even When Peak Computing Power Doubles?
Chip launches typically emphasize peak computing power, but whether that power can be utilized depends on memory bandwidth, i.e., the speed at which data is transferred from memory to compute units.
The value obtained by dividing peak computing power by memory bandwidth is called the Roofline knee point. Its meaning is: for every 1 byte of data read from memory, how many operations must the compute unit perform to avoid idling. For example, a chip with 500 TFLOPS of computing power and 4 TB/s of bandwidth has a knee point of 125.
Large model inference consists of two phases, falling precisely on either side of the knee point.
In the prefill phase, the model reads the entire prompt at once, allowing the same weights to participate in computation repeatedly. The number of operations per byte of data far exceeds the knee point. Here, the bottleneck is computing power; the stronger the chip, the faster it runs.
In the decode phase, the model generates only one token at a time but must re-read all weights and historical cache. Each byte of data supports only a few to a dozen operations. Here, the bottleneck is bandwidth, and most compute units are waiting for data. When we use large models and see text outputting word by word, the main time consumption occurs in this phase.
Therefore, doubling peak computing power does not equate to doubling inference speed. Only when the model's arithmetic intensity aligns with the chip's knee point can both computing power and bandwidth be fully utilized.
Benchmarks: Who Uses Hardware Most Efficiently?
It was previously widely believed that Chinese chips not only lagged in computing power but also suffered from software stack issues, resulting in low actual utilization. SemiAnalysis benchmarks in October this year yielded different results.
The tests covered eight accelerator cards and common matrix operations found in 1,055 open-source large models. Ranked by utilization, the Ascend 950PR had a median of 86.0%, ranking first; the NVIDIA H100 was 75.5%, B200 was 65.4%, and AMD MI355X was 51.9%.
Note that this is a ranking of utilization, not performance. In terms of absolute computing power, the Ascend 950PR sits between the NVIDIA A100 and AMD MI300X, still trailing behind the H100 and B300. Its high utilization is partly because NVIDIA and AMD flagship cards often hit power limits and throttle down under full load, and partly due to the multi-level on-chip buffer design of the Ascend matrix cores, which allows reading data, computing, and writing results to occur simultaneously.
The manufacturing process determines how high the roof is, while architecture and software determine whether you can reach it.
II. Aligning Algorithms with Hardware: How Large Models Match Chinese Chips from DeepSeek to Zhipu GLM-5?
Low arithmetic intensity in the decode phase is a common issue for all large models. The MLA (Multi-head Latent Attention) architecture proposed by DeepSeek compresses historical cache, increasing the arithmetic intensity during decoding by more than tenfold.
Through derivation, the arithmetic intensity of MLA is primarily determined by one parameter: the number of attention query heads. More heads mean higher arithmetic intensity. This implies that the number of heads in a model gives a rough indication of the type of chip it was designed for.
DeepSeek's 128 Heads
DeepSeek-V3 and R1 use 128 query heads per layer, with a decode arithmetic intensity of 241.8. At the time, they mainly used the NVIDIA H800, which has a knee point of 258.2, showing near-perfect alignment.
Chinese Chips Have Knee Points in Another Range
Constrained by advanced manufacturing processes, new-generation Chinese chips generally choose the route of "moderate computing power, larger memory." Calculating the knee points of major domestic chips in 2026: Iluvatar CoreX TianGai is 122.5, Huawei Ascend 950DT is 125.0, Moore Threads S4000 is 130.2, MetaX XiYun C600 is 138.9, Hygon DeepComputing III is 140.0, Kunlunxin P800 is 143.8, and Biren BR20X is 150.0.
These values cluster between 122 and 150, nearly half lower than the H800. If a model with an arithmetic intensity of 241.8 is placed directly on these chips, computing power will hit the ceiling first, leaving nearly half of the bandwidth idle.
Zhipu GLM-5's 64 Heads
Zhipu reduced the number of query heads from 128 to 64 in GLM-5 and GLM-5.3, while deepening other dimensions to ensure accuracy. The arithmetic intensity subsequently dropped to 120.8, falling precisely into the knee point range of Chinese chips. With unchanged hardware, single-card inference throughput improved by approximately double.
Previously, chip vendors adapted to models; now, Chinese large models are beginning to design architectures based on the characteristics of Chinese chips.
This bidirectional adaptation is driving market changes. Domestic supply of local AI chips in China is expected to double to approximately 3.5 million units in 2026, with non-GPU accelerator shipments exceeding GPU shipments for the first time in the first half of the year. Huawei holds nearly 30% of the domestic accelerator card market share; Huawei, Alibaba T-Head, and Cambricon collectively account for 95% of the non-GPU market share. Vendors such as Hygon, MetaX, Enflame, Kunlunxin, Biren, Moore Threads, and Iluvatar CoreX have also begun batch deliveries to telecom operators and internet clusters.
III. The Second Roofline: Breaking Through Bottlenecks When Matrix Cores Outpace Vector Cores
Beyond memory bandwidth, a second bottleneck has emerged within the chip.
With the widespread adoption of low-precision formats like FP8 and FP4, the speed of matrix multiplication units within the chip has multiplied several times. However, vector units responsible for exponential calculations and data format conversions, as well as on-chip caches, have not sped up synchronously. As a result, after the matrix units finish calculating, they must wait for the vector units to complete processing before continuing.
NVIDIA: Scaling Up Within the Chip, Adding Devices Outside
On Blackwell, the throughput gap between matrix units and exponential calculation units widened to 1024 times. When running DeepSeek-V3, the actual utilization of FP8 was only 20%–37%.
NVIDIA's response involves two steps. Inside the chip, the next-generation Rubin increases the throughput of exponential calculation units by four times. Outside the chip, it invested $20 billion to acquire Groq, using racks composed entirely of pure on-chip SRAM chips specifically for decoding, while Rubin handles prefilling.
Ascend 950 × DeepSeek: Solving It Within the Chip
Huawei and DeepSeek chose another path: modifying hardware and code together to digest the bottleneck within a single chip. SemiAnalysis's breakdown provides three specific examples.
First, establishing on-chip pathways. Attention calculations require repeated scaling of intermediate results, which the Ascend matrix units cannot do alone. Previous generation chips had to send data back to memory for processing. Ascend 950 expanded relevant on-chip caches to 256KB and established a direct connection channel from matrix units to vector units. DeepSeek correspondingly modified FlashMLA to place this part of the calculation within the vector units, ensuring data no longer leaves the chip.
Second, restructuring data formats for hardware. Early FlashMLA stored cache values and scaling factors separately, requiring Ascend to move data twice for each token's cache. DeepSeek merged them into a single continuous record, halving the movement count. This is direct evidence of software accommodating hardware.
Third, compiler-based automatic pipeline scheduling. DeepSeek integrated the Ascend backend into the trunk of its TileLang compiler, allowing the reading of the next block of data to automatically overlap with the calculation of the current block. Benchmarks showed that the time consumption for a single operator dropped from 17.42 microseconds to 10.52 microseconds, a 40% speedup.
When running DeepSeek-V4.1-Flash on 32 Ascend950DT chips, combined with speculative decoding, the average latency for generating each token was 4.33 milliseconds.
Other Chinese vendors are making similar attempts: Kunlunxin P800 uses hardware lossless compression to effectively expand bandwidth; MetaX C600 incorporates a hardware transpose engine to reduce data movement; Iluvatar CoreX TianGai 300 improves long-context attention efficiency to over 90% through multi-core shared caches.
Conclusion
In 2026, the focus of AI chip competition is shifting from peak computing power to effective computing power.
Facing the same problem—insufficient data supply during the decode phase—NVIDIA's approach is to add compute units and devices. Domestically, under constrained manufacturing processes, the choice is mutual adaptation between models and chips: Zhipu adjusts head counts to fit the knee points of Chinese chips, while DeepSeek rewrites underlying code for Ascend.
Absolute computing power gaps remain for Chinese chips, a fact that has not changed. However, software-hardware synergy is enabling more efficient use of limited computing power, representing the most noteworthy change this year.
References and Data Sources:
1. Bryan Shan & Dylan Patel, The Huawei-DeepSeek Symbiosis: Huawei Ascend 950PR Deep dive & µbenchmarks, DSv4.1 Flash Analysis, DeepSeek support for Ascend, AICPU, Tilelang Ascend, SemiAnalysis, October 21, 2026.
2. DeepSeek-AI, FlashMLA, TileLang (Ascend 950 Backend & AutoSchedule), DeepEP-Ascend, DeepInfer, GitHub Official Repositories, September–October 2026.
3. BAAI (Beijing Academy of Artificial Intelligence), FlagOS 2.2 & FlagGems Universal Operator Library Release Notes, October 20, 2026.
4. Hygon Information Technology, Cambricon Technologies, MetaX Semiconductor, Enflame Technology, Moore Threads, Iluvatar CoreX, Biren Technology, Q3 2026 Reports, H1 2026 Reports, and IPO Prospectuses, August–October 2026.
5. ISSCC 2026 / Hot Chips 2026 / arXiv, FlashAttention-4 (arXiv:2603.05451), Blackwell & Hopper Microbenchmarking (arXiv:2609.24270), Ascend 910B/950 & Long-Context Decode Roofline Analysis (arXiv:2605.13915, arXiv:2605.30571).0