Intel CEO Lip-Bu Tan: Over the past few years, the story of high-performance computing has been almost entirely about GPUs and other accelerators; in recent months, however, the CPU has begun to return to the center of AI systems. He put it more bluntly: the CPU is now the orchestration layer and critical control plane for the entire AI stack, and the ratio of server CPUs to accelerators deployed by customers is also swinging back toward the CPU.
To answer two questions: (1) As GPUs become increasingly powerful, why do AI servers actually need more CPUs? (2) As this shift reaches domestic AI servers, which link is missing? The judgment is laid out here first: Agents stretch a single model inference into a workflow that continuously calls tools, retrieves data, executes code, and updates states. The GPU handles the computation, while the CPU orchestrates this workflow.
During Intel's Q1 earnings call, CFO David Zinsner provided more specific figures: training workloads typically require 7 to 8 GPUs per CPU; inference narrows this to roughly 3 or 4 to 1; and for Agents and multi-agent systems, "it might even flip slightly in the opposite direction." In the same session, he also noted that Q1 revenue could have been higher, but "demand continues to outstrip our expanding supply."
TrendForce's perspective is similar: most of today's AI data centers operate at a ratio between 1:4 and 1:8, which will eventually drop to 1:1 or 1:2. The investment narrative for AI infrastructure still revolves around GPUs, but the first component to reach its limit is the very chip that has often been treated as a supporting actor over the past two years. The way AI works has changed.
I. GPUs Are Already This Powerful, So How Did the CPU Become a Bottleneck Again?
Placing the three workloads side by side, the differences are immediately apparent.
Table 1 | CPU and GPU division of labor across three types of workloads. In training and standard inference, the CPU is on the periphery; Agents push it to the center of the workflow.
In the first two types of workloads, the CPU primarily handles data preparation and task distribution; because GPU computation times are long enough, these tasks usually remain inconspicuous.
Agent execution is a series of loops: the model determines the next step and calls a tool; the tool executes on the host side or an external service, the results return to the context, and the model continues inference. The loop runs round after round, and each tool execution may pull the computation away from the GPU, waiting for the host side or external service to return the results.
Agents turn a single inference into a repeatedly switching workflow.
Figure 1 | Agent execution loop: GPU inference and CPU-side tool execution alternate. Traditional inference passes through once, with the CPU on the periphery; when an Agent executes a tool, the GPU waits for the host side or external service to return the results.
Dylan Patel, Chief Analyst at SemiAnalysis, described this shift bluntly in an interview this April: "In the early years of AI, the CPU was indeed severely lagging; you send a string, it returns a string, there was little demand on the CPU." But now, the model outputs content, checks the environment, and continues execution based on feedback, "this loop is getting tighter and tighter."
The reality he notes is that revenue from coding Agents surged from billions of dollars to over $10 billion in a very short time, and the duration of a single task has stretched from tens of seconds to continuous six to eight hours.
As tasks lengthen, tool calls and state switches continuously pile up on the CPU side. What Agents add is not just the number of Tokens, but control flows, toolchains, and system calls. GPUs can expand Token throughput by adding more cards, but control flows will not automatically speed up just by plugging in a few more cards.
II. GPUs Haven't Slowed Down; They Are Starved for Data
CPUs can drag down GPUs, usually getting stuck at tool calls, isolated environments, and scheduling/transfer tasks.
Tool Calls: The Agent's Hands Are on the CPU
For an Agent to work, it must call external tools: Python interpreters, search engines, database queries, web scraping and parsing, unit testing, and compilation. Most of these tasks still run on the CPU side. They are branch-intensive, I/O-intensive, and latency-sensitive, which are exactly the tasks the CPU is better at. (Vector search, some database operators, and numerical computations can be accelerated by GPUs, but in today's mainstream Agent frameworks, most of them still reside on the host side.)
A joint study by Georgia Tech and Intel put five typical Agent workloads on the test bench (data taken from the April 2026 revised paper), and the results were somewhat unexpected:
In the end-to-end latency of RAG (Haystack), vector search alone accounts for 81%–89%, almost determining the overall speed; the Bash execution of SWE-Agent accounts for 25% and 65% respectively under two configurations, with the CPU-side burden varying drastically with the environment; whereas Toolformer is the opposite, with LLM inference consuming 77%–88% of the latency, leaving the CPU relatively idle. ChemCrow and Web-Augmented Agents fall somewhere in between (see Figure 2 for the full distribution).
The degree of CPU dependency varies greatly among different Agents and cannot be generalized, but no category can completely bypass the CPU.
Even though they are all called Agents, the CPU-side latency proportion can range from 25% to 89%.
Figure 2 | CPU-side tool latency proportion for four types of tool-intensive Agent workloads. The ranges represent the measured values under different hardware configurations (Sys 1 / Sys 2) in the paper.
Note 1: Toolformer is excluded from the horizontal comparison in this figure because LLM inference dominates (77%–88%) and the CPU side is extremely light, which is the opposite of the above four categories; see the main text for details.
Note 2: Early abstracts of the paper once gave an upper limit of 90.6%, which Intel's tech blog summarized as 50%–90%; this article uniformly adopts the main text caliber of the v3 revision in April 2026 (approximately 89%). The above data only correspond to tool-intensive workloads and do not represent all Agent paradigms.
Sandbox Isolation: Each Concurrent Agent Consumes a Share of CPU Threads and Memory
Coding Agents are not suitable for sharing a single execution environment. They need to write files, install dependencies, run tests, and execute model-generated code; once shared, the side effects of one Agent might pollute the state of another, making it difficult to trace. The common solution is to isolate the execution environments: containers, micro-VMs, or independent processes.
Inference scales with the number of Tokens, while sandboxes scale with the number of concurrent Agents. For each additional concurrent Agent, an independent set of resource namespaces, process states, and working sets must be maintained. Image layers and read-only pages can be shared, and copy-on-write can save some resources; but the independent parts still press on the CPU's core count, thread count, and host memory, where the GPU cannot help.
Thus, a counterintuitive resource accounting emerges: the GPU utilization rate hasn't maxed out, yet the machine can no longer spin up new sandboxes. The paper quantifies this very intuitively: after increasing the batch size from 64 to 128, the average latency of the CPU-side summarization step in the Web-Augmented Agent stretched to 2.0 times (Sys 1) and 1.9 times (Sys 2), while the average latency of GPU-side LLM inference in the same range barely moved.
Table 2 | Average latency increase of each component when the batch size doubles under the SWE-Agent workload. The four tiers increase monotonically from GPU to CPU: when concurrency doubles, it is the CPU segment that grows.
The paper's attribution for Web-Augmented Agent, SWE-Agent, and ChemCrow is consistent: throughput saturates at a batch size of 128 due to core over-subscription by CPU-heavy tools. The authors' conclusion is more direct: the efficiency of CPU parallel strategies is lower than that of GPUs, which hits the throughput ceiling prematurely in Agent workloads, thereby dragging down the utilization rate of expensive GPU resources.
Scheduling and Data Movement: GPUs Start Waiting for Others
Besides tools and sandboxes, the CPU is also responsible for request orchestration, data transfer between sub-Agents, state management, and CPU-GPU synchronization. Individually, none of these are heavy, but together they crowd the critical path. The next batch of data and instructions for the GPU must wait for the CPU to be ready.
There is a term in manufacturing called "waiting for materials while idle": the machine isn't broken, the capacity is there, but the materials just don't arrive. In tool-intensive Agents, GPUs can fall into this exact state. Blackwell is much faster than Hopper, and Rubin will be even faster, but the GPU will repeatedly wait for the CPU or external services to return results. Intel calls this loss the GPU Idle Tax. You pay full price for the computing power but only use a fraction of it.
The key point is that what the CPU gets stuck on is serial control, external interaction, and isolated execution, while matrix operations are still primarily handed over to the GPU. Agents pile a batch of tasks that GPUs are not good at handling onto the CPU side, leaving the GPU stuck in waiting, thus generating the idle tax. At this point, what is truly tight is no longer floating-point capability, but core count, memory bandwidth, and low-latency response.
Obviously, the bottleneck has shifted from how fast the computation is to how timely the data is fed.
III. CPU:GPU from 1:8 to 1:2: Don't Fixate on This Number
Traditional 8-GPU training servers are typically equipped with 1 to 2 CPUs, corresponding to a 1:8 or 1:4 ratio. The CPU primarily handles host-side tasks such as data preparation, task distribution, and checkpointing. In many 8-GPU systems, one CPU is already sufficient, and adding more CPUs does not necessarily yield proportional returns.
Moving to inference, especially in Agent scenarios, tool calls shift from occasional to running several times per request, and the number of sandboxes increases with the number of concurrent Agents; meanwhile, the performance of a single GPU is significantly faster than a single CPU core. To prevent GPUs from waiting for data, the system has to configure CPUs more densely.
Product configurations provide a set of corroborating evidence. The NVIDIA GB200 NVL72 is configured with 36 Grace CPUs and 72 Blackwell GPUs in a single rack, making the CPU:GPU ratio exactly 1:2. The next-generation Vera Rubin NVL72 maintains the same ratio: 36 Vera CPUs paired with 72 Rubin GPUs; the miniaturized NVL4 has 2 Vera CPUs paired with 4 Rubin GPUs, also 1:2. A single Vera has 88 cores and 176 threads, connected to GPUs via 1.8TB/s NVLink-C2C; NVIDIA positions Vera as a CPU for data movement and Agent inference.
The host CPU has also once again become a position platform vendors are fighting for: according to Intel's Q1 disclosure, Xeon 6 was selected as the host CPU for the NVIDIA DGX Rubin NVL8 system. During the same period, Arm broke its 35-year tradition of only selling IP and began selling finished CPUs directly; NVIDIA also sells Vera as an independent product because customers need more flexible CPU-to-GPU ratios.
Three vendors, three moves, pointing to the same thing: the CPU is no longer just an accessory to the GPU, but a commodity that can be sold independently and is worth fighting for on its own.
Ratio Narrowing: From 1 CPU Feeding 8 Cards to 1 CPU Feeding 2 Cards
Figure 3 | CPU:GPU configuration comparison across three generations of platforms. The diagram shows the relative ratio normalized to 1 CPU, not the actual physical layout in the rack; traditional 8-GPU servers are counted as 1:8 with 1 CPU, and 1:4 with 2 CPUs.
Intel researchers provided a more aggressive range based on their own tests: in some Agent scenarios, the optimal CPU:GPU is about 0.8:1 to 1.4:1, close to one CPU per GPU; under certain workloads, the number of CPUs even exceeds that of GPUs. But the same blog also warns: the optimal ratio highly depends on the workload.
The 1:2 ratio of GB200 and Vera Rubin indicates that mainstream platforms are increasing CPU density; this does not equal a standard answer applicable to all AI servers. With different model structures, tool complexity, concurrency, sandbox granularity, and scheduling strategies, the optimal ratio will also differ.
So, will the ratio continue to narrow?
It mainly depends on two things: whether the proportion of Agents in inference will continue to rise, and whether tool calls and sandbox execution will be taken over by DPUs or dedicated hardware. The relative improvement speed of single-core CPU versus GPU performance will determine how fast and how far this curve will go.
However, software might rewrite this curve earlier than hardware. The two scheduling optimizations proposed in the aforementioned paper reduced service latency by up to 3.9 times without changing a single chip. This shows that a significant portion of "CPU shortage" is actually just "CPUs not being used well." If scheduling layer optimization runs ahead of hardware expansion, the market's incremental demand for CPUs could be significantly flattened. (See the caliber explanation at the end of the article for the full multiples of COMB and MAS.)
IV. Adding Two More CPUs Won't Help; Servers Need to Be Redesigned from Scratch
If the ratio truly moves from 1:8 to 1:2, at least four areas must be recalculated in engineering: CPU sockets, memory, interconnects, and power supply and cooling.
Cores, Threads, and Memory. Sandbox concurrency directly consumes cores, with the core count of a single CPU moving from dozens to over a hundred; in systems with host memory offloading enabled, the KV cache for long tasks will also overflow into the large memory on the CPU side. Once the core count rises, memory capacity and channel count must keep pace synchronously, otherwise the added cores cannot be fed.
CPU-GPU Interconnects and Data Paths. Synchronization between CPUs and GPUs becomes more frequent, turning link bandwidth and latency from "good enough" into system bottlenecks. The differences between PCIe generations, CXL memory semantics, and tightly coupled solutions like NVLink-C2C are not obvious under training workloads but will be amplified under Agent workloads.
Sandboxes, Virtualization, and Scheduling Software. The number of isolated instances a single machine needs to support increases by an order of magnitude, and virtualization overhead must be optimized; the 3.9 times mentioned earlier comes from this layer. Scheduling software is now part of system design and can no longer be treated as an afterthought post-deployment.
Power Supply, Cooling, and Chassis Structure. The power consumption proportion of CPUs and memory rises, requiring recalculation of power distribution in the rack, liquid cooling loops, and slot planning.
The change is not just in CPU procurement volume. AI servers are transforming from accelerator systems centered around GPUs with other components revolving around them, into heterogeneous computing systems co-designed with CPUs, GPUs, memory, and networking.
The cost structure will also be rearranged accordingly. The increased density of CPU configurations will synchronously amplify the demand for memory, interconnect components, and power modules, thus increasing the proportion of non-GPU parts in the overall BOM; TrendForce observed that Intel and AMD raised prices for some CPU product lines at the end of Q1 2026. The supply tightness is driven by both the demand changes brought by Agents and the impact of capacity ramp-up. Intel has explicitly stated it is expanding output, with Xeon 6 on the Intel 3 process entering full production ramp-up and 18A yields better than internal expectations; even so, it is still expected that demand will outstrip supply through this year and next.
As competition shifts to system balance, domestic manufacturers must face whether cores, interconnects, and software stacks can coordinate; single-point CPU performance is just one aspect.
V. The Real Lesson Chinese Vendors Need to Learn Is Not Inside the Chip
Ratio narrowing will shift the pressure from whether the cards are fast enough to whether CPUs, memory, interconnects, and scheduling can keep up. Domestic server CPUs still need to pass three layers of technical validation, with the latter two being harder; beyond that, it also depends on whether the products can truly enter the selection scope of system integrators.
First, look at core scale and memory channels. According to the public specifications of the Kunpeng community, Huawei's Kunpeng 920 is based on the 7nm process, with up to 64 cores, integrating an 8-channel DDR4 memory controller, and supporting PCIe 4.0 and CCIX; Alibaba's T-Head Yitian 710 adopts the 5nm process and 2.5D dual-die packaging, with each die having 64 cores, totaling 128 Armv9 cores, supporting DDR5 and PCIe 5.0; Phytium's Tengyun S5000C offers three configurations of 64/32/16 cores, targeting computing, storage, and AI server scenarios. Some products already meet the basic conditions for core scale and memory channels; having sufficient parameters on paper does not mean they can run real Agent workloads well.
At this layer, one must also ask: are the parameters sufficient but unavailable to system integrators? For example, the Yitian 710 is not sold externally and is only for Alibaba Cloud's internal use.
Next, look at the tightly coupled interconnect between CPUs and accelerators. In public information, this layer is the most worth watching. Overseas platforms are integrating CPUs and GPUs into a more tightly coupled shared memory system through coherent C2C interconnects: between Vera and Rubin is 1.8TB/s NVLink-C2C, making CPU-side memory nearly directly addressable by the GPU. From public product specifications, domestic combinations currently still rely mainly on standard PCIe, with CPUs and accelerator cards being loosely coupled; this gap is diluted by long operators under training workloads, but the high-frequency, small-batch synchronization in Agent workloads relies heavily on this exact segment.
Domestic vendors are also beginning to layout at this layer. According to Hygon Information Technology's 2025 Annual Report, the company released the HSL (Hygon System Link) system bus interconnect protocol in September 2025. The annual report defines it as an "open, high-bandwidth, low-latency interconnect bus protocol specification, primarily used to connect Hygon CPUs with various xPUs," aiming to achieve tightly coupled high-speed interconnection with xPUs, IO, operating systems, and OEM vendors; core content includes opening the complete bus protocol, providing IP reference designs, opening the instruction set, and supporting cache coherence and multi-link scaling.
Cache coherence is especially critical. The presence or absence of cache coherence is the most tangible difference between tightly coupled interconnects and standard PCIe attachment: it determines whether the CPU and accelerator card transfer data back and forth or share a single pool of data. Those high-frequency, small-batch, back-and-forth synchronization actions in Agent workloads feed exactly on this difference.
Hygon wrote its future goal in the annual report as moving from adaptation and compatibility to native collaboration. Releasing the protocol is just the first step; what follows is to see how many accelerator card, system, and operating system vendors truly integrate, as well as the measured bandwidth and latency after integration.
Finally, look at sandbox density and the scheduling software stack. Public data is the scarcest, and it relies most on long-term accumulation. The per-watt efficiency of high-concurrency isolated environments, virtualization overhead, and the synergy between the scheduler and CPU microarchitecture are not problems that can be solved just when a chip rolls off the line; they require long-term polishing with real workloads; experiments with COMB and MAS also show that software scheduling itself can bring significant benefits. Due to the lack of comparable public measurements, it is currently impossible to judge the capability gap of domestic products at this layer; what can be confirmed is that workload data and long-term running samples are still insufficient.
Table 3 | Three-layer capability validation for domestic server CPUs in the Agent era. Core counts and channels are based on public product specifications from each vendor; financial data is based on the caliber of periodic reports from listed companies; the third layer lacks comparable public measurement data, so no quantitative evaluation is made.
Among the three layers, the first looks at specifications, and the third looks at long-term operation. In public information, the interconnect is currently the most worth watching layer. Most domestic AI servers are still loosely coupled combinations of accelerator cards and general-purpose CPUs; Agent workloads precisely amplify that link between the CPU and the accelerator. Whether the protocol can form an ecosystem depends on how many accelerator cards, systems, and operating systems truly integrate.
Let's set a ruler first. To cross the interconnect layer, the CPU and accelerator must be made into a hardware cache-coherent shared memory, and it must be mass-produced for external supply. By this caliber, only NVIDIA in the world has completed the entire chain: NVLink-C2C brings Vera and Rubin into the same memory domain.
Applying this ruler domestically, the positioning of each player's approach becomes clear.
Only the first tier truly sticks to the interconnect layer, and all three are closed loops for internal use; the protocols serve their own chips and their own clouds, and not a single one supplies the interconnect as an open standard externally.
The second tier has all three pieces: CPU, accelerator card, and system, but there are no public mass-production cases of stringing them together with a coherence protocol. The third tier does not have CPUs in hand; whether they can plug into others' protocols directly determines whether the ecosystem succeeds.
Players in the software layer further down do optimizations above the protocol; the value is real, but the ceiling is determined by the tiers above. If the underlying layer is not connected, no matter how much the upper layer squeezes, it cannot squeeze out a shared pool of data.
Therefore, the domestic problem is not that no one is doing it, but that everyone forms their own closed loop, and a public layer that can mutually integrate has not yet grown.
Table 4 | Interconnect Layer: Where Chinese Vendors Stand. The classification dimension is the distance of each vendor from tightly coupled cache coherence, not a ranking of comprehensive strength.
Conclusion: The Next Generation of AI Servers Competes on System Balance
This round of changes shows that AI infrastructure is moving from stacking components at single points to systems engineering. Over the past two years, the competition was about who has more GPUs; going forward, it should be more about whose GPUs are actually computing.
Going forward, two types of numbers are more worth watching: the idle time of GPUs caused by waiting for CPUs, and the CPU:GPU ratio under different workloads. The former directly shows the bottleneck, while the latter will land on purchase orders and rack designs. The CPU-side latency proportion is a metric often looked at when dissecting the former.
For the domestic market, there is one more thing to watch closely: after the release of tightly coupled interconnect protocols, whether the second and third accelerator card and system vendors can truly plug in. That would indicate that the interconnect capability in the middle has been made up, and there is connection capability beyond just core counts and systems.
GPUs still determine the upper limit of computing power for AI systems, while CPUs, memory, interconnects, and scheduling increasingly determine whether this computing power can actually be unleashed. Rather than watching whose GPU scores higher in benchmarks, I am more interested in seeing how many domestic vendors have truly run through the coherent interconnect protocol in half a year.
References and Caliber Explanations:
The data in this article is current as of early August 2026. The article deliberately distinguishes among four types of evidence: independent research, vendor self-tests and disclosures, third-party institution forecasts, and announcements from listed companies: product specifications and performance metrics only represent the vendors' public calibers and do not form unified cross-platform conclusions; ratio numbers come from different calibers and measurement methods and cannot be directly added or averaged; the paper's measured data relies on specific hardware configurations and workloads and does not constitute universal conclusions. The domestic section only adopts public product specifications from vendors and the caliber of periodic reports from listed companies.
References:
[1] Intel | 1Q2026 Earnings Call — CEO/CFO Prepared Remarks (Lip-Bu Tan)
[2] Intel | INTC Q1 2026 Earnings Call Transcript — Q&A (David Zinsner)
[3] Intel / SEC | Form 10-Q for the quarterly period ended March 28, 2026
[4] Georgia Tech & Intel | Towards Understanding, Analyzing, and Optimizing Agentic AI Execution: A CPU-Centric Perspective, arXiv:2511.00739 (v1 2025-11-01; this article adopts v3, 2026-04-16)
[5] Intel | Avoid the GPU Idle Tax: Choosing the Right CPU to GPU Ratios for Agentic AI
[6] TrendForce | The Great Rebalance: How Agentic AI Is Reshaping the CPU:GPU Ratio
[7] NVIDIA | GB200 NVL72
[8] NVIDIA | Vera Rubin NVL72
[9] SemiAnalysis | Dylan Patel Interview (Paid Podcast)
[10] Cast AI | 2026 State of Kubernetes Optimization Report
[11] Huawei Kunpeng Community | Kunpeng 920 Processor
[12] Alibaba T-Head | Yitian 710 Release Information
[13] Phytium | Tengyun S5000C High-Performance Server CPU
[14] Hygon Information Technology (688041.SH) | 2025 Annual Report
#ShenXinMeng #WanXinExpo #SemiconductorIndustryResearch #Agent #GPU #CPU