When opening AI chat software, code assistants, and smart office tools, every question and answer, and every generated response, consumes a type of "AI traffic" called Token. As the smallest unit for measuring AI output, Token is gradually becoming the basic pricing unit in the AI industry, and its unit computing power cost directly determines the profit margin of AI service providers.
At the recently concluded 2026 World Artificial Intelligence Conference (WAIC), reporters observed that "reducing Token costs" is becoming a consensus among numerous chip and server manufacturers. "Cost-effectiveness" is now "on the table," becoming a new dimension of competition following computing power metrics and the scale of adapted models.
At WAIC, Zhonghao Xinying showcases its self-developed chips and intelligent computing hardware solutions
High Computing Power Costs Await a Breakthrough
As AI agents and large models are deployed at scale, Token consumption has surged, plunging many companies into the dilemma of "the more they use, the more they lose." Yang Gongyifan, founder and CEO of Zhonghao Xinying, stated in an interview that when agents call APIs, they carry and transmit the entire historical dialogue context in a package. Even if the user's prompt is only about a thousand words, the Token volume brought by the request package can reach 200,000.
At this stage, computing power expenditure has become the highest item in the total cost of AI companies, with the gross profit margin of most AI software service providers falling below 40%. "If computing power costs remain high, many AI businesses will fall into losses and their business models will fail to close the loop," Yang Gongyifan admitted. "However, if the computing power cost of output Tokens can be halved, customer profits will multiply, and the gross profit margin will increase from 40% to 70%."
From the perspective of the AI inference market, computing power workloads can be divided into two types of computing power demands with completely differentiated underlying logic, corresponding to two types of market services: Fast Tokens for high real-time interaction and Slow Tokens for offline batch processing. Different industries and application scenarios have different needs and requirements for AI services.
A domestic computing power chip company also stated in an interview with reporters from China Electronics News that building independent computing power clusters for different inference businesses will lead to long-term hardware idleness and a doubling of hardware and power investments, while also increasing the O&M costs of multi-architecture supporting software.
To help AI application providers reduce computing power costs, domestic computing power chip manufacturers have explored two implementation paths:
First, making a single chip more competitive. Redesigning the internal computing structure of the chip to reduce invalid operations and power consumption, producing more Tokens under the same power conditions; simultaneously achieving deep adaptation between the chip and large model software to avoid resource consumption caused by idle computing power.
Second, spending less on server room infrastructure. Optimizing the data transmission architecture between servers to reduce investments in high-priced supporting equipment such as switches and high-end NICs.
In addition, to control the production costs of the chips themselves, some computing power chip companies are also using methods such as forward order locking and advance stocking to smooth out cost fluctuations caused by price increases in raw materials for wafers, memory, and packaging; adopting the Chiplet solution to disassemble large chips into multiple small chiplets for separate tape-outs, in order to cope with the high difficulty and low yield rate of tape-outs for large-size single-piece wafers.
Innovating Chip Architecture
At this stage, GPUs remain the most numerous and widely used chip type in data centers. However, in the AI era, Yang Gongyifan holds a different view on the role of GPUs: "A general-purpose GPU architecture is like a universal toolbox; although it can do everything, its efficiency in processing large language model tasks based on the Transformer architecture is relatively low." In his view, after 30 years of iteration in GPU technology, it is difficult to make disruptive innovations to the underlying core architecture; at the same time, the complete CUDA software ecosystem has become the most solid moat for GPU manufacturers, making it very difficult for latecomers to overtake them on the curve.
In contrast, TPUs have broader room for innovation. "As long as the AI industry continues to expand, and the Transformer and Attention operators remain the underlying core computing units of large models, the industry value of the TPU architecture will continue to amplify, possessing greater room for performance and cost-effectiveness improvement compared to general-purpose GPUs," Yang Gongyifan stated.
In the future, inference computing power will undergo specialized separation: one type is Prefill, which reads all historical dialogues, and the other is Decode, which outputs content; the computing logic of the two is completely separated. Yang Gongyifan stated that general-purpose GPUs rely on a single set of hardware to handle both types of operations, making it impossible to optimize performance at both ends. At the chip memory level, the built-in high-speed cache capacity of general-purpose GPUs is limited, requiring frequent reading and writing of external video memory to retrieve data, which not only slows down computing speed but also generates extra power consumption. "The second-generation 'Xuyu' chip launched by Zhonghao Xinying adapts to the hardware separation of Prefill and Decode, enabling collaborative computing with one Prefill chip paired with one to two Decode chips. It is highly adapted to long-dialogue scenarios for agents and can significantly reduce power consumption caused by repeated calculations," Yang Gongyifan said.
Reducing Infrastructure Expenditures
In large AI computing power clusters, data transmission between servers relies on hardware such as switches and high-speed NICs. In a data center, the cost of data switching-related hardware accounts for about 40% of the entire system.
At the 2026 World Artificial Intelligence Conference, reporters noticed that Tsingmicro Intelligent unveiled its self-developed REX81 Supernode domestic 4K supernode solution for the first time. This solution adopts an innovative computing power grid interconnection technology, realizing direct connection between chips and servers without switches via optical fibers. Such adjustments have reduced the comprehensive cost of cluster interconnection hardware by 90% compared to the traditional switch architecture.
Tsingmicro Intelligent showcases the reconfigurable supernode system
"Hardware parameters are no longer the only competitiveness; software ecosystem, deployment scale, and customer reputation have become more important competitive dimensions," Zhang Lei, Board Secretary and CFO of Tsingmicro Intelligent, introduced in an interview with reporters from China Electronics News. "Tsingmicro Intelligent's self-developed reconfigurable dataflow architecture enables synchronous execution of data flow and computing, significantly improving hardware resource utilization; a single chip can simultaneously support inference tasks with different latency requirements, reducing the cost of repeated hardware deployment."
Authors | Ji Xiaoting, He Ziyi (Intern)
Editor | Zhang Xinyi
Art Editor | Maliya Supervisor | Lian Xiaodong