Large model technology is transforming the computing ecosystem at the edge and end devices. As small and medium-sized models with around 30B parameters demonstrate capabilities approaching those of flagship models, the focus of AI inference is shifting from the cloud to local environments.
"Past cloud-based AI relied on networks for single-turn Q&A, where the latency of data upload and download was uncontrollable, and context could not be maintained. Future Agentic AI will build a closed loop of continuous perception, memory, reasoning, and action locally. This paradigm shift requires the underlying core chip, the NPU, to transition from a one-time inference accelerator to the computing foundation for AI agents," said Dr. Shu Hao, Head of NPU Product Line and Chief Architect at Arm China, at the "Arm China AI Forward-Looking Conference" held during WAIC 2026.
The Conflict Between Fragmented Scenarios and Diversified Workloads
Dr. Shu pointed out that deploying agents to edge devices faces stringent physical constraints. First is the high diversification of workloads. Data shows that in the token consumption of large model inference, code programming accounts for 52%, role-playing for 20%, and engineering technology for 10%. Different applications have varying performance requirements for chips; code programming heavily relies on prompt length and the storage capacity of the KV Cache, while role-playing is extremely sensitive to interaction latency.
Figure | Huge differences in requirements for edge and end-device AI inference scenarios; Source: Arm China
Another constraint lies in the environmental limitations of the edge market, which are defined by five dimensions: power consumption boundaries, real-time performance, software reuse, data formats, and reliability. Mobile phones and PCs must primarily balance battery life and local privacy; robotics and autonomous driving require low latency and functional safety for multi-modal sensing; industrial edge devices emphasize task determinism and long lifecycles.
Attempting to adapt a single hardware architecture to all scenarios would result in a dual waste of wafer area and power consumption. Consequently, the market's evaluation criteria for chips are shifting towards system efficiency, scenario adaptability, and overall deployment capability.
Layered Architecture with Software-Hardware Decoupling
To resolve the contradiction between software adaptation costs and hardware efficiency, a layered and decoupled chip architecture design has become an inevitable trend.
Dr. Shu believes that at the software and software-hardware adaptation layer, by defining a unified compiler toolchain, a unified Instruction Set Architecture (ISA), and a unified hardware interface, developers can fully reuse their software work for model quantization, memory scheduling, and deployment tuning, whether facing milliwatt-level wearable devices or hundred-watt-level high-compute edge nodes. This directly breaks the pattern of software fragmentation at the edge.
At the hardware layer, differentiated hardware abstractions can be introduced under the unified ISA to interface with vastly different underlying hardware instances. A dedicated architecture is adopted for wearable devices to improve computing efficiency; a general-purpose architecture is used for mobile phones and PCs to retain programming flexibility; and even Compute-in-Memory (CIM) and non-von Neumann architectures are introduced to break the memory wall and KV Cache bandwidth limitations at the underlying level.
Figure | X3-Pro: Same Architecture, Multiple Configurations; Source: Photographed by eefocus
Through this layered design, combined with Arm China's "Zhouyi" NPU product roadmap, multiple hardware configurations such as the X3-Pro Lite, X3-Pro Premium, and X3-Pro Subsys subsystem can be derived in the future, precisely covering diverse scenarios ranging from low-power IoT to high-concurrency intelligent cockpits.
Conclusion
From the perspective of the underlying logic of hardware evolution, the focus of chip optimization has shifted from single convolution acceleration to mixed-precision computing and memory hierarchy optimization. To cope with the frequent computation of non-linear activation functions in large models, hardware has added dedicated operator acceleration units and custom transcendental function computation units.
Meanwhile, coupled with a configurable memory hierarchy architecture and flexible Cluster configurations, the chip can dynamically allocate SRAM and bandwidth resources when processing the dynamic Prefill and Decode stages, thereby mitigating the systemic latency caused by multi-round iterations of AI agents.
For the deployment of large models on physical devices, the core of competition lies in energy efficiency, the maturity of software toolchains, and the hardware's throughput capability for long contexts. Unified software architecture and hardware tailored to specific scenarios are gradually becoming the technical path for edge and end-device chips to break through.