As the focus of generative AI shifts from the cloud to the edge, intelligent cockpits and Agent Boxes (AI agent boxes) have become key battlegrounds for large model deployment. However, a fundamental physical contradiction exists between the limited hardware resources of edge devices and the massive scale of large models, which often contain billions or even tens of billions of parameters. The challenge facing the entire industry is how to reduce model size while preserving logical capabilities in long-text understanding, complex reasoning, and multi-turn dialogue.
The Multi-Front Struggle Between Compute, Memory, and Precision
In edge scenarios, large model inference faces multiple physical bottlenecks.
Take intelligent cockpits as an example. The Decode phase of large model inference is a typical memory-bandwidth-bound workload. Generating each token requires reading the full set of model parameters. Currently, the memory bandwidth of mainstream cockpit platforms is around 60-70GB/s. Even with extreme compression of a 7B model, the theoretical throughput is only about 20 tokens/s. After accounting for actual overhead, this barely meets the baseline experience for voice streaming interaction, leaving no room for improvement.
Meanwhile, the Prefill phase involving multimodal long inputs is a compute-bound scenario, imposing strict requirements such as keeping time-to-first-token under 1 second. More complexly, large models must share the cockpit SoC's computing power, bandwidth, and power budget with tasks like IVI rendering, multi-screen display, and DMS. Within the automotive-grade temperature range of -40°C to 85°C and a power red line of just over ten watts, simply "stacking compute" is physically unfeasible.
In Agent Box scenarios, the workload profile changes again. Agent inference often involves dozens of turns of multi-turn dialogue, context lengths exceeding hundreds of K, and frequent tool calls. Performance bottlenecks shift to storage and I/O systems dominated by KV-Cache. This creates a mismatch between "capability and resources": large models have sufficient capability but cannot fit or run efficiently on edge devices; small models run fast but are prone to crashing during tool calls. Once faced with enterprise-level concurrent sessions, latency per token deteriorates sharply.
To address volume and memory constraints, the industry has previously widely adopted INT4 (4-bit integer) quantization. However, integer formats lack the dynamic range of floating-point numbers, leading to noticeable degradation in long-text understanding and logical reasoning after quantization. In edge scenarios with low error tolerance, such as vehicle control commands, this loss of capability becomes a direct risk to reliability.
In summary, limited computing power, restricted bandwidth, capped power consumption, and difficult precision preservation constitute four constraints that limit the scalable deployment of edge-side large models.
Why is FP4 a More Suitable Compression Solution?
To maintain model capabilities at lower bit depths, computational precision is evolving from early 16-bit floating point (FP16/BF16) to lower bit counts. Data centers have already established FP8 as the standard for inference and accelerated training, doubling computational throughput with almost no loss in model accuracy. However, in resource-constrained edge environments, the volume reduction benefits of FP8 have hit a bottleneck.
Compared to FP8, FP4 (4-bit floating point) can further reduce model size by 50%, theoretically doubling computational throughput again, making it a core technology for extreme model compression.
Moreover, compared to traditional INT4, FP4 offers significant mathematical structural advantages. As mentioned earlier, INT4 uses a linear distribution without the dynamic range of floating points, making it extremely difficult to participate directly in training tuning and preventing compensation for quantization-induced performance losses during the training phase. In contrast, FP4 typically adopts the E2M1 structure (1 sign bit, 2 exponent bits, 1 mantissa bit), achieving a non-linear distribution through the combination of exponent and mantissa bits. Its characteristic is denser granularity near zero and sparser granularity away from zero.
This numerical distribution perfectly aligns with the "normal/bell-shaped distribution" characteristics of large model data itself. Leveraging the dynamic range of floating points, FP4 better captures "outliers" within the model, thereby preserving long-context memory and logical reasoning capabilities at extremely low bit widths.
FP4 Viewed as the Core Standard for Next-Generation AI Computing
Precisely because FP4 demonstrates this mathematical potential to balance volume and precision, global chip and algorithm giants have long regarded it as the core standard for next-generation AI computing.
In the cloud data center domain, NVIDIA has made the NVFP4 format (a 4-bit floating point specifically designed for neural networks) a core selling point in its Blackwell chip architecture. NVIDIA achieves this by providing a hardware-native FP4 computation path via fifth-generation Tensor Cores and introducing fine-grained scaling techniques such as Micro-tensor scaling. This compresses weights to 4 bits while ensuring inference accuracy loss remains less than 1% compared to high-precision FP8.
Simultaneously, the MX Alliance, formed by industry giants including Advanced Micro Devices (AMD), Intel, NVIDIA, and Arm, is promoting the implementation of open low-precision data standards like MXFP4 (Microscaling 4-bit Floating Point) under the Open Compute Project (OCP) framework. This industry consensus directly influences the upper-layer algorithm ecosystem; for instance, both OpenAI and DeepSeek have applied native MXFP4 quantization to MoE (Mixture of Experts) weights, which account for over 90% of their models.
Notably, the core technology behind this standard is the concept of Block Floating Point (BFP). In traditional floating-point calculations, every value carries its own independent exponent bit. At the extreme 4-bit width, this squeezes precious mantissa space, leading to precision collapse. The BFP technology specified by the OCP MX standard innovates through "exponent sharing." It divides tensors into continuous small hardware blocks (e.g., 32 elements per block). All elements within a block share a high-precision 8-bit scale factor (typically in E8M0 format), while each element itself uses only 4 bits (E2M1) to focus on expressing high-precision mantissas and signs.
This "block-level adaptive quantization" not only brings the actual storage footprint close to 4 bits (the effective storage overhead of MXFP4 is only 4.25 bits/weight) but also provides the entire data block with a strong dynamic range due to the shared high-precision exponent. Without significantly increasing chip area, it effectively mitigates precision loss caused by "outliers" in large model activations, becoming an industry-general solution that balances hardware efficiency and model accuracy in the sub-8-bit computing era.
Edge Hardware Exploration of FP4 Engineering
With the establishment of the "Block Floating Point/FP4 dividend" in the cloud, how to bring this standard down to edge devices with power consumption of just over ten watts and memory bandwidth of only tens of GB/s—such as AI PCs, smart driving SoCs, and Agent Boxes—has become the focal point of competition among major hardware chip vendors. Large models running on the edge cannot uniformly adopt pure FP4; they must use mixed-precision computing (retaining FP16 in sensitive layers, using FP8 in less sensitive layers, and FP4 in regular layers). If chips lack specialized underlying hardware scheduling and alignment capabilities, computing utilization will drop significantly.
Currently, several representative hardware solutions have evolved in the market for implementing FP4 on the edge.
In the AI PC and high-end mobile sectors, NVIDIA and MediaTek jointly launched the RTX Spark super chip, natively integrating a 20-core Arm CPU and a Blackwell RTX GPU on the same SoC. Through unified memory and NVLink-C2C interconnect, it delivers 1 Petaflop of computing power at FP4 precision, supporting local resident deployment of models with billions of parameters on laptops and edge devices.
In edge domains like smart driving, domestic chip vendors are accelerating hardware precision iterations. For example, Axera Semiconductor released the M9 series high-level smart driving chip. Through its self-developed NPU architecture's mixed-precision computing technology, it natively supports multi-precision computations including FP4, FP8, INT8, and BF16. This improves computing utilization, frame rates, and low-latency performance with lower bit precision, providing a computing foundation for VLA, world models, and one-stage end-to-end implementations.
Different from the product form of complete chips mentioned above, Arm China, as a foundational IP supplier, attempts to provide universal tools at the source of chip design—the NPU IP level. Its recently released "Zhouyi" X3 NPU IP also achieved early native support for FP4 features on the edge.
The "Zhouyi" X3 possesses full-precision fusion computing genes ranging from INT4~32, FP4~32, to BF16, allowing flexible mixing of FP4 with FP8 and FP16 in layer-by-layer inference. In terms of hardware architecture, it adopts a DSP+DSA general-purpose architecture with single-core bandwidth reaching 256GB/s. In actual tests targeting 7B models, the Prefill computing utilization exceeds 70%. This means the "Zhouyi" X3 can truly convert the theoretical bandwidth and computing gains brought by FP4 into actual generation speed improvements and reduced time-to-first-token on the edge.
Furthermore, regarding quantization support, the "Zhouyi" X3 supports W4A16, prioritizing accuracy to adapt to high-fidelity scenarios like cockpit interactions. It also implements hardware-native support for W4A8 quantization (weight parameters quantized to FP4, activations quantized to FP8), prioritizing performance to raise the optimization ceiling for high-concurrency Agent scenarios, while simultaneously optimizing the compute-bound Prefill stage and the bandwidth-bound Decode stage.
In terms of software support, Arm China provides an end-to-end toolchain covering quantization, compilation, and deployment, along with out-of-the-box demos via the Compass AI software platform. Cloud FP4 models can be seamlessly migrated to the edge. Combined with the compatibility of the global Arm ecosystem and the rapid response of a 100% local team, this offers dual certainty of "ecosystem + service" for mass production scenarios such as intelligent vehicles and Agent Boxes.
Conclusion
From NVIDIA's NVFP4 practices in the cloud, to the Block Floating Point (BFP) standards promoted by the OCP MX Alliance, to the collective follow-up by edge hardware/IPs such as RTX Spark, Axera M9, and Arm China's "Zhouyi" X3, low-bit floating-point conversion has become an inevitable trend for improving large model efficiency.
The unique power walls and bandwidth limitations of edge devices are forcing chip architectures to provide more flexible multi-precision scheduling capabilities at the microarchitecture level. Whether through powerful hard solutions via edge super chips or mixed-precision reconstruction from the NPU IP source, the industry-wide engineering push towards FP4 as an "extreme compression format" is collectively opening the door to the scalable deployment of edge-side large models.