The scale of a city determines the kind of night watch it requires.
In a small town at night, a night watchman walks through the streets and alleys with a lantern, handling whatever he encounters, relying on his stamina and experience. When this town grows into a metropolis, with streets expanding from a few to thousands and the population surging from hundreds to millions, even if the watchman runs his legs off, he cannot patrol even a fraction of it. He needs a map, streetlights, and preferably, someone to help him handle the troubles before they arise.
Computing clusters are undergoing such expansion. And the night watchmen of this city—the O&M teams—are facing unprecedented pressure.
Three years ago, 10,000-card AI training clusters were still the industry ceiling; today, 10,000 cards are no longer news. A trillion-parameter large model requires tens of thousands of accelerator cards. As model parameters continue to scale up, computing power demands enter the 100,000-card level, and the number of optical modules breaks through one million, equivalent to the total number of streetlights in a medium-sized city. Every single one must be controllable in real time, with second-level fault response.
The old way of night watch has reached its end, and a new approach is knocking at the door. In September this year, Huawei Connect released the industry's first computing O&M intelligent agent, Suan Xiaowei, and the cluster digital map was simultaneously upgraded to 2.0.
This combination of moves aims to answer an increasingly urgent question: How exactly should the night watch be conducted in a 10,000-card city?
In early 2024, China's daily token consumption was about 100 billion. By 2026, this figure reaches 140 trillion. In two years, a thousand-fold increase.
Data from the China Academy of Information and Communications Technology (CAICT) shows that by the end of June 2026, China's intelligent computing power scale has reached 2,185 EFLOPS. Meanwhile, over 13.73 million standard racks have been put into operation, and 42 10,000-card-level intelligent computing clusters have been built.
Looking only at this curve, you might think this is a story about "more": more cards, more racks, more power. But as soon as you pull the camera from the curve back to the computer room, the direct challenge brought by scale expansion is the exponential increase in O&M complexity.
It is not enough just to get computing power running; the real challenges are computing power utilization and reliability. The completion and operation of a cluster is only the starting point of computing power value. Whether it can output continuously, stably, and efficiently is the key to determining the return on this investment.
However, as the cluster scale advances from 1,000 cards to 10,000 cards, the difficulty of O&M also doubles exponentially. A 1,024-card super node will generate over 80,000 digital objects, corresponding to tens of millions of monitoring metrics. No matter how diligently the night watchman shines his lantern, he cannot illuminate every corner of this city.
Moreover, some "fires" cannot be seen with the naked eye. Hidden issues such as dirty or loose optical modules and component aging do not trigger direct alarms, but may suddenly break out when a training task reaches a critical node.
At the same time, the business side's reliability requirements for computing infrastructure have reached unprecedented heights. In financial inference scenarios, the service reliability of a 1,000-card cluster needs to reach 99.95%, meaning the total controllable business interruption time throughout the year cannot exceed 4.4 hours; in 10,000-card training scenarios, every day the cluster is down results in the evaporation of CNY millions to tens of millions in computing costs. If the night watchman takes a nap, the price could be the shutdown of the entire city.
Endless alarms, invisible hidden dangers, and unaffordable fault downtime... As the industry has developed to this stage, the era of relying on humans for night patrols has clearly turned the page. What should the new night watchman look like?
What Kind of Night Watchman Does a 10,000-Card City Need?
We might get a glimpse of the answer from Huawei CCAE's latest achievement, the computing O&M intelligent agent Suan Xiaowei.
To understand what an intelligent agent means for O&M, it is helpful to first review the evolution of computing O&M.
The earliest O&M was in the manual era, where engineers logged into servers and troubleshot command by command. Subsequently, it entered the automation era, with monitoring, alarming, and batch script execution. The toolchain continued to improve, but decision-making still relied on humans. The emergence of intelligent agents is precisely to liberate manual inspections.
Simply put, an intelligent agent is an intelligent O&M engineer equipped with perception, decision-making, and execution capabilities. Huawei CCAE's computing O&M intelligent agent Suan Xiaowei has two core capabilities. The first is fault diagnosis, which has accumulated over 1,400 fault patterns, relying on continuous accumulation and iteration by Huawei CCAE in massive global O&M scenarios. The second is intelligent energy saving. The system can perceive the operating status of the whole machine equipment, business load tides, and idle periods in real time. Through strategies such as dynamic frequency adjustment, intelligent load scheduling, and refined hardware frequency reduction, it achieves a 10% reduction in cluster idle energy consumption without affecting the performance of AI training and inference businesses.
Taking fault analysis as an example, on the fault leaderboard of computing clusters, dirty optical links have long occupied the top spot. In a 10,000-card cluster, a flash break in an optical link can trigger a chain reaction: the same link reports 3 LINK DOWNs within 3 seconds, plus 30 derived alarms such as IPv6 link state changes. In the traditional mode, O&M personnel need to check alarms, performance, and connections one by one, switching back and forth between 6 pages. It takes about 1 hour to determine that it is caused by abnormal optical module receiving power and link degradation, and the accuracy rate is only about 70%.
After the intelligent agent Suan Xiaowei takes over, this process is compressed to the minute level, and the accuracy rate is also significantly improved. It no longer requires humans to "find a needle in a haystack" in the ocean of alarms, but directly provides root cause judgment and repair suggestions.
Behind this capability is Huawei CCAE's long-term accumulation in data and experience. Specifically, the computing O&M intelligent agent first released by Huawei CCAE synchronously models new devices of various generations, reversely summarizes rules from over 7,000 real fault cases on the live network, and forms professional know-how for fault classification and diagnosis Skills. Different skills are divided by category and cover scenarios by category, with a coverage rate reaching over 95%.
From humans to tools, and then from tools to intelligent agents... The night watchmen of the 10,000-card city have transformed from a group of red-eyed engineers into an intelligent system that can perceive, think, and act. Technology has liberated humans to the greatest extent, but can patrolling intelligent agents alone really escort the 10,000-card cluster?
The reason why 10,000-card clusters are difficult to maintain lies not only in the large quantity but also in the complex links, fast fault propagation, and deeply hidden root causes.
Inside a 10,000-card-scale cluster, tens of thousands of high-speed links, hundreds of thousands of optical modules, and tens of thousands of switching nodes are intertwined. A slight degradation in the receiving optical power of a single optical module can trigger chain reactions such as link flash breaks and RDMA retransmissions. The night watchman needs a "base map" that is sufficiently detailed, real-time, and interactive for support.
The cluster digital map 2.0, simultaneously upgraded by Huawei CCAE, is designed to solve this very problem.
Its essence is to map the physical cluster one-to-one into logical objects within the O&M system. From large-scale cluster objects of tens of thousands of cards to the UDIE port of a single NPU, all have completed digital modeling.
At the same time, all connections in the cluster have also been modeled. Computing units such as NPUs and CPUs are interconnected through the UnifiedBus. The interconnection includes electrical interconnection within a POD and optical interconnection between PODs. This means that the real-time status of every chip will be monitored, and any abnormal component or link can be quickly located on the digital map.
This is like drawing a real-time updated patrol map for a city. In the past, O&M personnel faced rows of cabinets and cables, and could only rely on experience and luck to judge where the problem was. Now, every streetlight has a coordinate, and every line has a label. O&M personnel have a global, real-time, and interactive field of vision.
Looking further, a cooperative relationship of eyes and brain has formed between the digital map and the intelligent agent Suan Xiaowei: the digital map is responsible for "seeing," translating the complex state of the physical world into computable and inferable structured information in the digital world; the intelligent agent Suan Xiaowei is responsible for judgment, performing root cause analysis, making decisions, and executing on this map.
Without a map, the intelligent agent is like a blind man feeling an elephant; without an intelligent agent, the map is just a beautiful static blueprint. The combination of the two truly transforms the night watch in the 10,000-card city from a human sea tactic to systematic intelligent O&M.
However, who will use this map? Will it illuminate a small computing power town or the computing power cities of thousands of households?
The security of a city cannot rely solely on one night watchman.
The significance of Huawei CCAE's computing O&M intelligent agent Suan Xiaowei lies not only in handing in a technical answer but also in drawing an industrial dividing line. Before this, intelligent O&M was a competition of single-point tools. After the release of Suan Xiaowei, the competition will move towards systematic capabilities and ecological patterns. And the more clusters it serves and the richer the faults it sees, the smarter the intelligent agent becomes.
In other words, O&M is like an amplifier for computing power value. With the same cards and the same power, if the O&M level is a notch lower, the output may fall far behind. The losses brought by a single unplanned downtime may far exceed a year's O&M investment; while a successful prediction of hidden dangers may save precious time for an entire batch of training tasks.
From chips to clusters, from hardware to software, full-stack O&M experience, coupled with an annual R&D investment of over CNY 100 million in the cluster autonomous intelligent engine, has allowed Huawei CCAE to build a moat that is difficult to replicate in the short term.
Verified through actual combat in massive complex scenarios worldwide, Huawei CCAE currently serves over 350 AI clusters globally, with the availability of 10,000-card-level AI training clusters reaching 98.5%. Increasing availability from 98% to 98.5% may seem like just 0.5 percentage points, but for a cluster with an annual output value of hundreds of millions of CNY, it means millions of CNY in incremental revenue.
However, Huawei CCAE has not chosen to keep these capabilities in its own hands. At the Computing Management Software sub-forum titled "Cluster O&M Intelligent Agent: Ushering in a New Era of Intelligent O&M," Lu Chi, President of Huawei's Computing Management Software Product Line, proposed that they will gradually participate in industry standard construction next year. Intelligent O&M is still in its early stages, and the industry lacks unified technical specifications and evaluation systems. The establishment of standards means the industry will move from fighting alone to collaborative evolution, ultimately benefiting the entire computing power ecosystem.
Looking back, the way of night watch has been changing. The night in a small town relies on the stamina and experience of a single night watchman; the night in a metropolis relies on maps, streetlights, and an automatically operating patrol system. The night watchmen of the 10,000-card city are transforming from humans into a group of intelligent agents, relying on a complete set of night watch systems that can be replicated, shared, and co-built.
After the upgrade of Huawei CCAE, the digital map illuminates the city, the intelligent agent patrols the night on behalf of humans, the skill library passes down the experience, and the open ecosystem allows more cities to light up along the way.
One lamp lights up one street, and ten thousand lamps light up a city. Only when all cities are lit up can we truly step into the era of intelligent O&M.