Author | Wei Rong, Editor | Bao Yonggang
As AI (Artificial Intelligence) chips evolve rapidly, previous-generation GPUs have not depreciated as quickly as traditional hardware.
"The batch of H100 chips we deployed in 2023 now commands an even higher rental price than when they were initially launched," Stephen Balaban, co-founder and CTO of cloud service provider Lambda, stated recently on the podcast The MAD Podcast.
This is not a "legendary story" of a specific chip generation. In the past, people expected GPU computing power to eventually become standardized and commoditized like traditional cloud computing resources, as supply increases and hardware iterates: higher performance, lower prices, and diminishing differences among various suppliers.
However, AI computing power has not entirely followed this path. The value a single GPU can provide increasingly depends on the system it is placed in, how it collaborates with other GPUs, and whether these expensive devices can be continuously utilized. Chip performance remains important, but it can hardly explain alone how much computing power an AI infrastructure can deliver.
Then, why has computing power not yet become a truly standardized commodity like electricity or bandwidth, given the increasing supply of GPUs and faster product iterations?
01. Computing Power Is Not a Simple Addition; the Challenge of Super Nodes Lies in Collaboration
What frontier large models are changing is not how computing power is priced, but at what level computing power is commoditized. At the top level, AI services are gradually standardizing model outputs into tokens and billing by the token.
In this sense, the computing results ultimately purchased by customers already possess increasingly obvious commodity attributes. However, behind these tokens, the computing systems that produce them are far from achieving the same level of standardization.
One of the reasons is that frontier AI tasks can hardly be completed independently by a single GPU. Model parameters, computing tasks, and data need to be split across multiple GPUs or even multiple servers. Whether a single training or inference run can proceed smoothly no longer depends solely on how fast a single chip can compute, but also on whether these GPUs can exchange data quickly.
This creates a problem for AI computing power that differs from traditional standardized computing resources: the same quantity and model of GPUs do not necessarily deliver the same computing capacity.
If the communication efficiency between GPUs is insufficient, the chips may spend more time waiting for data; differences in network topology, memory, storage, and software organization will also affect the computing power that can ultimately be unleashed. Therefore, adding a GPU does not mean the system will gain the additional output corresponding to the nominal performance of one GPU.
It is precisely against this backdrop that super nodes have begun to emerge as a crucial form of AI Infrastructure (AI Infra). A super node is not simply about increasing the number of GPUs, but rather organizing multiple GPUs into a tightly coordinated computing unit through high-speed interconnects, minimizing the cost of data exchange between GPUs, and making them work like a single, larger machine.
Balaban cited NVIDIA's GB300 NVL72 as an example, where 72 GPUs within a single rack are interconnected via high-speed NVLink; when scaling across multiple racks, they are connected via InfiniBand or high-speed Ethernet. The large-scale GPU clusters built by Lambda also adopt architectures that minimize network congestion, ensuring sufficient communication bandwidth between different GPUs.
This systemic capability primarily serves large-scale training, but frontier inference is also becoming increasingly reliant on multi-GPU collaboration. Balaban mentioned that small, compressed models can fit into a single GPU, whereas larger models may need to run across multiple GPUs, servers, or even racks.
Therefore, while the GPU model and quantity still determine how much computing resource a system possesses, it is increasingly difficult for them alone to represent how much computing power these resources can ultimately deliver.
This is also the key reason why AI computing systems are difficult to quickly turn into standard commodities: while the chips themselves can be standardized, how the chips are organized and how much performance they can ultimately deliver still highly depend on the entire system.
02. Three Clocks of a GPU: Generational Iteration, Depreciation, and Economic Lifespan
If an AI computing system is not a standard commodity, then "how much a GPU is worth" can hardly be determined solely by the chip model.
Balaban stated that in the cost of a GPU hour, the largest proportion is equipment depreciation. GPUs, servers, and network equipment are used for years after procurement, and the initial investment needs to be amortized over every subsequent actual hour of use.
Behind this is a typical capital-intensive business. According to Balaban's estimates, in a gigawatt-scale AI factory, power generation facilities require approximately $2 billion to $3 billion, the data center itself costs about $10 billion to $15 billion, and the investment in servers and computing equipment can reach $35 billion to $45 billion, with GPUs accounting for the largest share.
Therefore, judging the value of a GPU requires looking beyond whether it is the latest generation; it is more important to see how many computing tasks it can continuously handle and how much revenue it can generate throughout its entire usage cycle.
This is also the reason why Balaban believes previous-generation GPUs still have a long economic lifespan: as long as the equipment can still handle computing tasks and there is sustained demand, it will not immediately lose its economic value just because a new generation of chips hits the market.
Balaban also mentioned that many companies adopt an accounting depreciation cycle of about six years, but the book depreciation period does not equal how long the equipment can actually be used, let alone how long it can continue to generate revenue. As long as the H100 can still handle computing tasks and someone is willing to rent it, its economic lifespan could be longer than the market initially expected.
In other words, there are at least three different "time scale" measurement systems for a GPU: product iteration determines whether it is the latest generation, accounting depreciation determines how book costs are allocated, and economic lifespan determines exactly how long it can continue to generate revenue.
This is another reason why AI computing systems have not been rapidly commoditized. What customers truly need is not just GPUs, but also high-speed networks, storage, CPU servers, and the software that organizes these devices into clusters. Whether training and inference tasks can run depends on whether the entire system can jointly deliver computing power.
This systemic complexity also explains why Neocloud (AI computing cloud) cannot be simply understood as GPU rental providers. Balaban believes that GPU computing power is not a simple standard commodity; behind it is a vertically integrated service, ranging from land, power permits, and data center construction, all the way to high-performance computing design, virtualization, and cloud software.
When computing power becomes such a capital-intensive and operation-heavy business, the next question determining costs arises: how much time are these expensive devices actually in a working state?
03. GPU Utilization Is Widening the Gap in Token Costs
How long a single GPU can work determines how much more value it can create; while how efficiently an entire GPU system can operate determines how expensive computing power ultimately is.
For customers, tokens can be priced in a unified unit; however, inside the data center, the production cost behind each token varies significantly.
Balaban outlined a chain from energy to tokens: at the very beginning are energy sources like sunlight and natural gas, which are converted into electricity through power generation facilities; after electricity enters the data center, a portion is used to drive server operations, while another portion is consumed by infrastructure such as cooling and power distribution; GPUs, networks, and storage then jointly complete the computing, and ultimately, the model generates tokens and delivers them to users.
Along this chain, the same data center with the same number of GPUs does not mean the same token output. Power utilization efficiency, communication efficiency between GPUs, and model and software optimization will all affect how much computing result the energy and equipment can ultimately be converted into.
And in Balaban's view, one of the most realistic variables among them is utilization rate.
GPUs, servers, and network equipment do not stop depreciating just because there are no tasks. If a GPU is truly used for computing only half the time, its equipment cost can only be borne by fewer actual hours of use. Balaban gave an example that at a 50% utilization rate, the depreciation cost borne by each actual GPU hour is effectively doubled.
Therefore, the problem facing AI infrastructure is not just "whether there are GPUs," but whether expensive GPUs can be kept idle as little as possible.
But when it comes to large-scale clusters, this is no longer as simple as finding an idle GPU. Balaban introduced that a 10,000-GPU cluster, besides the GPUs themselves, also includes CPU servers responsible for organizing tasks, storage systems, and multiple sets of networks undertaking different functions.
When a customer applies for a portion of this computing power, the supplier actually needs to allocate computing, storage, and network resources simultaneously for this task and ensure that these resources can work together.
This is also the value of scheduling and cluster software. They determine whether a batch of expensive physical equipment can be split, combined, and continuously delivered to different customers. Without this layer of software capability, a data center with a massive number of GPUs would also find it difficult to truly turn these devices into on-demand computing power.
As AI Infra enters the era of super nodes and large-scale clusters, purchasing GPUs is only the beginning of the investment. What truly determines the cost of computing power is whether these expensive devices can maintain long-term operation and continuously complete training and inference tasks with the coordination of networks, storage, and software.
GPUs remain the most expensive equipment, but what determines the value of computing power is no longer just the GPUs themselves.