[Relevant Companies Mentioned in This Article] TSMC (Taiwan Semiconductor Manufacturing Company), AMD, Tower, GF (GlobalFoundries), Nokia.
What Is the Most Critical Factor for AI Computing Power at the Current Stage
Over the past decade, computing clusters competed purely on computing power, and the "arms race" for AI chips had only one metric: TOPS. However, in the design of ultra-large-scale training and inference clusters today, the most critical bottleneck for the industry is "effective computing power" rather than "theoretical peak computing power." The ceiling of effective computing power is no longer determined by how wide the MAC (Multiply-Accumulate) unit of a single GPU in the cluster is, but by how low the latency, how high the bandwidth, and how low the power consumption can be when completing a task between two chips, within a single rack, or between racks. The concept of racing against time has become incredibly tangible at this moment.
The driving force behind such a massive shift in AI computing power is the change in AI workloads. Dr. Matthew pointed out that on the training side, the parameter scale of a single model is still growing, but more critically, agent modes and multimodal inference have emerged on the inference side. It is no longer about running a model once per query, but continuously inferring in the background, calling the model multiple times, and repeatedly verifying results. Distillation technology compresses the capabilities of large models into smaller ones, deploying them on edge devices such as smartphones and laptops, leading to an exponential amplification of the total inference volume. Meanwhile, on the user demand side, generative AI has transformed from a merely "fun tool" into one that genuinely enhances productivity; tools like CodeX and Claude can indeed help users and enterprises complete specific tasks.
Consequently, the entire backend computing cluster must not only process a larger volume of data but also perform far more frequent all-to-all communications internally. When discussing networking in large-scale computing clusters, the industry usually focuses on the Scale-out network external to the racks, which is the Ethernet layer that feeds data into the racks. This is because the previous focus was on user growth, where massive but relatively simple external access demands required a massive Scale-out network. However, the real pressure point has now shifted to the Scale-up network inside the racks. Ultra-large-scale clusters interconnecting tens of thousands of cards need to tightly connect thousands of computing chips into a single "giant computer," which requires a Scale-up network. Dr. Matthew also emphasized a highly useful piece of information: The total bandwidth required for Scale-up is nearly an order of magnitude larger than that for Scale-out.
This means system designers no longer just look at the on-paper computing power of a single GPU, but shift the evaluation focus to unit shoreline bandwidth density (Shoreline bandwidth density refers to how many Tbps of interconnect bandwidth can be pushed per millimeter along the chip edge) and energy consumption per bit. These two core parameters determine whether a computing cluster can scale linearly. Since interconnects play a decisive role, the scaling speed of Scale-up far exceeds the limits that electrical signals over copper cables can carry, so optics must be used. In the past, copper cables could handle this Scale-up layer, but now only optics can barely sustain it.
Scale-up Is Where Silicon Photonics Is Most Needed
Following this logic, Dr. Matthew discussed NVIDIA's NVLink copper backplane and Google TPU's ICI, which basically use differential pair copper cables to solve connections within a few meters. Copper cables are indeed still usable for short distances, but as speeds increase from 400G to 800G, copper cables are completely insufficient. In the history of interconnect networks, the dividing line between electrical and optical signals is roughly at 100 Gbps x m (a common physical constraint is: bandwidth × distance). Once the single-channel rate reaches 112 Gbps or 224 Gbps, even over a short distance of just one or two meters, the loss, crosstalk, and reflection of copper cables have reached a point where retimers are absolutely necessary. Adding each stage of retimer increases power consumption, chip area, and cost. Especially when the Scale-up bus requires hundreds or even thousands of links, the power consumption of retimers directly accounts for nearly 20%. Without calculating a single bit of data, a huge chunk of power efficiency is naturally cut off, which anyone would consider uneconomical.
On the other hand, the density of copper cables is not light, posing a significant challenge to the mechanical tensile strength and heat dissipation of high-density backplanes. This is why, in NVIDIA's iteration from Hopper to Blackwell, although the GB200 NVL72 still uses copper cables for intra-rack connections, it has already reached the extreme edge of copper's limits. The general consensus in the industry is that for the next-level NVL576 or clusters of similar scale, longer-distance Scale-up between racks or within racks must rely on optical pathways.
Dr. Matthew described optical interconnects as a ladder; with each step up, the optical path gets closer to the ASIC chip. First, pluggable optical modules are currently the mainstream for Scale-out, but in Scale-up scenarios, continuing to use pluggable solutions encounters a fatal problem: the electrical channel is too long. Pluggable optical modules need to be installed on the front panel of the rack, and the ASIC connects high-speed signals all the way to the panel. The distance for electrical signals is too far, resulting in massive loss and power consumption along this path, still requiring retimers to consolidate and convert them into optical signals.
Onboard optics, also known as NPO (Near-Package Optics), takes a step further by moving the optical engine from the front panel to the substrate to shorten the electrical channel distance. The trade-off is also huge: maintainability is extremely poor, and you can no longer just replace the faulty module like with pluggable optical modules, so reliability must be up to standard. However, the benefits are equally obvious: shortening the channel reduces latency and power consumption.
A more thorough approach is Co-Packaged Optics (CPO), which directly packages the optical engine for optoelectronic conversion next to the ASIC (or even stacked vertically), so electrical signals only need to travel a few millimeters. In this tightly coupled form, Scale-up interconnect power consumption can be reduced to a very low level. When TSMC previously introduced its COUPE platform, it also compared a 6.4 Tbps OE (Optical Engine) CPO solution, showing a 50% reduction in power consumption. Subsequently, they will develop a CPO with XPU solution with power consumption only one-tenth of the pluggable module solution, and latency reduced to one-twentieth. Of course, Dr. Matthew repeatedly emphasized not to just look at the pJ/bit of the chip itself; all components such as the laser source, driver, transimpedance amplifier, heater, and heat dissipation must be included for a full-link comparison from ASIC to ASIC. Taking a single channel at 200 Gbps as an example, the copper cable interconnect solution consumes about 200 mW per meter, whereas CPO packaging places the optical engine around the ASIC, reducing the loss to less than 100 mW. Multiplied by the total bandwidth of 230 PB/s for the entire system, the saved power consumption will be astonishing.
Why the Industry Must Rely on Silicon Photonics
So why must optical interconnects rely on silicon photonics? The industry does not only have silicon photonics as a solution; Indium Phosphide (InP), Thin-Film Lithium Niobate (TFLN), VCSEL, and even microLED have all stepped out of the lab and onto the industrial stage. This requires returning to the application scenarios of Scale-up: packaging high-density modulation and transceiver channel routing in an extremely small area, requiring low power consumption, using wavelength-division multiplexing for all optical paths to expand bandwidth, ensuring no significant material and process errors between them, and achieving mass production. Silicon photonics is truly the most suitable technological path.
Dr. Matthew also shared AMD's newly disclosed microring modulator optical Chiplet solution. While Chiplet is very mature in electrical chip solutions, optical Chiplets are still relatively rare. AMD's solution packages components such as filtering, signal detection, and modulators onto a single silicon photonics chip. The microring modulator is inherently extremely friendly to CMOS-compatible silicon photonics processes, typically with a radius of only 5 to 10 micrometers. By using a bus waveguide, it can selectively modulate specific wavelength signals, which perfectly fits wavelength-division multiplexing. Multiple microrings can be placed in series on the same waveguide, with each ring corresponding to a wavelength, naturally forming multi-wavelength multiplexing at the transmitter end and demultiplexing at the receiver end. In contrast, traditional Mach-Zehnder modulators (MZM) require a length in the millimeter range to achieve similar modulation bandwidth and extinction ratio. Therefore, to build a transceiver with 6 groups, 16 wavelengths per group, and a total of 96 channels on a silicon photonics chip of less than 10 square millimeters, AMD could only choose the microring structure.
On the other hand, regarding energy efficiency, a microring can be seen as a capacitor with a capacitance in the tens of fF range. The energy required for driving is far less than the tens-of-ohms transmission lines or pF-level lumped components of traveling-wave MZMs. Furthermore, silicon photonics can be hybridly integrated with CMOS circuits, placing drivers and transimpedance amplifiers directly below or above the photonic chip, reducing parasitic parameters and further lowering dynamic power consumption. AMD's optical Chiplet uses pure NRZ on-off keying, with a single channel at 50 Gbps. Without forward error correction, the bit error rate is extremely low (10^-12), and the end-to-end energy consumption has been achieved far below the mass-producible reference line of 5 pJ/bit. In comparison, the VCSEL solution is very unfriendly to dense wavelength-division multiplexing; its single-fiber bandwidth density is hard to compete with silicon photonics, and it is difficult to scale to distances of tens to hundreds of meters. Although Thin-Film Lithium Niobate has strong performance, the mass production of large-scale multi-channel arrays and the maturity of CMOS integration remain at the laboratory level, leaving a significant gap compared to current silicon photonics. The InP platform faces dual pressures of yield and heat dissipation during large-area integration, although the industry is now also focusing heavily on InP materials. In short, what truly determines the life or death of a solution is the system-level trade-offs; packaging, reliability, thermo-mechanical strength, cost, and yield must ultimately find a perfect balance point.
However, silicon photonics is not perfect. Silicon has a large thermo-optic coefficient; when the temperature changes, the refractive index drifts, and the microring resonance peak shifts accordingly. Therefore, a micro-heater and control circuit must be placed next to each microring for closed-loop compensation. AMD is very advanced in this area. Actual tests show that in an environment with a deliberately applied large temperature gradient, the thermal controller can respond instantly to lock the wavelength, ensuring the bit error rate remains unchanged. In CPO packaging, the optical engine is very close to the ASIC, and the computing chip generates a massive amount of heat during operation, making the thermal controller next to the microring extremely important. Of course, another solution in the industry is to use glass substrates, utilizing their low coefficient of thermal expansion and better optical properties to stabilize alignment and isolate heat as a replacement for silicon materials, but engineering this will take time.
Finally, the on-site Q&A session was equally full of valuable insights. External light source or integrated? This has always been a hot topic of debate in the industry. AMD's answer is to start with an external light source; an external light source under long-term full load means it is easy to replace. Higher maintainability brings a huge cost advantage. However, the industry is still moving towards the direction of heterogeneous integration for light emission on silicon. Competitor Intel mentioned in a recently published paper that the reliability of heterogeneously integrated light sources is quite good, but mass production yield and multi-wavelength light source arrays will be a hurdle.
Conclusion
At the current stage, the industry is not "choosing silicon photonics," but "must use silicon photonics." Because it is almost the only technological platform capable of packaging so many functions on a millimeter-scale chip while still possessing mass production, yield, and cost advantages. AMD's optical Chiplet has essentially drawn a baseline for the entire industry: the path of silicon photonics is no longer a laboratory miracle, but a mass-producible engineering reality.
The illustrations in this article are generated by Generative AI
Reference: "AMD's Streshinsky discusses silicon photonics: The scale-up bandwidth for AI server racks can ultimately only be provided by optics."
#Shenxinmeng #WanxinExpo #Semiconductor Industry Research #CPO #AMD