How it works · Part 5

How 100,000 GPUs talk

The network is the part of an AI cluster nobody explains, and the part with the most new public companies. Eleven exhibits from the eight GPUs on one board to the fat tree that joins a hundred thousand of them: NVLink, the NIC, InfiniBand against Ethernet, the transceiver, the fibre, co-packaged optics, and the all-reduce that the whole thing exists for. Play the tour, or pick an exhibit and drag it.

A gallery, not a single machine: each exhibit is one idea, drawn to its own scale. Figures are for Nvidia's Hopper and Blackwell systems unless a step says otherwise.

Playing the tour

Step 1 of 12

No model fits on one GPU

A frontier model has hundreds of billions of weights and trains on trillions of tokens; one GPU holds a few hundred billion bytes. So training is split across thousands of GPUs, and several times a second every one of them has to average what it learned with every other. That averaging is network traffic, and while it runs the arithmetic waits.

NVLink per GPU (Blackwell): 1.8 TB/s
Network port per GPU: 400–800 Gb/s
GPUs in one NVLink domain: 72
Switch chip, 2026: 102.4 Tb/s
Tiers for 100,000 GPUs: 3
Transceivers in that cluster: ~300,000
Time spent communicating: 10–30 %
Power per port, CPO vs pluggable: 3.5× lower

The words that come up

Twelve terms from the data sheets and the IPO prospectuses.

Scale-up: The fabric that makes a few dozen GPUs behave as one: NVLink and NVSwitch, over copper, inside a rack. Terabytes per second per GPU.
Scale-out: The fabric that joins thousands of racks: one network card per GPU, switches, and optical fibre. Hundreds of gigabits per second per GPU.
NVLink / NVSwitch: Nvidia's proprietary GPU-to-GPU link and the crossbar chip that joins eighteen of them per GPU. 900 GB/s on Hopper, 1.8 TB/s on Blackwell.
UALink: The open scale-up standard from AMD, Intel, Google, Microsoft, Meta and others, finished in 2025, so accelerators from different vendors can share a fabric.
RDMA: Remote direct memory access: a network card writes straight into another machine's memory with no CPU in the path. GPUDirect extends it to GPU memory.
InfiniBand: The lossless, RDMA-native network from Mellanox (now Nvidia) that ran most early AI clusters. Quantum is the current switch family.
RoCE / Spectrum-X / Ultra Ethernet: RDMA over Ethernet, and the tuned Ethernet fabrics (Nvidia's Spectrum-X, Broadcom's Tomahawk with the Ultra Ethernet spec) that now match InfiniBand at lower cost.
Transceiver: The pluggable module at each end of a fibre that turns electrical signals into laser light and back: a DSP, lasers, modulators, photodiodes, in a thumb-sized case.
SerDes: Serialiser/deserialiser: the circuit that drives one lane of a link at 100 or 200 Gb/s. A switch chip has hundreds; their speed sets the chip's capacity.
Co-packaged optics (CPO): Putting the photonics on the switch package next to the chip, so only fibre leaves the box. Several times less power per port, far fewer pluggables.
Fat tree / Clos: The switch topology of nearly every AI cluster: leaf, spine and core tiers with as much bandwidth up as down, so any GPU reaches any other at full speed.
All-reduce / NCCL: The collective operation that leaves every GPU holding the sum of all GPUs' gradients, and the Nvidia library that chooses how to run it over the topology.

Figures from Nvidia's Hopper, Blackwell, NVLink, Quantum and Spectrum-X materials, Broadcom's Tomahawk 5 and 6 announcements, Meta's Llama 3 paper (cluster sizes and interruptions), the Ultra Ethernet Consortium's specification, and company statements on cluster sizes; power-per-bit figures are indicative. Simplified: real boards, switches and transceivers have far more parts than are drawn here.

Next in the series: The data centre · The whole series · Part 4: How a language model works · Part 3: A wafer's three months in the fab · Part 2: How a GPU works · Part 1: How EUV lithography prints a chip