dhar.win

How it works · Part 2

How a GPU works

The chip the AI boom runs on, from the thousands of small cores on its die to the rack it sits in. Twelve exhibits, each one idea, built around Nvidia's H100. Play the tour, or pick any exhibit and drag the model.

Diagram of a GPU package: the die in the centre with its grid of streaming multiprocessors, flanked by six stacks of high-bandwidth memory on a silicon interposer
A gallery, not a single machine: each exhibit is one idea, drawn to its own scale. Figures are for Nvidia's H100 unless a step says otherwise.
Playing the tour

    Step 1 of 12

    A few fast cores, or thousands of slow ones

    A CPU runs one stream of instructions as fast as possible. A GPU fills the same area with thousands of small cores and asks them all to do the same operation on different numbers at once, which is exactly what a neural network needs.

    Transistors (H100)
    80 billion
    Cores
    16,896 + 528 tensor
    Matrix maths, FP16
    ~990 TFLOPS
    Memory bandwidth
    3.35 TB/s
    Power
    700 W
    Link to other GPUs
    900 GB/s
    GPUs in one NVL72 rack
    72
    Price, roughly
    $30,000

    The words that come up

    Twelve terms from the spec sheets and the earnings calls.

    CUDA core
    One of the thousands of small arithmetic units on an Nvidia GPU; CUDA is also the software platform that programs them.
    Streaming multiprocessor (SM)
    The GPU's building block: 128 cores, four tensor cores, a scheduler, registers and shared memory. An H100 has 132.
    Tensor core
    A unit that multiplies small matrices in a single clock. Nearly all of a neural network's arithmetic runs through them.
    FLOPS
    Floating-point operations per second. A teraflop is a trillion; a petaflop a thousand trillion. Always check the precision and whether sparsity is assumed.
    Warp
    A group of 32 threads that execute the same instruction together on 32 cores.
    HBM
    High-bandwidth memory: DRAM dies stacked and wired through each other, sitting beside the die on the package. 80 GB on an H100, 192 GB on a B200.
    CoWoS
    TSMC's packaging that puts the die and the memory stacks on one silicon interposer. Its capacity has set how many GPUs could be built.
    NVLink
    Nvidia's chip-to-chip link: 900 GB/s per H100, 1.8 TB/s per Blackwell GPU, so a rack of them can share memory.
    Precision (FP16, FP8, FP4)
    How many bits each number uses. Fewer bits mean roughly double the throughput per halving, at some cost in accuracy.
    Memory wall
    When a chip waits for data rather than computing. Generating text one token at a time is almost always memory-bound.
    TDP
    The chip's design power: 700 W for an H100, about 1,000 W for a B200. It sets the cooling and the electricity bill.
    Accelerator
    Any chip built mainly for AI maths: Nvidia and AMD GPUs, Google's TPUs, Amazon's Trainium, Microsoft's Maia.

    Figures from Nvidia's Hopper and Blackwell architecture papers and product pages, Meta's Llama 3 paper (training interruptions), and company statements on cluster sizes; prices as reported in the trade press. Simplified: real dies, boards and racks have far more parts than are drawn here.

    Next in the series: A wafer's three months in the fab · Part 1: How EUV lithography prints a chip