NVIDIA GPUs, CPUs, NVLink and PCIe, Explained Visually
From a single die to a 72-GPU rack: how NVIDIA hardware fits together, why Ada and Hopper are different animals, and which one your workload actually needs
Sep 7, 2026 · 15 min read
Every conversation about AI infrastructure now runs on the same dozen words: GPU, Hopper, Ada, NVLink, PCIe, HBM, domain, superchip. Most people using them have a rough sense of what they mean and no confident sense of how they fit together, which is fine for reading a headline and a problem for signing a purchase order. This piece builds the picture up from the bottom, with a diagram for every layer, so that by the end a spec sheet reads like a description rather than a code.
Nothing here needs a hardware background. If you know that a model is a large pile of numbers and that running it means multiplying those numbers by other numbers, you have all the prerequisites.
Start with the split: what a CPU does and what a GPU does
A CPU is a small number of very capable cores. Each one is built to take a complicated program with branches and loops and unpredictable data, and get through it as fast as possible. A GPU is a very large number of simple cores, built to take one operation and apply it to an enormous amount of data at the same time. Neither replaces the other. The CPU runs the program; the GPU does the arithmetic the program hands it.
Neural networks are matrix multiplication, layer after layer, and matrix multiplication is exactly the "same operation, huge amount of data" shape that a GPU exists for. That is the whole reason the AI boom runs on GPUs. It is also why the GPU cannot work alone: something still has to load the data, run the web server, decide what to compute next, and talk to the storage and the network. That something is the CPU, and the connection between the two is where a lot of the interesting engineering happens.
Anatomy of a GPU
The diagram below is a schematic of a data-centre GPU package, the kind of thing that sits on an HGX board or a PCIe card. Step through it; each stage lights up the part being described.
Billions of transistors, one purpose
The big rectangle in the middle is the die, the actual chip. It is tiled with identical blocks called streaming multiprocessors, or SMs. An H100 has 132 of them, each holding 128 small arithmetic units. That is roughly 17,000 cores doing simple maths in lockstep, which is why a GPU is so good at workloads where the same operation runs on a very large amount of data.
A CPU core is built to be clever about one task. An SM is built to be relentless about ten thousand identical ones.
Where the AI numbers come from
Inside every SM sit tensor cores, dedicated units that do one thing: multiply small matrices and add up the results, in a single step. Neural networks are almost nothing but that operation. Tensor cores also work at low precision (FP16, FP8, and FP4 on Blackwell), trading a little numerical accuracy for a lot of speed. When a spec sheet quotes "989 TFLOPS FP8", that is the tensor cores.
Hopper added a Transformer Engine that switches precision automatically per layer, which is why it pulled ahead for large-model training.
The part that usually limits you
The stacks either side of the die are High Bandwidth Memory, HBM, mounted on the same package so the wires are short and wide. An H100 has 80 GB at 3.35 TB/s; an H200 has 141 GB at 4.8 TB/s. Consumer and Ada-class cards use GDDR6 on the board instead: cheaper, more of it per dollar, but far less bandwidth. For most inference, the model is waiting on memory, not on arithmetic.
Rule of thumb: a model needs about one byte per parameter at INT8, two at FP16. Memory capacity decides what fits; bandwidth decides how fast it runs.
Keeping the hot data close
Between the SMs and the memory sits a large L2 cache, 50 MB on Hopper. It holds the data that many SMs need at the same moment, so it is fetched from HBM once rather than a thousand times. Software that arranges its work to fit the cache runs dramatically faster than software that does not, which is much of what libraries like cuBLAS and FlashAttention are doing for you.
Cache is why two programs with identical arithmetic can differ several-fold in speed on the same GPU.
NVLink out, PCIe out
Everything the GPU learns or produces goes through two kinds of port. NVLink connects it to other GPUs at up to 900 GB/s on Hopper and 1.8 TB/s on Blackwell. PCIe connects it to the CPU, storage and the network at around 128 GB/s on Gen5. That gap, roughly seven times, is why multi-GPU systems are built around NVLink and why the CPU is kept out of the hot path.
Ada-class cards have no NVLink ports at all. If a job needs several of them to talk, it goes over PCIe, and that is slow.
Why the same chip comes in two shapes
The SXM module form, used in DGX and HGX systems, draws up to 700 W on Hopper and needs the host to supply power, cooling and an NVLink board. The PCIe card form drops into an ordinary server slot at 300 to 350 W but with lower clocks and, on Hopper, a slower NVLink bridge. Ada cards like the L4 go as low as 72 W, which is why they can be air-cooled in a dense inference rack.
Power is the honest measure of a data-centre GPU. Everything else, cooling, rack density, electricity bill, follows from it.
NVIDIA's CPU: Grace, and why a GPU company builds one
For most of the last decade the CPU next to an NVIDIA GPU was an Intel or AMD part, and the two talked over PCIe. That works, but it means every piece of data the GPU wants from the CPU's memory has to be copied across a relatively narrow bus, and the two sides keep separate copies of everything. As models and datasets outgrew GPU memory, that copy became a wall.
Grace is NVIDIA's answer: an Arm-based server CPU with 72 cores and up to 480 GB of fast LPDDR5X memory, designed from the start to be soldered onto the same board as a GPU and wired to it with NVLink-C2C, a chip-to-chip version of NVLink running at 900 GB/s. The result is a superchip, one CPU and one or two GPUs sharing a single, coherent memory space. The GPU can read the CPU's memory directly, slower than its own HBM but without a copy step, and the CPU can see what the GPU has done without asking.
Grace Hopper (GH200) pairs one Grace with one Hopper GPU. Grace Blackwell (GB200) pairs one Grace with two Blackwell GPUs and is the building block of the NVL72 rack. Vera, the successor CPU with 88 custom Arm cores, arrives with the Rubin generation from late 2026. The naming convention, once you see it, is simply CPU name followed by GPU name.
PCIe and NVLink: the roads
PCIe, Peripheral Component Interconnect Express, is the universal bus inside a computer. Graphics cards, network cards, NVMe drives and almost everything else plug into it. It is measured in lanes; a GPU uses sixteen. Each generation roughly doubles the speed per lane, so a Gen4 x16 slot moves about 64 GB/s in both directions combined, Gen5 about 128 GB/s, and Gen6, arriving now, about 256 GB/s. That is a lot of bandwidth for a drive or a network card. It is not a lot for two GPUs that want to share the middle of a matrix multiply.
NVLink is NVIDIA's own point-to-point link between GPUs, with far more wires and a protocol built for one job. Fourth-generation NVLink on Hopper gives each GPU 900 GB/s; fifth on Blackwell, 1.8 TB/s; sixth on Rubin, 3.6 TB/s. Roughly seven to thirty times PCIe, depending on the generations you compare.
Two practical consequences. First, if a model has to be split across several GPUs, the links between them set the speed, not the GPUs themselves, so a rack of Hopper GPUs on NVLink will beat a rack of Hopper GPUs on PCIe by a wide margin for that workload. Second, Ada-generation cards have no NVLink at all. They are superb for jobs that fit on one card, and a poor choice for jobs that do not.
The NVLink domain
A domain is the set of GPUs that can reach each other's memory at NVLink speed, as if they were one large device. It is the most useful single concept for understanding why hardware is packaged the way it is, and it has grown by an order of magnitude with each generation.
One GPU, one memory
A single H100 has 80 GB. A 70-billion-parameter model at FP16 needs 140 GB before you have loaded a single token of context. It does not fit, and no amount of clever software changes that. This is the first reason multi-GPU systems exist: capacity, not speed.
Quantizing to INT4 would get that model to about 35 GB and onto one card, at some cost in quality. Fine-tuning it would not fit at all.
A bridge between two
Put two GPUs in one server and join them with an NVLink bridge, and software can split the model across both. Layers 1 to 40 on the first card, 41 to 80 on the second, with every layer's output crossing the bridge. On Hopper PCIe cards the bridge runs at 600 GB/s; on SXM it is 900 GB/s. The pair now behaves as one 160 GB device.
This only works because the bridge is fast. Do the same split over PCIe and the model spends most of its time waiting for the bus.
The classic node
Eight GPUs on one HGX baseboard, wired to NVSwitch chips so that every GPU reaches every other GPU at the full 900 GB/s simultaneously, not just its neighbours. This is the DGX H100, the unit most training clusters are built from, and the shape most cloud "8×H100" instances expose. 640 GB of HBM (1.1 TB with H200) in one address space.
NVSwitch is the difference between a ring and a mesh. Without it, GPU 1 talking to GPU 5 would have to hop through 2, 3 and 4.
The rack is the computer
Blackwell moved the domain from a board to a whole rack. GB200 NVL72 puts 72 GPUs in 18 compute trays, links them through 9 NVSwitch trays in the middle, and gets 130 TB/s of NVLink bandwidth and 13.5 TB of HBM in a single domain. Vera Rubin NVL72, shipping from late 2026, keeps the shape and doubles the link speed with NVLink 6.
A trillion-parameter model fits in one domain. That is the whole reason the rack was designed this way, and why "NVL72" is now a unit of purchase.
Where NVLink stops
Between racks, GPUs talk over the network, InfiniBand or NVIDIA's Spectrum-X Ethernet, at roughly 100 GB/s per GPU with ConnectX-8 or -9 adapters. That is ten to thirty times slower than inside the domain. Training a model across a thousand GPUs is the art of arranging work so that the fast links do the heavy traffic and the slow ones carry only what they must.
Scale-up means a bigger domain. Scale-out means more domains. Both matter, and they are engineered completely differently.
Ada and Hopper: same year, different jobs
NVIDIA names GPU architectures after scientists. Ada Lovelace and Hopper (for Grace Hopper) both launched in 2022, both on TSMC's 4-nanometre-class process, and they are routinely confused because of that. They are built for different rooms of the building.
Ada is the graphics lineage. It powers the GeForce RTX 40 series, the RTX 6000 Ada workstation card, and the L4 and L40S data-centre cards. It has ray-tracing cores, video encoders, display outputs, and fourth-generation tensor cores that support FP8. Its memory is GDDR6 on the board, it ships as a PCIe card, and it has no NVLink. The L4 draws 72 W and fits in a slim air-cooled server; the L40S draws 350 W and carries 48 GB. These are the cards you reach for to serve models that fit on one GPU, to run vision and video workloads, and to fine-tune small models cheaply.
Hopper is the data-centre compute lineage. It is the H100, the H200 and the GH200 superchip. It drops the graphics hardware entirely: no ray-tracing cores, no display outputs. In their place it has HBM memory at several times the bandwidth, NVLink 4 at 900 GB/s, NVSwitch support for 8-GPU domains, Multi-Instance GPU partitioning, confidential computing, and the Transformer Engine, which manages FP8 precision automatically and was the single biggest reason large-model training moved to it. It ships as a 700 W SXM module or a 350 W PCIe card with reduced clocks.
| Ada Lovelace | Hopper | |
|---|---|---|
| Named after | Ada Lovelace, first programmer | Grace Hopper, compiler pioneer |
| Launched | 2022 | 2022 |
| Products | RTX 40 series, RTX 6000 Ada, L4, L40, L40S | H100, H200, GH200 (with Grace) |
| Memory | GDDR6, up to 48 GB, ~0.9 TB/s | HBM3 / HBM3e, 80 to 141 GB, 3.35 to 4.8 TB/s |
| GPU-to-GPU | PCIe only | NVLink 4, 900 GB/s; NVSwitch domains of 8 |
| Precision | FP8 on 4th-gen tensor cores | FP8 with Transformer Engine (automatic scaling) |
| Graphics hardware | RT cores, NVENC/NVDEC, display outputs | None |
| Partitioning | No MIG | MIG, up to 7 instances |
| Form and power | PCIe card, 72 to 350 W | SXM 700 W, or PCIe 350 W |
| Built for | Inference of models that fit one card, graphics, video, small fine-tunes | Training and serving models that span cards; anything memory-bound |
The mental shortcut: if the workload lives on one card, Ada (and its Blackwell successors, the RTX PRO 6000 line) is usually the better buy per dollar and per watt. If the workload spans cards, Hopper and its successors are the only real option, because the answer is in the interconnect and the memory, not in the raw compute.
Where they sit now: Blackwell and Rubin
Hopper and Ada are no longer the newest, though both remain in production and both remain the bulk of what is deployed. Blackwell, launched in 2024, is the current mainstream data-centre generation: the B200 with 192 GB of HBM3e at 8 TB/s, NVLink 5 at 1.8 TB/s, FP4 support, and the GB200 NVL72 rack that made the 72-GPU domain standard. On the graphics side it is the RTX 50 series and the RTX PRO 6000 Blackwell, which brings 96 GB of GDDR7 to the workstation and inference card that Ada's L40S occupied.
Rubin, named for astronomer Vera Rubin, is the next step and is entering production as this is written, with cloud availability from the second half of 2026: 288 GB of HBM4, NVLink 6 at 3.6 TB/s, the Vera CPU, and a Vera Rubin NVL72 rack with 260 TB/s of internal bandwidth. NVIDIA's own figures claim up to a tenfold reduction in cost per token for large mixture-of-experts models against Blackwell. Rubin Ultra follows in 2027, and Feynman after that.
Reading a spec sheet
The terms that appear on every product page, briefly:
Which one do you actually need
The architecture names matter less than three questions: does the model fit on one card, does the work span cards, and how many watts can the room take. Rough guidance:
| You want to | Reach for | Why |
|---|---|---|
| Serve a small model (under 10B) to a modest number of users | L4, L40S, RTX PRO 6000 (Ada or Blackwell) | A quantized model fits in 24 to 96 GB of GDDR; no GPU-to-GPU traffic; low power; runs in any server. |
| Fine-tune an 8B model with LoRA | One L40S, A100 or H100 | Fits on one card; NVLink irrelevant; HBM helps but is not required. |
| Serve a 70B model at low latency | 2 to 4 Hopper or Blackwell GPUs in one NVLink domain | The model spans cards; every token crosses the link; PCIe would dominate latency. |
| Train or fine-tune a large model end to end | 8-GPU HGX nodes, or an NVL72 rack | Gradients cross the domain constantly; NVSwitch keeps it a mesh, not a chain. |
| Run many small isolated jobs | Hopper with MIG, or several Ada cards | Isolation and cost per job matter more than peak throughput. |
| Work with datasets larger than GPU memory | Grace Hopper or Grace Blackwell superchips | The GPU reaches CPU memory over NVLink-C2C without copying through PCIe. |
The through-line of all of this is that NVIDIA stopped selling chips a while ago and started selling domains. A GPU is defined as much by what it can talk to, and how fast, as by what it can compute. Once you read the product line that way, Ada and Hopper stop being confusing siblings and become what they are: one card for the work that fits on a card, and one node for the work that does not. Blackwell and Rubin are the same two stories, told at a larger scale.