When you think "network" you think TCP/IP, sockets, the kernel copying packets, latencies measured in hundreds of microseconds or milliseconds. Throw that away. A supercomputer's interconnect is a different animal built for one job: let thousands of nodes exchange data as if they shared memory. The dominant technologies are InfiniBand, Cray's Slingshot (a specialized high-speed Ethernet), and — inside a node, between GPUs — NVIDIA's NVLink. All of them are engineered around the same brutal priority: minimize latency, because at scale the network is the machine.
RDMA: getting the OS out of the way
The key idea is RDMA — Remote Direct Memory Access. On an ordinary network, sending data means: your program copies it into a kernel buffer, the kernel builds packets, the NIC sends them, the far side's kernel receives and copies them up to its application. Every step burns CPU and adds latency. RDMA collapses all of it. The network card reads directly from your process's memory and writes directly into the remote process's memory — no kernel involvement in the data path, no copies, no CPU. It's called kernel bypass, and it's the reason HPC networks hit single-digit-microsecond latency where TCP measures in the hundreds. A classic RDMA-based MPI over InfiniBand reported small-message latency around 6–7 µs two decades ago; modern fabrics are near 1 µs.
TCP/IP path RDMA path
app buffer ─┐ app buffer ─┐
↓ copy │ NIC reads
kernel buf │ │ directly
↓ build packets ↓
NIC ──────┘ ~100+ µs, CPU busy NIC ────┘ ~1 µs, CPU idle
(writes straight into
remote app memory)
Why the interconnect is the ceiling
Recall the coordination problem from the first page: every timestep, nodes trade boundary data and then hit a barrier where no one advances until everyone arrives. That means the job's speed is bounded by how fast the network delivers messages and how fast a collective (a coordinated all-to-all exchange) completes. A huge fraction of real HPC jobs are communication-bound — the processors spend meaningful time waiting on the network. Push latency down or bandwidth up and those jobs get faster with no change to the compute. This is why the interconnect is a first-class design decision, not plumbing, and why papers dissect a new fabric like Slingshot in the same depth others reserve for a CPU: its adaptive routing and congestion control determine whether a 10,000-node job scales or stalls.
Three tiers of "network"
- Within a node — NVLink. GPUs in one box talk over NVLink/NVSwitch, not PCIe. Fourth-gen NVLink moves 900 GB/s per GPU; the current generation reaches 1.8 TB/s per GPU — bandwidth an order of magnitude past PCIe, because GPUs sharing a computation need to exchange data almost as fast as they read their own memory.
- Between nodes — InfiniBand or Slingshot. This is the RDMA fabric stitching thousands of nodes into one machine, typically 200–400 Gb/s per link with microsecond latency and topologies (fat-tree, dragonfly) designed so any node can reach any other in a few hops.
- To the outside world — Ethernet/IP. The boring management network you actually SSH over. It has nothing to do with how the job runs.
Why it matters: distributed training lives or dies on this. Every step, thousands of GPUs must average their gradients via an all-reduce — a collective that is pure interconnect work. The GPUs finish their math and then wait for the all-reduce to complete across the fabric; scale the cluster up and the run becomes bounded by network latency and bandwidth, not FLOPS. It's why AI clusters are built on InfiniBand and NVLink, why NCCL (NVIDIA's collective library) is tuned to the topology, and why "we added more GPUs and it didn't get faster" almost always means you hit the interconnect. The network that isn't your network is the one that decides whether your training run scales.