The last two pages were about the ceiling: peak FLOPS, and the roofline that says which ceiling binds. This page is about the floor — the gap between the FLOP/s a chip can do and the FLOP/s it actually does over a wall-clock hour of your job. That gap is enormous, and it is rarely the GPU's fault. It is the pipeline feeding it.
The chip is the end of a supply chain
Every batch of work travels the same path before a tensor core touches it: read from parallel storage → land in host memory → decode/augment on the CPU → cross the network and PCIe/NVLink → land in HBM → finally, compute. The GPU is the fastest link in that chain by a wide margin, which is exactly why it's the one most often idle: any slower upstream stage becomes the bottleneck, and the compute units stall waiting. A slow data loader, a straggler in an all-reduce, a checkpoint write to shared storage — each one drops the whole tensor array to zero throughput for its duration, while the expensive silicon does nothing.
"GPU utilization" is a lie counter
The number most people watch — nvidia-smi's "GPU-Util" — is not what its name implies. It reports the fraction of time at least one kernel was resident on the device over a sampling window. It says nothing about whether the kernel used 5% or 95% of the arithmetic units. A GPU stuck running a single tiny memory-bound kernel reads as "100% utilized" while delivering a rounding error of its peak FLOPS. To see the truth you need SM-occupancy and tensor-core-active counters (via NVIDIA DCGM / Nsight), not the headline util percent. Confusing the two is the most common self-inflicted wound in GPU work.
What "GPU-Util 100%" actually guarantees: a kernel was on the device. What it does NOT tell you: - SM occupancy (were most cores active, or a handful?) - tensor-core active % (are the fast units even running?) - memory-bound stall (waiting on HBM, not computing) - pipeline starvation between steps (loader / network / I/O) Trust: dcgm DCGM_FI_PROF_PIPE_TENSOR_ACTIVE, SM_ACTIVE, DRAM_ACTIVE Not: nvidia-smi's single "GPU-Util" number.
What real clusters actually deliver
This is not a small effect at the top end. In a two-month trace of a large multi-tenant training cluster at Microsoft, Jeon et al. found that locality constraints, gang scheduling, and job failures systematically eroded delivered GPU utilization — GPUs behaving as a "monolithic resource that cannot be shared at a fine granularity" left capacity stranded across the fleet. A later Microsoft study drove the knife in at the job level: examining 400 real deep-learning jobs whose average GPU utilization was 50% or less, the authors traced 706 distinct low-utilization issues to two families of cause — "insufficient GPU computations" (batch too small, CPU-bound work on a GPU node) and "interruptions caused by non-GPU tasks" (checkpointing, data loading, synchronous uploads). In other words: the chip was fine; it was starved and interrupted.
Feeding the beast
The fixes are all about the pipeline, not the processor: prefetch and overlap data loading with compute; stage datasets onto fast local NVMe so storage isn't the roof; raise batch size to give each weight load more work (the roofline argument from page 11); overlap communication with computation so all-reduce hides behind the backward pass; and build fault tolerance so a single node failure doesn't idle a thousand others. Modern training frameworks exist largely to keep the tensor cores fed — because at cluster scale, a few points of sustained utilization is the difference between one datacenter and two.
Why it matters: GPUs are the most expensive compute humanity rents by the hour, and the industry runs them at a fraction of capacity because the supply chain feeding them is neglected. "Buy more GPUs" is the reflex; "keep the ones you have fed" is almost always the cheaper win. If you can't answer "what was the tensor-core-active percent," you don't know your real cost per useful FLOP — and you're probably paying for idle silicon.