FLOPS, and why "peak" is a lie

A FLOP is one floating-point operation — a single add or multiply on real numbers. FLOP/s (often written FLOPS) is the rate: operations per second. "Peak" FLOPS is a pure counting exercise: take the number of floating-point units, multiply by operations they can retire per cycle, multiply by clock frequency and by the number of chips. Nobody measures it — it is computed from the datasheet. That is the first tell. Peak is what the hardware could do if reality cooperated; it is an upper bound, not a measurement.

Why LINPACK is the benchmark that flatters

The TOP500 list ranks supercomputers by HPL — High-Performance LINPACK — which solves a dense system of linear equations Ax = b by LU factorization. The operation count is fixed and known: 2/3·n³ + O(n²) double-precision FLOPs. HPL was not chosen because it looks like real science. It was chosen, in the TOP500's own words, because the problem "is very regular" and so "the performance numbers give a good correction of peak performance." Translation: it is a benchmark engineered to sit as close to peak as anything realistic can, because its arithmetic-to-data ratio is enormous — a dense matrix multiply reuses each byte many times, keeping the FLOP units fed.

The list reports two numbers side by side, and the gap between them is the whole point:

• Rpeak — theoretical peak, counted from the hardware.
• Rmax — the best HPL run actually sustained.

Their ratio, Rmax/Rpeak, is HPL efficiency. On a well-tuned machine it lands around 60–80%. And that is on the benchmark designed to hug peak. The TOP500 is blunt about the limit: LINPACK "does not reflect the overall performance of a given system, as no single number ever can."

What real applications get

Because HPL overstates delivered performance, Dongarra, Heroux and Luszczek introduced a deliberate counterweight: the HPCG benchmark (High-Performance Conjugate Gradient). HPCG uses the sparse-matrix, memory-bound access patterns typical of actual scientific codes — the kind that constantly fetch scattered data and reuse it little. On the same machines, HPCG routinely delivers a few percent of peak, one to two orders of magnitude below HPL. Same silicon, same Rpeak; the fraction you keep collapses the moment the work stops looking like a dense matrix multiply.

System peak (Rpeak):            100 PFLOP/s   (from the datasheet)
HPL sustained (Rmax):            ~70 PFLOP/s   (~70% of peak)
HPCG sustained (real-app proxy):  ~2 PFLOP/s   (~2% of peak)

Same hardware. The workload picks which number you live with.

The GPU spec-sheet version of the same trick

Every AI accelerator is sold on a headline TFLOPS figure — and it is the same peak count, now often measured in a low-precision format (FP16, FP8) with the tensor cores fully saturated. That number is real in the sense that a dense matmul at the right shape and precision can approach it. It is a lie in the sense that your actual model — with attention, normalization, activation functions, small batch sizes, and memory-bound decode steps — will keep a fraction of it. The next page, on the roofline model, is exactly the tool for predicting which fraction, and why.

Why it matters: when a vendor quotes "X TFLOPS," they are quoting Rpeak — the number you cannot buy. What you actually get is set by your workload's data-reuse pattern, not the sticker. Learn to ask "at what precision, at what sustained fraction, on what shape?" and you stop overpaying for peak you'll never occupy — and you start sizing clusters on delivered throughput instead of marketing.