Reliability is usually quoted as MTBF — mean time between failures. A good node might have an MTBF of, say, 5 years. That sounds bulletproof until you remember a supercomputer job runs on thousands of them at once, and any one failure takes down the whole calculation. Independent failures compose the unforgiving way: the MTBF of the system is the per-node MTBF divided by the number of nodes.
The math that ruins your week
Take 10,000 nodes, each with a 5-year MTBF. Five years is about 43,800 hours. Divide by 10,000 and the system-level mean time between failures is 4.4 hours. On average, something fails roughly every four hours — not because the hardware is bad, but because there is so much of it. Schroeder and Gibson, studying real failure logs from LANL and other DOE machines, found exactly this scaling and projected that petascale (and now exascale) systems would see interruptions on the order of hours, sometimes less.
So a job that needs 150 hours of compute cannot simply run for 150 hours. It will be interrupted, dozens of times. The only way through is checkpoint/restart: every so often, pause and write the complete application state — all the memory that matters — to the parallel filesystem. When a node dies, the scheduler reschedules the job, it reads the last checkpoint, and resumes from there. You lose only the work since the last save, plus the cost of the restart.
How often should you checkpoint?
Not too rarely — a failure then costs hours of lost progress. Not too often — writing terabytes of state to disk is itself expensive and wasted if no failure occurs. There is an optimal interval that balances the two, and John Daly derived the classic estimate. To first order:
optimal interval τ ≈ sqrt( 2 · C · M ) C = time to write one checkpoint (e.g. 0.1 h) M = system mean time to failure (e.g. 4.4 h) τ ≈ sqrt(2 · 0.1 · 4.4) ≈ 0.94 h → checkpoint hourly
The intuition is worth keeping: as your machine gets less reliable (M shrinks) or checkpointing gets cheaper (C shrinks), you checkpoint more often — but only as the square root, so a machine that fails 4× as often only wants checkpoints twice as frequent. Daly's contribution was a higher-order correction that stays accurate when failures are frequent enough that the simple square-root formula starts to drift.
What actually dies at hour 47
The failure is rarely dramatic. A power supply flickers, a DIMM throws an uncorrectable error, an optical cable degrades, a GPU falls off the bus, a network link flaps and a collective (see page 06) hangs forever. The job doesn't crash cleanly — it stalls, one rank waiting on a peer that will never answer. Detecting that, killing the job, and restarting from checkpoint is its own engineering discipline. And the restart isn't free: 10,000 nodes all reading their checkpoint shards from the same filesystem at once can saturate the storage (see the metadata wall, page 04), so checkpoint I/O is often the thing you tune hardest.
Why it matters: training a frontier model runs for weeks on tens of thousands of GPUs — a regime where the system MTBF is measured in hours and a failure is guaranteed many times over. That is why every serious training stack checkpoints model and optimizer state continuously, keeps hot spares ready to swap in, and is judged partly on how fast it can detect a dead rank and resume. Meta's Llama 3 team reported hundreds of interruptions across a single run; the entire investment is worthless if you can't restart. Daly's square-root rule from 2006, written for physics simulations, is the same math a GPU cluster uses today to decide how often to snapshot a half-trained model.