A supercomputer is a shared machine with far more demand than capacity: thousands of users, one pool of nodes. If everyone just launched work by hand, the machine would thrash — jobs stepping on each other, big allocations never assembling, small jobs starving. So access is indirect. You write a batch script that declares what you need — nodes, cores, GPUs, memory, wall-time — and submit it to a workload manager, almost always Slurm. It queues your request against everyone else's and dispatches it when the resources and the policy line up. Think air-traffic control for compute: nothing lands until the controller says so.
What Slurm actually is
Slurm ("Simple Linux Utility for Resource Management") began at Lawrence Livermore National Laboratory and is now the dominant scheduler on the TOP500. Its job is three things, stated plainly in the original design paper: allocate resources to users for some duration, provide a framework to start and monitor work on those resources, and arbitrate contention by managing a queue of pending work.
Architecturally it's a small set of daemons:
slurmctld— the central controller. Holds the queue, makes scheduling decisions, tracks node state. There's a backup that takes over if it dies.slurmd— one per compute node. Launches and supervises tasks, reports health back. Think of it as a remote shell that waits for orders.slurmdbd— the accounting database. Records every job for fair-share, reporting, and limits.
You interact through sbatch (submit a script), srun (launch tasks), squeue (see the queue), and scancel. Resources are grouped into partitions (queues with rules — e.g. a gpu partition, a debug partition with short limits) and shaped further by QOS policies.
Why your job waits: the queue is a policy, not a line
The naive mental model is FIFO — first come, first served. Real schedulers don't work that way, because strict FIFO wastes the machine. Suppose a 500-node job is next in line but only 480 nodes are free; under pure FIFO those 480 nodes sit idle until 20 more free up. Instead Slurm uses backfill: it computes when the big job can start, reserves that slot, and then lets smaller jobs that fit in the meantime run now — as long as they finish before the reservation. This is why an accurate wall-time estimate helps you: a short, honest time limit makes your job eligible to slip into gaps. Over-request wall-time and you wait longer.
On top of backfill sits fair-share: priority decays for users/groups who've recently consumed a lot, so one heavy user can't monopolize the machine. Your position in the queue is a computed priority — age, size, partition, QOS, and fair-share factor — not your arrival time.
#!/bin/bash #SBATCH --job-name=train #SBATCH --partition=gpu #SBATCH --nodes=4 #SBATCH --gpus-per-node=8 #SBATCH --time=02:00:00 # honest limits backfill better srun python train.py
#SBATCH directives are the resource request the scheduler reasons about.The unglamorous, load-bearing part
A badly tuned scheduler leaves a multi-million-dollar machine half-idle while users wait hours — and getting Slurm to behave on exotic hardware is genuine engineering, not configuration. On Cray XT/XE systems the compute nodes weren't launched like ordinary Linux boxes; jobs went through Cray's ALPS/BASIL application-launch layer, so Slurm had to be taught to drive that interface instead of its usual node-launch path. That integration work — making a general scheduler cooperate with a vendor's proprietary launch stack on a 20-cabinet machine — is exactly the kind of coordination problem this guide keeps returning to.
Why it matters: whether you're on a national supercomputer or a shared GPU cluster at work, the scheduler decides your throughput — and most "the cluster is slow" complaints are really "I don't understand the queue policy." Learn backfill and fair-share and you move from waiting on the machine to working it: right-sized requests, honest wall-times, the right partition. That single skill is often worth more than a hardware upgrade.