A language model generates text one token at a time: to produce token n it must attend to tokens 1…n−1. That autoregressive loop is inherently sequential and, per request, leaves a big GPU mostly idle — a single user can't saturate the hardware. So the entire art of inference serving is overlapping many requests to fill that idle silicon, without letting any one user's latency blow up. Three ideas carry most of the weight.
Continuous batching: pack the GPU, don't wait
The naive approach is static batching: collect, say, 16 requests, run them together, return all 16, repeat. The problem is that requests finish at different lengths — a 20-token reply and a 500-token reply are in the same batch, and the whole batch waits for the slowest while finished slots sit idle. Orca (OSDI 2022) replaced this with iteration-level scheduling — now widely called continuous batching: the scheduler makes decisions every token-generation step, evicting finished sequences and admitting new ones mid-flight. No request waits for a batchmate to finish. On GPT-3-scale models Orca reported dramatic throughput gains over prior serving systems at the same latency. This is now table stakes in every serious inference stack.
The KV-cache is the real bottleneck
To avoid recomputing attention over the whole prefix at every step, the model caches the key and value tensors for every past token — the KV-cache. It's what makes generation fast, and it's enormous: it grows with sequence length and with the number of concurrent requests, and it lives in the same precious GPU HBM as the model weights. This is why memory, not FLOPS, is usually what caps how many users you can serve at once. Early systems allocated one big contiguous KV block per request sized for the worst case, wasting 60–80% of it to internal fragmentation and over-reservation. vLLM's PagedAttention (SOSP 2023) borrowed the operating-system trick this whole field guide keeps returning to: split the KV-cache into fixed-size blocks mapped to non-contiguous physical memory, like virtual-memory paging. Near-zero waste, plus blocks can be shared across requests with a common prefix — yielding 2–4× the throughput of the previous best at similar latency.
KV-cache size ≈ 2 (K and V) × layers × 2 (bytes, fp16)
× hidden_dim × sequence_len × batch
Llama-2-13B, 4096-token context, ONE request:
≈ 2 × 40 × 2 × 5120 × 4096 ≈ 3.4 GB per sequence
→ A 40 GB GPU holds the ~26 GB of weights, leaving room for only
a handful of long-context requests. Memory, not compute, sets the
ceiling — which is exactly what paged KV memory attacks.
Speculative decoding, and the tradeoff you actually tune
Speculative decoding attacks the sequential loop itself: a small, cheap "draft" model proposes several tokens ahead, and the large model verifies them in one parallel pass, accepting the run that matches. When the draft guesses well, you get several tokens for roughly the cost of one big-model step. Underneath all of this sits one governing tension: throughput vs latency. Bigger batches use the GPU more efficiently (higher tokens/sec across all users) but make each individual user wait longer per token. Serving is the continuous act of choosing where on that curve to sit — and memory is usually what decides how far toward throughput you can go before you run out of room.
Why it matters: the cost of an AI product is dominated by inference, and inference cost is set by how many concurrent users one GPU can serve — a number governed by KV-cache memory, not raw compute. Understanding continuous batching and paged KV memory is the difference between buying three GPUs and buying thirty for the same traffic. When someone says a model is "too expensive to serve," the fix is usually in the serving layer, not a smaller model.