There are two ways to run a large language model. One is to send text to someone else's GPU over HTTPS and get tokens back. The other is to load the weights into memory on a machine in front of you and generate the tokens yourself. The first is convenient; the second is educational. When the model is in your RAM, every abstraction the API hid from you becomes concrete: how much memory the weights need, why the first token is slow but the rest are fast, how batch size trades throughput against latency, and exactly where your data goes — which is nowhere, because it never left the box.
The memory wall, and why quantization exists
The blunt constraint is memory. A model's weights are numbers, and in native 16-bit precision each weight costs two bytes. A 7-billion-parameter model is therefore roughly 14 GB just to hold still — before any activations, KV cache, or headroom. A 70B model is ~140 GB, which is several high-end GPUs. Most people don't have that. Quantization is the answer: store each weight in fewer bits. Drop from 16-bit to 4-bit and the same 7B model shrinks from ~14 GB to ~4 GB — small enough to sit in a consumer GPU, or even in system RAM on a laptop.
The obvious worry is that throwing away bits throws away accuracy. The surprising, load-bearing result of the last few years is that — done carefully — it mostly doesn't. Two papers define the modern approach. Dettmers et al.'s LLM.int8() showed that transformer activations contain a few systematic "outlier" features that carry most of the numerical weight; isolate those in higher precision and quantize the rest to 8-bit, and a 175B model loads in half the memory with no measured accuracy loss. Frantar et al.'s GPTQ pushed further — a one-shot, second-order method that quantizes those same giant models to 3–4 bits per weight in about four GPU-hours with negligible degradation, enough to run a 175B model on a single GPU for the first time.
What it looks like in practice
You rarely implement this yourself. The llama.cpp project turned these ideas into a plain C/C++ inference engine that runs quantized models on CPUs, consumer GPUs, and Apple Silicon with no Python and no framework. Its GGUF format ships weights pre-quantized at 2- through 8-bit. The naming tells you the trade-off directly:
Model: Llama-family 7B (base weights ≈ 7e9 params) format bits/weight file size fits on -------- ----------- --------- ----------------------- f16 16.0 ~13 GB 24 GB GPU / big RAM Q8_0 8.5 ~7 GB 8 GB GPU Q4_K_M ~4.8 ~4 GB laptop RAM, 6 GB GPU Q2_K ~2.6 ~2.6 GB fits, but quality drops
Pick Q4_K_M and a 7B model runs on a machine you already own. That single choice — how many bits — is the whole game in miniature: memory footprint, speed, and answer quality are all downstream of it.
So which should you use?
For production traffic at scale, the cloud usually wins on economics and elasticity — someone else amortizes the GPUs. But for learning, for privacy, and for iteration, local is unmatched. There is no per-token bill, so you experiment freely. No data leaves your control, which matters for anything sensitive. And you develop the intuition that API users never get: you have watched a model saturate your memory bandwidth, felt the difference between a 4-bit and an 8-bit answer, and learned that "the model is slow" almost always means "I'm memory-bound," not "the GPU is busy."
Why it matters: the same quantization that lets a 7B model run on your laptop is what lets a datacenter fit more models per GPU and serve inference cheaply — the frontier labs quantize aggressively for exactly this reason. Running a model locally is the cheapest possible on-ramp to understanding GPU-inference infrastructure: bits, bandwidth, and the KV cache are the whole job, whether the box is under your desk or in a rack of H100s.