Running a capable model at home
Local inference is no longer a curiosity. A mid-range GPU with 12 GB of memory can run a model that answers questions, summarizes documents and writes decent code. The interesting part is knowing what “capable” means at each memory tier, and where the effort stops paying off.
Start with the memory bill
Parameters are not the whole story, but they set the floor. At 4-bit quantization a model needs roughly
0.55 GB per billion parameters for the weights, plus 1–3 GB for the KV cache depending on
context length and batch size.
- 8 GB — 7–9B models at 4-bit. Fine for summarization, extraction and chat.
- 12 GB — 12–14B at 4-bit, or 8B with a long context.
- 24 GB — 27–32B at 4-bit. This is where coding assistants become genuinely usable.
- 48 GB and up — 70B-class models, or a 32B with a large batch for serving several users.
If the model does not fit, the runtime spills layers to system RAM. It still works, but throughput drops by an order of magnitude — often from 30 tokens per second to 3. That is the difference between a tool and a demo.
Quantization is the main lever
Nearly everything you can download is quantized. The useful formats in practice are GGUF for CPU and mixed CPU/GPU setups, and AWQ or GPTQ for pure GPU inference. A well-made 4-bit quant loses surprisingly little: on most tasks it is within noise of the half-precision model, and the failures show up in long chains of arithmetic and in rare languages rather than in everyday writing.
Below 4 bits the story changes. 3-bit and 2-bit variants exist and run, but the quality drop is visible: the model starts repeating itself, loses instruction adherence, and becomes unreliable with structured output. If you need 2-bit to fit, you probably need a smaller model instead.
Throughput: what to expect
On a modern consumer card, 8B at 4-bit generates roughly 40–80 tokens per second; 32B lands around 12–25. Prompt processing is much faster but happens in one burst, so the first token after a long document can take a second or two.
Two details matter more than the peak number:
- Context length costs memory and speed. Doubling the window is not free; the KV cache grows linearly and attention slows down.
- Batching beats parallelism for serving. If several people use the model, one process with a small batch is more efficient than one process per user.
When local actually wins
Local inference wins on privacy, on predictable cost, and on tasks where latency to a remote API matters. It loses on raw capability: a frontier model with a reasoning budget will beat a 32B local model on hard problems, and no amount of prompt engineering closes that gap.
A sensible split: local for bulk processing, private documents and always-on background jobs; remote for the hard cases. Routing between the two is a dozen lines of code and saves both money and embarrassment.