AI Learn Home
Local inference

Running a capable model at home

3 November 2026 · 7 min read

Local inference is no longer a curiosity. A mid-range GPU with 12 GB of memory can run a model that answers questions, summarizes documents and writes decent code. The interesting part is knowing what “capable” means at each memory tier, and where the effort stops paying off.

Start with the memory bill

Parameters are not the whole story, but they set the floor. At 4-bit quantization a model needs roughly 0.55 GB per billion parameters for the weights, plus 1–3 GB for the KV cache depending on context length and batch size.

If the model does not fit, the runtime spills layers to system RAM. It still works, but throughput drops by an order of magnitude — often from 30 tokens per second to 3. That is the difference between a tool and a demo.

Quantization is the main lever

Nearly everything you can download is quantized. The useful formats in practice are GGUF for CPU and mixed CPU/GPU setups, and AWQ or GPTQ for pure GPU inference. A well-made 4-bit quant loses surprisingly little: on most tasks it is within noise of the half-precision model, and the failures show up in long chains of arithmetic and in rare languages rather than in everyday writing.

Below 4 bits the story changes. 3-bit and 2-bit variants exist and run, but the quality drop is visible: the model starts repeating itself, loses instruction adherence, and becomes unreliable with structured output. If you need 2-bit to fit, you probably need a smaller model instead.

Throughput: what to expect

On a modern consumer card, 8B at 4-bit generates roughly 40–80 tokens per second; 32B lands around 12–25. Prompt processing is much faster but happens in one burst, so the first token after a long document can take a second or two.

Two details matter more than the peak number:

When local actually wins

Local inference wins on privacy, on predictable cost, and on tasks where latency to a remote API matters. It loses on raw capability: a frontier model with a reasoning budget will beat a 32B local model on hard problems, and no amount of prompt engineering closes that gap.

A sensible split: local for bulk processing, private documents and always-on background jobs; remote for the hard cases. Routing between the two is a dozen lines of code and saves both money and embarrassment.