Quantization, explained without the math
A language model is a very large pile of numbers. Quantization means storing those numbers with fewer bits each. That is the whole idea. The interesting questions are which numbers, how few bits, and what breaks first.
Why it works at all
Most weights are small and clustered near zero. Instead of storing each one with 16 bits of precision, you can group them and store a shared scale plus a small index into a grid of values. The model keeps the shape of its function while using a quarter of the memory.
This works because neural networks are tolerant: they were trained with noise, dropout and approximate arithmetic from the start. A small perturbation in the weights rarely changes the answer for ordinary text.
What degrades, and when
The failures are not random. They cluster in a few places:
- Long arithmetic. Multi-step calculations accumulate error. A 4-bit model that summarizes perfectly can still get a five-digit multiplication wrong.
- Rare tokens and languages. Low-resource languages have fewer examples per parameter, so the same absolute error hurts more.
- Strict formatting. JSON schemas and exact citation formats are where a small drift becomes a parse error.
- Very long generations. Errors compound over thousands of tokens; short answers hide the problem.
Notice what is not on the list: tone, fluency, basic reasoning, summarization. Those survive 4-bit quantization so well that most people cannot tell the difference in a blind test.
Why perplexity is a bad guide
Quantized models are usually compared with perplexity, a measure of how surprised the model is by a reference text. A 4-bit model often shows a perplexity increase in the low single digits — which sounds negligible and is, for text continuation. But perplexity averages over everything, so it dilutes exactly the failures that matter: the wrong digit, the invalid JSON, the invented citation.
If a number averages over your whole task distribution, it will hide the subset you care about.
Practical rules
- Use 4-bit unless you have a specific reason not to. The memory saving is 4× and the quality cost is usually small.
- Prefer a well-made quant from a known publisher over a self-made one; calibration data matters more than the algorithm name.
- Test on your own examples, especially the structured-output ones.
- Keep the full-precision model around for a reference run when something looks off — it is the fastest way to tell “the model is wrong” from “the quant broke it”.
Quantization is not a trick or a compromise. It is how these models are deployed. The skill is knowing which parts of your task are sensitive to it.