AI Learn Home
Fundamentals

Quantization, explained without the math

21 October 2026 · 5 min read

A language model is a very large pile of numbers. Quantization means storing those numbers with fewer bits each. That is the whole idea. The interesting questions are which numbers, how few bits, and what breaks first.

Why it works at all

Most weights are small and clustered near zero. Instead of storing each one with 16 bits of precision, you can group them and store a shared scale plus a small index into a grid of values. The model keeps the shape of its function while using a quarter of the memory.

This works because neural networks are tolerant: they were trained with noise, dropout and approximate arithmetic from the start. A small perturbation in the weights rarely changes the answer for ordinary text.

What degrades, and when

The failures are not random. They cluster in a few places:

Notice what is not on the list: tone, fluency, basic reasoning, summarization. Those survive 4-bit quantization so well that most people cannot tell the difference in a blind test.

Why perplexity is a bad guide

Quantized models are usually compared with perplexity, a measure of how surprised the model is by a reference text. A 4-bit model often shows a perplexity increase in the low single digits — which sounds negligible and is, for text continuation. But perplexity averages over everything, so it dilutes exactly the failures that matter: the wrong digit, the invalid JSON, the invented citation.

If a number averages over your whole task distribution, it will hide the subset you care about.

Practical rules

Quantization is not a trick or a compromise. It is how these models are deployed. The skill is knowing which parts of your task are sensitive to it.