Latency, throughput, and the numbers that actually matter
“It does 80 tokens per second” is the most quoted and least useful number in local inference. It describes a benchmark, not your experience. What you feel is time to first token and time to finish, and both depend on things the headline number hides.
Three numbers, three different problems
- Time to first token (TTFT). How long after pressing enter the model starts answering. On a long prompt this is dominated by prefill — reading the input — and can be seconds even on fast hardware.
- Generation speed. Tokens per second once it starts. This is the number everyone quotes, and the one that matters least for short answers.
- Total time. TTFT plus generation multiplied by length. A chat reply is 20 tokens; a refactor is 2000. Same speed, completely different experience.
Why the headline number lies
A model that generates 80 tokens per second but takes four seconds to read a thirty-page document feels slower than one that generates 40 but starts in half a second. For chat, prefill dominates. For code generation, generation dominates. Same model, different verdict depending on the task.
Batching makes it worse. Serving several users in parallel improves throughput — total tokens per second across everyone — while increasing each individual request’s latency. A single-user benchmark will never show this.
What to measure for your workload
Take ten real requests at the length you actually use. Measure:
- p50 and p95 of TTFT;
- p50 and p95 of total time;
- tokens per second after the first token;
- memory peak, because swapping destroys all of the above.
Then compare against the threshold where the tool stops being used. For autocomplete it is roughly 200 ms. For chat, two seconds. For a batch job, minutes do not matter at all.
Two cheap wins
Cache the prompt. If your system prompt or document is reused, caching prefill removes most of TTFT. Both local runtimes and hosted APIs support some form of it, and it is usually free to try.
Cut the output. Asking for less is the most reliable optimisation there is: a shorter answer is faster, cheaper and often better. “Explain in three bullets” beats any inference tuning.
Rule of thumb: optimise for the metric your users complain about. If nobody complains, you are probably measuring the wrong thing.