What inference actually costs

Economics 14 November 2026·5 min read

Inference pricing looks simple: a number per million input tokens and a larger number per million output tokens. The bill is rarely simple. Four effects decide whether you pay hundreds or thousands for the same product.

1. Output dominates

Output tokens typically cost three to five times more than input. A model that writes two paragraphs where one would do is not “slightly more expensive” — it can double the bill. If your prompts do not specify length, they are silently choosing the expensive option.

2. Retries multiply everything

One request that fails validation and is retried costs twice: the failed attempt and the successful one. At a 15% retry rate you are paying 15% more for every feature, forever, and it never appears in a price comparison. Measure retries separately; it is the most under-reported cost in production systems.

3. Context is paid repeatedly

In a chat, every turn re-sends the history. A conversation that grows to 20k tokens costs 20k input tokens on the twentieth message. Caching helps — often dramatically — but only if the prefix is stable. Re-ordering the system prompt or injecting a timestamp at the top defeats it silently.

4. Reasoning budgets are invisible in advance

With a reasoning dial, the same request can cost one cent or twenty depending on how long the model thinks. Setting the budget per task — not globally — is the difference between a predictable invoice and a surprise.

A useful exercise: take last month’s logs and compute cost per solved task, including retries and abandoned sessions. The number is almost always higher than the estimate, and it is the only one worth optimising.

Where to cut first

  • Shorter outputs. Specify length and format explicitly.
  • Stable prefixes. Keep the cached part at the beginning and never reorder it.
  • Smaller models for easy cases. Route by difficulty; most traffic is easy.
  • Fewer tokens of context. Retrieval instead of a full document is often cheaper than the extra input tokens suggest, because it also shortens the output.

None of this requires a new model. It requires knowing which line on the invoice is growing.

← All articles