Local or cloud: a decision that ages well
The local-versus-cloud question is usually argued with benchmarks. It is better argued with constraints: how private the data is, how predictable the load is, and how much latency you can afford.
A four-question test
- Does the data leave the building? If it cannot, local is the only option, regardless of quality. If it can, continue.
- Is the task stable? A task with a fixed format and a thousand examples per day is a good candidate for a small local model. A task that changes weekly benefits from a frontier model you did not have to fine-tune.
- What is the latency budget? Sub-200 ms favours local. Seconds are fine remotely.
- What is the volume? At low volume, cloud is almost always cheaper; at high steady volume, hardware amortises.
The break-even that surprises people
A 24 GB card and a small server cost roughly the same as a few months of moderate API usage. But the card only wins if it is busy. At 10% utilisation you are paying for idle silicon; at 70% the same hardware is dramatically cheaper per token than any API.
This is why “local is cheaper” and “cloud is cheaper” are both true, depending on a number most teams never measure: actual utilisation.
Keep the choice reversible
The expensive mistake is not picking wrong — it is building something you cannot move. Put the model behind a small interface with one method, keep prompts and evaluation data outside it, and routing becomes a config change.
Then you can start on the API, measure, and move the easy 80% of traffic to local hardware when the volume justifies it. Or the reverse, if the local model stops keeping up.
Most teams end up hybrid: a small local model for bulk and private work, a frontier model behind a router for the hard cases. That is not a compromise — it is the cheapest configuration that keeps quality.