Using a long context without drowning
Context windows are now measured in hundreds of thousands of tokens, and vendors compete on the number as if it were horsepower. In practice, a large window is a tool with a sharp edge: it removes the need to choose what to include, and that is exactly the problem.
What it is good for
Long context shines when the answer depends on a small detail inside a large, coherent document: a clause in a contract, a line in a log file, a paragraph in a specification. Reading the whole thing is more reliable than guessing which chunk to retrieve.
It is also excellent for one-shot comparisons: “here are three versions of this file, what changed and which is safer?” Retrieval cannot do that, because the value is in the relation between the parts.
What it costs
- Money. You pay for every token on every call. Re-sending a 200-page manual per request is the fastest way to a surprising invoice.
- Latency. Prefill grows with input; a huge prompt means a slow first token.
- Attention quality. Models still lose things in the middle of a very long input. “It fits” is not “it reads it”.
- Discipline. When everything fits, nobody prunes. Prompts become folders.
Three rules that keep it sane
1. Put the question at the end. Models attend more reliably to the beginning and the end. A long document followed by the question works better than the reverse.
2. Label the sections. Headings and short identifiers make the document navigable for the model, not just for you. “Section 4: termination” is worth more than three blank lines.
3. Ask for citations. Requiring a quote or section number forces the model to actually use the document, and makes the failure mode visible: if it cannot cite, it is likely guessing.
When to go back to retrieval
If the corpus changes often, if it is much larger than the window, or if the same content is queried many times, retrieval plus caching is cheaper and faster. The rule I use: long context for reading, retrieval for scale.
The best prompt is the one you would be comfortable reading yourself. If it is a wall of unlabelled text, the model’s job is harder than it needs to be.