RAG that works
Retrieval-augmented generation has a reputation problem. The demo takes an afternoon; the system that survives real questions takes a month. Almost all of the difference is in three places: how you cut the documents, how you search them, and whether you can measure the result.
Chunking is the whole game
A chunk is the unit you retrieve. Too small and it loses context; too large and it dilutes the signal. Fixed 512-token windows are a reasonable default for prose, but they are wrong for most real documents:
- Manuals want chunks that follow headings and keep a step list together.
- Contracts want clause boundaries, plus the definitions section glued to every clause.
- Code wants functions, not lines, and the docstring must travel with the body.
Two cheap improvements: keep a short summary of the parent section attached to each chunk, and store the document title in the chunk text. Both give the retriever something to match when the question uses words the chunk does not.
Hybrid retrieval, not just vectors
Embeddings are good at meaning and bad at identifiers. Ask for “error code 4312” and a dense retriever will happily return a paragraph about a different error that sounds similar. Lexical search — BM25 or a simple inverted index — nails the identifier but misses paraphrase.
The practical answer is to run both and merge the rankings. A weighted combination is enough; the sophistication of the fusion function matters less than having both signals at all.
Rerank, then compress
Retrieve thirty candidates, rerank them with a cross-encoder or a small model, keep the top four or five. That single stage typically improves answer quality more than switching to a larger generator, and it reduces the prompt size at the same time.
After reranking, drop sentences that do not contain query terms or their neighbours. Long chunks with one relevant sentence are how systems end up ignoring the right passage.
Evaluate retrieval separately
This is the step that is usually missing. If you only measure the final answer, you cannot tell whether a failure came from retrieval or from generation — and you will spend weeks tuning the wrong component.
Build a set of questions with the document that contains the answer. Measure two numbers:
- Recall@k — how often the right document is in the retrieved set.
- Answer accuracy given retrieval — how often the generator is correct when the right document is present.
If recall is low, fix chunking and search. If recall is high and answers are wrong, fix the prompt or the model. The two problems have nothing in common except that they both look like “RAG is bad”.
Failure modes worth designing for
- No answer exists. The system must say so. A retriever always returns something; the generator must be allowed to reject it.
- Conflicting documents. Prefer the newer source explicitly, and show both if the conflict is real.
- Stale index. A perfect pipeline over last year's documents is worse than no pipeline, because it is confidently wrong.
None of this is exotic. It is just work that does not fit in a demo.