AI Learn Home
Retrieval

RAG that works

8 October 2026 · 8 min read

Retrieval-augmented generation has a reputation problem. The demo takes an afternoon; the system that survives real questions takes a month. Almost all of the difference is in three places: how you cut the documents, how you search them, and whether you can measure the result.

Chunking is the whole game

A chunk is the unit you retrieve. Too small and it loses context; too large and it dilutes the signal. Fixed 512-token windows are a reasonable default for prose, but they are wrong for most real documents:

Two cheap improvements: keep a short summary of the parent section attached to each chunk, and store the document title in the chunk text. Both give the retriever something to match when the question uses words the chunk does not.

Hybrid retrieval, not just vectors

Embeddings are good at meaning and bad at identifiers. Ask for “error code 4312” and a dense retriever will happily return a paragraph about a different error that sounds similar. Lexical search — BM25 or a simple inverted index — nails the identifier but misses paraphrase.

The practical answer is to run both and merge the rankings. A weighted combination is enough; the sophistication of the fusion function matters less than having both signals at all.

Rerank, then compress

Retrieve thirty candidates, rerank them with a cross-encoder or a small model, keep the top four or five. That single stage typically improves answer quality more than switching to a larger generator, and it reduces the prompt size at the same time.

After reranking, drop sentences that do not contain query terms or their neighbours. Long chunks with one relevant sentence are how systems end up ignoring the right passage.

Evaluate retrieval separately

This is the step that is usually missing. If you only measure the final answer, you cannot tell whether a failure came from retrieval or from generation — and you will spend weeks tuning the wrong component.

Build a set of questions with the document that contains the answer. Measure two numbers:

If recall is low, fix chunking and search. If recall is high and answers are wrong, fix the prompt or the model. The two problems have nothing in common except that they both look like “RAG is bad”.

Failure modes worth designing for

None of this is exotic. It is just work that does not fit in a demo.