RAG in Production: Retrieval That Does Not Invent Things

The pitch for RAG fits on a napkin: embed the documents, search them, paste the results into the prompt. That version works on a Saturday and falls apart in production. The hard part was never the retrieval call. It is getting the right context, in the right shape, in front of the model at the right moment.

Bad retrieval, confident nonsense

A language model can only reason over what you hand it. Feed it the wrong passages and it will build a fluent, well-organized, completely wrong answer on top of them. Most complaints that begin with “the AI hallucinated” are retrieval failures in costume. Fix what gets retrieved and a large share of them stop happening.

Chunking is a design decision, not a config value

How you split a document decides what can ever be found. Chunks that are too large bury the signal in noise. Chunks that are too small strip away the context that made the passage mean anything. Split along the document’s own structure — sections, headings, semantic boundaries — and overlap enough that a single idea is never cut down the middle. A table separated from its caption is a table nobody can use.

Search is a pipeline, not a call

Pure vector similarity carries a system a long way and then flattens out. The setups that hold up combine methods: keyword search to catch exact names, part numbers, and error codes; vector search for meaning; and a reranking pass that reorders the top candidates before anything reaches the model. Each stage fixes a failure the others cannot see.

• Run hybrid search — keyword and semantic, not one or the other
• Rerank the top candidates before they enter the prompt
• Attach metadata (source, date, section, permission) and filter on it
• Return citations so every claim maps back to a specific chunk

The failure modes nobody mentions

Real corpora rot. Documents go stale, near-duplicates multiply, and the page that ranks highest is often the one that was replaced last year. Without recency signals and deduplication, the system will cheerfully quote a policy that no longer exists — and cite it, which makes it worse. You need a way to retire content, not only a way to add it.

Permissions are a retrieval problem

The moment a corpus spans more than one team, retrieval has to respect who is asking. Filtering after generation is too late: the answer already contains what the user was not allowed to see. Access control belongs in the query, not in the output.

“RAG does not fail loudly. It fails by confidently retrieving yesterday’s truth — which is why measurement matters more than model choice.”

Measure it, then move it

You cannot improve retrieval you do not score. Write down a set of representative questions along with the passages that should surface for each, and track whether they do. Once recall and precision are numbers, every chunking tweak and reranker swap becomes an experiment with an answer instead of an argument. That baseline is the first thing we build before a RAG system takes real questions.