RAG Isn’t a Silver Bullet, But This Setup Works Often

The problem RAG actually solves

Retrieval-Augmented Generation is genuinely good at one thing: grounding a model's answer in your content instead of its training data. That is worth a great deal when the cost of a confident wrong answer is high and your knowledge changes faster than any fine-tune cycle. The trouble is that RAG has become a reflex. Teams reach for a vector store the moment anyone says "the model should know our docs," then spend the next two quarters fighting latency, cost, and answers that shift between runs because retrieval is noisy.

The discipline that separates reliable RAG from a science project is sequencing. You do not start with embeddings. You start by asking whether your questions can even be answered from your corpus, and you keep retrieval hybrid and lean rather than dumping every plausible passage into the context window. Done this way, RAG lands a first-pass answerability rate above 80 percent at a P95 latency under 2 seconds. Done as a reflex, teams routinely see answerability stall near 55 percent while token spend triples, because more context is not more accuracy; past a point it is more noise and more hallucination surface.

There is also a decision that most teams skip: whether to use RAG at all. RAG earns its complexity when knowledge is large, unstructured, and changes faster than a fine-tune cycle. It is the wrong tool in three common cases. For stable, narrow domains, a small fine-tuned or instruction-tuned model beats RAG on both latency and cost. For highly structured data, a SQL query or an API call through function-calling is more accurate than stuffing tables into a context window and hoping the model reads them correctly. And when the source content is outdated or self-contradictory, adding retrieval simply surfaces the contradictions faster; the fix is the content pipeline, not the vector store. Deciding this up front saves the quarter you would otherwise spend tuning a retrieval stack that never needed to exist.

The setup that works often

The reference setup is deliberately unglamorous. It leads with answerability measurement, uses hybrid retrieval so keyword recall and semantic recall cover each other, reranks a small candidate set rather than over-retrieving, and forces the prompt to cite sources and refuse gracefully when confidence is low. Each layer below has a specific job and a number to hold it to.

A worked example makes the discipline concrete. An internal IT help desk pointed a naive RAG assistant at 12,000 knowledge articles: pure embeddings, fixed 512-token chunks, top 15 passages stuffed into context. Answerability sat at 58 percent, P95 latency reached 3.4 seconds, and answers for acronym-heavy queries like "reset MFA on VPN" were frequently wrong because the embedding model paraphrased away the exact terms. The team rebuilt to the reference setup. Section-aware chunking cut fragment count by a third. Adding BM25 alongside embeddings recovered the acronym queries that pure vectors missed. Reranking 40 candidates down to 6 dropped context tokens by roughly 60 percent, which pulled P95 to 1.8 seconds and cut per-query cost by more than half. A "not in corpus" refusal path with a search-link fallback ended the confident wrong answers on unanswerable questions. Answerability rose to 84 percent. Nothing exotic changed; the sequencing did.

Build it in the right order

• Ship a 50 to 100 item golden set first and measure answerability before you provision a vector store. If the corpus cannot answer, no retrieval stack will save you.
• Attach source_id, doc_type, effective_date, and access_scope to every chunk, and enforce row-level access at retrieval time. Never filter permissions in the prompt.
• Run hybrid retrieval: BM25 for keyword recall plus embeddings for paraphrase, then rerank 20 to 40 candidates down to 5 to 8. Do not over-retrieve.
• Require the prompt to cite source_ids and to refuse gracefully when confidence is low, with few-shot examples of a correct "not in corpus" refusal and a helpful fallback.
• Set a 2-second P95 budget, profile each hop from network to vector store to rerank to model, and cache at both the retrieval and final-answer layers keyed by policy version.

Where RAG quietly fails

• Everything becomes RAG. For calculations or status lookups, stuffing tables into context is slow and wrong. Fix: call a tool or an API and generate prose from the result.
• Over-chunking into tiny fragments that lose context and raise hallucination risk. Fix: chunk by semantic section with titles and keep overlap under 15 percent.
• Embedding drift ignored. New model versions shift similarity geometry and quietly degrade recall. Fix: run canary queries daily and alert when results move.
• Prompt-only guardrails on access. Filtering permissions in the prompt leaks restricted content. Fix: enforce access at retrieval, before the model ever sees a passage.
• Shipping RAG on stable or highly structured data where a fine-tune or a SQL tool would win on latency and cost. Fix: apply the skip rules and reserve RAG for changing, unstructured knowledge.

Ship the reliable version

• 100-item golden set live; answerability tracked weekly against a pre-launch baseline.
• Hybrid BM25 plus embeddings with rerank; final context capped at 5 to 8 passages.
• Prompt enforces source citations and an explicit "not in corpus" refusal with fallback.
• 2-second P95 budget set; retrieval and answer caches keyed by policy version.
• Skip rules written down: fine-tune for stable narrow domains, tools for structured data, fix the pipeline before retrieval when sources are not trusted.