When a RAG pipeline produces a bad answer, the instinct is to look at the model. Was the prompt unclear? Was the model confused? Should we add more examples?
In the majority of cases, the model is fine. The problem is that the model was given the wrong context to work with. Retrieval failed -- not catastrophically, not with an error -- but quietly, in a way that left the model answering a question about content that didn't actually address what the user asked.
The three retrieval failure modes
RAG retrieval fails in three distinct ways. Understanding which failure mode you're dealing with changes the fix entirely.
Empty retrieval: The query returns no results, or results below the relevance threshold, and the system either presents an empty context to the model (which then confabulates) or falls back to a default response. This is the most visible failure -- you can usually detect it by tracking chunk_count == 0 in your retrieval spans. The causes are typically: a narrow index that doesn't cover the query domain, an embedding model mismatch between the index and the query encoder, or a query that's too specific to match any indexed chunk.
Wrong-domain retrieval: The query returns chunks, and the relevance scores look reasonable, but the chunks are about related topics rather than the specific question asked. This is the insidious failure mode because it's invisible from the outside -- the retrieval step returns results, the context isn't empty, and the model produces a response. The response just doesn't actually answer the question from relevant material.
Stale retrieval: The chunks returned are accurate as of when they were indexed but are no longer current. In any domain where underlying facts change -- product information, pricing, company data, support knowledge -- stale retrieval produces responses that are confidently wrong.
Detecting retrieval failures in production
Empty retrieval is detectable with a single boolean attribute on your retrieval span: retrieval_empty: true when chunk_count == 0. Alert when this rate exceeds your baseline. A sudden increase in empty retrieval usually means an index issue or a shift in query distribution.
Wrong-domain retrieval is harder to detect automatically. The most useful proxy is the top relevance score distribution. If your typical top-result relevance score for a query class is around 0.78-0.84 and you see it drop to 0.55-0.62, retrieval quality has degraded -- you're getting related content rather than directly relevant content. Track the 5th percentile of top relevance scores as a trailing metric; drops here are early signals of index quality degradation.
A second proxy for wrong-domain retrieval is context diversity: if multiple retrieved chunks are from the same source document or have near-identical content, the retrieval returned depth in one narrow area rather than breadth. Track source diversity across retrieved chunks.
Stale retrieval requires either a freshness timestamp on each indexed document (and an alert when the freshness distribution shifts) or periodic ground-truth evaluation on known queries. The former is infrastructure; the latter is process. Both are necessary in domains where knowledge changes.
What changes in response to retrieval failures
The fixes for retrieval failures are categorically different from prompt engineering fixes. Understanding this matters because they require different owners and different timelines.
Empty retrieval fixes are usually index coverage improvements: adding documentation, reindexing with a better chunking strategy, broadening the relevant domain. These are data operations, not model operations.
Wrong-domain retrieval fixes are usually chunking and embedding improvements: better chunk boundaries, a higher-quality embedding model, a hybrid search approach that combines semantic similarity with keyword matching for specific query types. These are index architecture decisions.
Stale retrieval fixes are data freshness operations: incremental reindexing, TTL policies on indexed documents, monitoring upstream data sources for changes. These are data pipeline decisions.
The model can be tuned to be more cautious about answering from insufficient context, and that's worth doing. But it doesn't fix the underlying retrieval problem -- it just produces "I don't know" responses more often instead of wrong responses. The actual fix is always in the retrieval layer.
Building a retrieval health view
A minimum viable retrieval health view has four metrics tracked over time:
- Empty retrieval rate -- the fraction of queries returning 0 useful chunks
- Median top relevance score -- aggregate signal for retrieval quality across all queries
- Retrieval latency p95 -- the tail latency of your retrieval step, which dominates pipeline latency when slow
- Chunk count distribution -- how many chunks are being returned per query; spikes may indicate a configuration change
These four numbers, tracked as time series, give you enough signal to catch most retrieval degradations before they become sustained quality problems for users. The alternative -- discovering them from user reports -- is slower by several hours and produces no data to help diagnose the cause.