Why Your RAG System Gives Wrong Answers: The Retrieval Blame Test for Finding the Real Fault

A retrieval-augmented assistant gives a confident, wrong answer to a question your team has asked a hundred times. Within a day the verdict inside the company is "the model is hallucinating", and the proposed fixes follow: a bigger model, a stricter system prompt, a different vendor.

Often none of those help, because the model was never the problem. It answered the question it was given, using the text it was handed, and the text was wrong, incomplete, out of date or missing. In a retrieval-augmented system the model can only be as right as the context in front of it.

This guide is for CTOs, product owners and AI leads running a RAG system that is not yet trustworthy. It gives you a repeatable way to locate the actual fault for each bad answer before spending money on a fix: the Retrieval Blame Test.

Why "the model hallucinated" is usually the wrong diagnosis

The original retrieval-augmented generation design paired a language model with a retriever so the model could draw on an external document store rather than only what it memorised in training (Lewis et al., 2020). That split is the whole point of RAG, and it is also why diagnosis is hard. A wrong answer can come from either half, and from the output alone the two failures look identical.

Three mechanisms push retrieval failures out of sight.

**The model is fluent whatever it receives.** Give a capable model a paragraph about last year's refund policy and it will write a polished answer about last year's refund policy. Nothing in the output signals that the right paragraph was never retrieved.

**Teams test the model, not the pipeline.** Prompt tweaks are fast and visible, so they get tried first. Changes to parsing, chunking or indexing need a re-ingest and an evaluation set, so they get deferred.

**Retrieval quality is rarely logged.** Many production systems record the question and the answer but not which chunks were retrieved, their scores or their source versions. Without that trace, every failure defaults to the last component anyone can see.

The fix is not a better guess. It is a fixed order of checks, run on real failures, that tells you which layer broke.

The Retrieval Blame Test

For every wrong or weak answer, run six checks in order. Stop at the first one that fails: that is where the fault lives, and fixing anything downstream of it is wasted effort.

1. Exists: is the correct answer in the corpus at all?

   Search the source documents by hand. If the answer is not written down anywhere the system can index, no retriever or model will find it. This is a content gap, and the fix is ownership of the source, not engineering. The knowledge often lives in someone's inbox or a policy update nobody uploaded.

2. Extracted: did ingestion preserve it?

   The answer may exist in a PDF yet never reach the index in usable form. Tables flattened into a stream of numbers, scanned pages with no text layer, headers and footers merged into body text, and multi-column layouts read across the columns are all classic causes. Open the indexed text for that document and read it. If a human cannot find the answer in the extracted text, the model cannot either.

3. Retrieved: was the right chunk in the candidate set?

   Re-run the query and inspect the retrieved candidates. If the correct chunk is not among them, the failure is recall. Typical causes are chunks cut mid-thought, chunks that lost their surrounding context, vocabulary mismatch between how users ask and how documents are written, and pure vector search missing exact identifiers such as part numbers or clause references. Anthropic's published experiments on this problem are useful here: prepending chunk-specific context to embeddings, combining them with BM25 keyword search and adding a reranker cut their top-20 retrieval failure rate by 35%, 49% and 67% respectively on their test sets (Anthropic, Contextual Retrieval, 2024). The figures are specific to their data, but the mechanism generalises: a chunk that does not say which product or period it describes cannot be matched to a question that does.

4. Ranked: did it survive into the prompt, in a usable position?

   A reranker can push it down, a context budget can truncate it, and a deduplication step can drop it in favour of a near-duplicate from an older document. Position matters too. Research on long contexts found that models tend to use information at the beginning or end of the input more reliably than information in the middle (Liu et al., "Lost in the Middle", 2023). Check the final assembled prompt, not just the retriever output.

5. Current: was it the right version, for this user?

   The chunk that reached the model may be accurate for a superseded policy, a different region or a product line the user does not own. Stale versions left in the index, missing effective dates and absent metadata filters all produce answers that are correct for someone else. A related failure is permissions: a system that filters out documents the user may not see can also, if misconfigured, filter out the one that answers the question. Version and access metadata are design decisions, and in a regulated or ISO 27001-governed environment like the one described on our security page, they also decide what the system is allowed to show.

6. Used: given the right context, did the model answer faithfully?

   Only now is the model the suspect. Give it the correct, current chunk in a clean position and see whether it still answers wrongly, adds claims the context does not support or ignores an instruction to say it does not know. The evaluation literature calls this faithfulness, measured separately from whether the retrieved context was relevant in the first place (Es et al., RAGAS, 2023). If the model fails here, prompt structure, citation requirements, refusal behaviour and model choice are legitimate levers.

How to run the test on your own system this week

The test is only useful on real failures, so start with them.

• **Collect thirty to fifty bad answers.** Pull them from user feedback, support escalations and internal testers. Include the exact question and the answer given.
• **Write down the correct answer and its source.** For each one, a subject-matter expert names the document and passage that should have been used. This becomes your first evaluation set, and it is the asset most teams are missing.
• **Turn on tracing.** Log retrieved chunk IDs, scores, document versions and the final prompt for every query. Without this, checks three to five are guesswork.
• **Run the six checks in order and record the first failure.** One label per bad answer: Exists, Extracted, Retrieved, Ranked, Current or Used.
• **Count the labels.** The distribution is your roadmap.

The count is the point. If most failures land in Extracted, the next sprint is document parsing, not prompt engineering. If most land in Retrieved, it is chunking, hybrid search and query handling. If most land in Used, model and prompt work finally earn their budget. Skip the count and teams fix whatever is easiest to change, which is rarely what is broken.

What each failure costs to fix

The six labels also map to very different kinds of work, which matters when you plan budget and staffing.

**Exists** failures are cheap in engineering terms and expensive in organisational terms. Someone has to own the missing content, and that is a business process decision.

**Extracted** failures are a data engineering job: better parsers, table-aware extraction, OCR where needed and a re-ingest of the affected collections.

**Retrieved** and **Ranked** failures are search engineering: chunking strategy, contextual enrichment, hybrid keyword and vector retrieval, reranking, and query rewriting. This is where an evaluation set pays for itself, because each change can be measured instead of argued about.

**Current** failures are information architecture: version metadata, effective dates, region and product tags, and access filters that reflect how your organisation actually grants access. They often touch identity and document management systems, so they need integration work.

**Used** failures are model work: prompt structure, answer formats that require citations, explicit "not in the provided documents" behaviour, and sometimes a model change. It is the most popular fix and the last check in the order.

If your diagnosis points to a pipeline that needs rebuilding rather than tuning, our AI and ML development services team builds retrieval systems with evaluation and tracing designed in from the first sprint, and our enterprise AI practice covers the governance and access-control side.

When the test says RAG is the wrong tool

Sometimes the distribution reveals something larger. If many failures are questions that need calculation across records, aggregation over a database or a live system state, retrieval of text passages is the wrong mechanism, and no amount of chunk tuning will fix it. Those questions belong to structured queries, tool calls or a conventional report. We cover that scoping decision in detail in when RAG is the wrong answer, and there is a short self-assessment on our When RAG Is Wrong page.

The opposite also happens: a team convinced it needs a new architecture finds most failures are Extracted, meaning the system is sound and the PDFs were being read badly. That is a far smaller project. For the broader architecture choices behind a production system, see our guide to enterprise RAG development.

Questions buyers ask about fixing RAG accuracy

Will a bigger or newer model fix our wrong answers?

Only for failures labelled Used. For the other five, a stronger model receives the same missing, garbled or outdated context and produces a more articulate wrong answer.

How large does the evaluation set need to be?

Large enough to show a distribution and small enough that experts will actually write the correct answers. Thirty to fifty real failures is a practical starting point. Grow it every time a new failure is reported, and rerun it after every pipeline change.

Can we automate the checks?

Partly. Checks three, four and six can be scored automatically once you have an evaluation set with known source passages and full tracing. Checks one, two and five usually need a human the first time, because they depend on knowing what the documents should say. After that, regressions can be caught automatically.

How long does a diagnosis take?

With tracing already in place, a focused team can label a few dozen failures in days rather than weeks. Without tracing, the first task is adding it, which also makes every future incident diagnosable.

Do we need to rebuild the system?

Often not. A system whose failures cluster in one or two layers needs targeted work there, not a rebuild. The test exists to show which layers, so you avoid rebuilding the parts that already work.

Run the Retrieval Blame Test with us

If your RAG assistant is giving answers your team no longer trusts, send us a sample of the questions it gets wrong. A senior engineer from our applied AI team will run the Retrieval Blame Test with you, label where each failure originates and tell you which layer to fix first, along with a realistic view of the effort involved. If the answer is that a small parsing or metadata fix will do, we will say so.

Book a RAG diagnosis conversation and bring your worst five answers. You can also see the kind of systems we build on our case studies page.