When RAG Is the Wrong Answer: 5 Retrieval Limits

Retrieval-augmented generation has become the default proposal for almost every enterprise AI request that involves a company's own documents. Often that is the right call, and we have written a full decision guide to enterprise RAG development for the cases where it is.

This article argues the other side. There is a class of question that retrieval structurally cannot answer, and no amount of chunking strategy, re-ranking, or prompt tuning will fix it, because the limitation is in the shape of the question rather than the quality of the implementation. Projects that discover this after the build are the expensive ones. They are usually recoverable, but the recovery is a different architecture, not a tuning pass.

Below are the five question shapes that should make you stop and reconsider, what to build instead in each case, and a scoping rule you can apply before anyone writes code.

Why retrieval has a ceiling

It helps to be precise about what a RAG system does. When a question arrives, the system embeds it, compares it against an index, and pulls back the passages that look most similar. Those passages go into the prompt as grounding, and the model composes an answer from them. The original formulation combined a retriever with a generator so that outputs could be traced back to source documents (Lewis et al., 2020).

Notice what that pipeline assumes: that the answer exists inside a small number of passages, and that similarity is a good way to find them. Both assumptions hold for a large share of enterprise questions. Neither holds universally, and the top-k retrieval step is where they break. A retriever returns the *k* most similar chunks. It does not return all relevant chunks, it has no concept of having read everything, and it cannot tell you that it found nothing because nothing exists rather than because the embedding missed.

Every failure below is a consequence of that one property.

The five question shapes

1. Aggregation and counting

   *"How many of our supplier contracts come up for renewal in Q3?"*

   A retriever will happily return ten contracts that mention renewal dates. The model will then count them and report a number, and that number will be confidently wrong, because the system counted its retrieval window rather than the corpus. The user has no way to see the difference. This is the most dangerous shape on the list precisely because the output looks like an answer rather than an error.

   **Build instead:** extract the renewal date into a structured field at ingestion time, then answer the question with a query rather than a retrieval. Once the field exists, a text-to-SQL or semantic-layer approach lets a model translate the question into a query that touches every row. The research problem of mapping natural language onto database queries is well-studied and benchmarked in its own right (Yu et al., 2018).

2. Absence and negation

   *"Which of our vendor agreements have no indemnity clause?"*

   You cannot retrieve evidence of absence. Similarity search finds passages that resemble the query, and a contract missing an indemnity clause contains nothing that resembles one. The strongest matches will be agreements that discuss indemnity at length, which is the exact inverse of the answer.

   **Build instead:** invert the problem. Run a per-document extraction pass that asks, for every document, whether the clause is present, and store the result. The question then becomes a filter over a table. The expensive part is that this is O(corpus) rather than O(k), so it belongs in a batch pipeline rather than a request path.

3. Whole-corpus comparison and ranking

   *"Which of our master service agreements has the most permissive liability cap?"*

   Superlatives require reading everything. Retrieval reads a sample. A system asked to rank across a corpus it has only partially seen will produce a ranking of the passages it happened to retrieve, presented as a ranking of the corpus.

   **Build instead:** a map-reduce pattern. Extract the comparable attribute from every document in a batch pass, then rank the extracted values deterministically. The model does the reading; ordinary code does the comparing. Resist the temptation to let the model do both, because comparison is where the errors become invisible.

4. Computation and derivation

   *"If we shifted this contract to the new rate card, what happens to blended margin?"*

   Language models approximate arithmetic rather than performing it, and a retrieved passage containing the right inputs does not make the calculation reliable. Multi-step derivations compound the problem: a small error early is amplified rather than corrected.

   **Build instead:** give the model tools and let deterministic code compute. The model's job is to identify which calculation is being requested and to supply the arguments; the calculation itself runs in a function whose output can be unit-tested. This also gives you something a pure RAG system cannot offer, which is a reproducible audit trail from inputs to figure.

5. Point-in-time and temporal reasoning

   *"What did our refund policy say on the date this claim was filed?"*

   Most document stores hold the current version of a document. Ask what the policy said eighteen months ago and the system will confidently answer with today's policy, because today's policy is what the index contains. This is a data-modelling gap wearing the costume of a retrieval gap, and it surfaces most often in exactly the regulated contexts where being wrong is expensive.

   **Build instead:** model effective dates explicitly. If the business genuinely needs to reason about what was true at a past moment, that is a bitemporal data problem, and it needs to be solved in the schema before retrieval is layered on top. Timestamps should be stored in an unambiguous, offset-aware format (RFC 3339) rather than as free text inside the document body.

The scoping rule

Here is the rule we apply before quoting a retrieval project.

Write down the twenty questions the system is actually expected to answer. Not the demo questions, and not the questions the sponsor thinks are impressive, but the ones users will type in week one. Then sort each into one of two buckets: answerable from a handful of passages, or falling into one of the five shapes above.

**If more than roughly a third land in the second bucket, this is not a RAG project.** It is a data extraction and modelling project that may have a retrieval interface bolted onto the front once the structured layer exists. Scoping it as retrieval will produce a system that demos well and then fails in the specific, credibility-destroying way where it answers a counting question with a plausible number.

That rule disqualifies work we would otherwise be paid to do, which is the point of having it.

What this means for cost

The second bucket is not necessarily more expensive, but it is expensive in a different place. Retrieval costs scale with query volume and conversation length. Extraction costs scale with corpus size and are incurred largely up front, then again whenever the corpus changes materially. Teams that budget for the first shape and build the second are the ones that get surprised in year two.

Where the boundary actually sits

None of this is an argument against retrieval. Summarisation, policy lookup, troubleshooting against a knowledge base, drafting grounded in prior work, question answering where the answer genuinely lives in one or two passages: retrieval is the correct and cost-effective architecture for all of it, and it remains the fastest route from a private corpus to something useful.

The distinction worth internalising is that retrieval answers *"what does the corpus say about X"* extremely well, and answers *"what is true across the whole corpus"* poorly. Most enterprise question sets contain both. The mistake is not choosing retrieval; it is failing to notice that half the requirements were never retrieval questions, and discovering it after the architecture is fixed.

A useful early signal is the state of your source data. If the underlying content is unversioned, inconsistently structured, or missing the fields the hard questions depend on, the retrieval layer will inherit every one of those gaps. That work is worth doing on its own terms, which is the subject of our guide to building an AI-ready data strategy.

Frequently asked questions

Can a bigger context window solve this?

Partly, and only for small corpora. If the entire corpus fits in context, aggregation and comparison questions become tractable because the model genuinely sees everything. At enterprise corpus sizes this stops being viable on both cost and latency, and the failure returns in the same shape. Treat large context as a way to raise *k*, not as a way to change the class of question you can answer.

Is fine-tuning the alternative?

Rarely, for these five shapes. Fine-tuning changes how a model behaves, not what it can count or look up. A fine-tuned model asked how many contracts renew in Q3 has the same problem as a retrieval system: no reliable access to the complete set. Fine-tuning addresses tone, format, and task-specific behaviour, which is a different problem from the one described here.

How do we know we have this problem already?

Ask your system a counting question you know the answer to, then ask it again with the retrieval depth changed. If the answer moves, the system is reporting on its retrieval window rather than your data. This takes an afternoon and is worth running against any deployed assistant.

Does this apply to agentic systems too?

An agent that can call a query tool escapes several of these limits, because it can execute a deterministic query rather than relying on retrieval. That only works if the structured layer exists for it to query. Agents change how the answer is fetched; they do not create the underlying data model. We cover the orchestration side in our guide to multi-agent systems in the enterprise.

Scoping a retrieval project honestly

The cheapest hour in an enterprise AI programme is the one spent sorting real user questions into the two buckets above before committing to an architecture. It occasionally kills a project. More often it changes the sequence, putting extraction and data modelling first and retrieval second, which is the order that survives contact with production.

If you are weighing a retrieval build and want a second opinion on whether the question set actually fits, our AI and ML development team runs this scoping exercise as a fixed-scope engagement, and will tell you when the answer is that you do not need the system you asked us to quote. Start a conversation and bring your twenty questions.