Five question shapes retrieval cannot serve, what to build instead for each, and the scoping test that disqualifies a project before anyone estimates it.
The Moweb RAG Fit Test is a disqualification check that identifies questions retrieval-augmented generation cannot answer, whatever the quality of the implementation. Developed at Moweb Technologies, it names five question shapes where fetching the most similar passages cannot in principle contain what the question needs: questions requiring aggregation across an entire corpus; questions whose answer depends on the newest possible state of a system; questions requiring reasoning across a chain of documents; questions with one correct answer that must never be approximated; and questions the documents do not actually answer. A project failing any of the five needs a different architecture, not a better retriever.
Retrieval is the default proposal, including where it cannot work.
Ask most suppliers about enterprise AI and the answer is a retrieval-augmented assistant over your documents. It is a genuinely good pattern, it is what we build most often, and it is proposed for a large number of problems it structurally cannot solve.
The failure is rarely subtle. The system answers anyway, fluently, with a citation attached, and the answer is wrong in a way that takes a domain expert to notice. Below are the five shapes where that happens by construction rather than by accident.
Retrieval-augmented generation: answers are grounded in the client's own material, then verified before they are shown. The evaluation set and production monitoring are what make changes measurable rather than anecdotal.
The five shapes
the answer Where retrieval cannot produce.
Not cases where it is harder. Cases where fetching the most similar passages cannot, in principle, contain what the question needs.
01
Questions that require aggregation across the whole corpus
“How many contracts renew next quarter?” “What is our average payment term across all suppliers?”
Why retrieval fails. Retrieval returns the passages most similar to the question. It does not count, sum, or scan everything. A question whose answer depends on every document cannot be served by fetching the top handful, and the model will answer anyway from whatever it received, which is worse than failing.
Build instead. Extract the fields into a database and query them. This is a structured data problem wearing natural language, and the language layer belongs on top of a query, not on top of a search.
02
Questions whose answer depends on the newest possible state
“What is the current stock level?” “Is this customer past due right now?”
Why retrieval fails. A vector index is a snapshot. Between re-indexing runs it is confidently out of date, and there is no signal in the answer to say so. For anything that changes minute to minute, the index is a stale copy of a system that already has the correct answer.
Build instead. Call the system of record directly through a tool or API. Use retrieval for the parts that explain policy and the live call for the parts that state fact.
03
Questions that require reasoning over a chain of documents
“Does this clause conflict with the amendment we signed in 2019?”
Why retrieval fails. Similarity search finds documents that resemble the question. It does not follow a reference from one document to another, so the amendment that changes the answer is retrieved only if it happens to look like the question - and an amendment usually does not.
Build instead. Model the relationships explicitly, so the system traverses from a contract to its amendments rather than hoping both surface in one search. Retrieval can then work over a resolved set.
04
Questions with one correct answer that must never be approximated
“What dose is this patient on?” “What is the penalty clause?”
Why retrieval fails. A retrieval system that is nearly always right is a very different product from one that is always right, and the failure mode is a confident, fluent, wrong answer with a plausible citation attached. Fluency makes the error harder to catch, not easier.
Build instead. Return the source passage itself as the answer rather than a generated summary of it, and put the human in the loop by design. Where the answer must be exact, retrieval is a search tool, not an answering tool.
05
Questions the documents do not actually answer
“Why did we lose that renewal?” “Should we extend this credit line?”
Why retrieval fails. The most common failure, and it is a scoping failure rather than a technical one. The knowledge lives in judgement, in conversations, or in nobody's head at all. Retrieval will still produce something, assembled from documents that share vocabulary with the question.
Build instead. Nothing, usually. This is the point at which the honest answer is that software is not the constraint, and the money is better spent on the process that would capture the reasoning in the first place.
The scoping test
Before any technical conversation, write down twenty questions your users would genuinely ask and sort them into the five shapes above. If more than about a third land there, this is not a retrieval project. Building one anyway produces a system that answers a minority of questions well and the rest confidently wrong, which is the version that gets switched off six months after launch.
Including when it is the right answer, and what actually drives the running cost.
Write down twenty questions your users would actually ask, before any technical discussion. Sort them into the five shapes above. If more than roughly a third fall into them, this is not a retrieval project, and building one will produce a system that answers a minority of questions well and the rest confidently wrong. That test costs an afternoon and it is the single most useful thing to do before scoping.
Tell us what your users would actually ask and we will tell you which shapes they fall into, and whether retrieval is the right thing to build - including when the answer is that it is not.
