AI Incident Response: The Recall Test for Handling LLM Failures in Production

Every engineering team knows what to do when a server falls over. There is an alert, an on-call engineer, a runbook, a status page and a postmortem. Far fewer teams know what to do when the server is perfectly healthy and the AI feature running on it tells a customer something false, offensive or expensive.

That second kind of incident is now routine. An LLM-powered assistant quotes a refund policy that does not exist. A summarisation feature drops the one clause that mattered. An agent takes an action nobody approved. Uptime dashboards stay green throughout, because nothing crashed. The system did exactly what it was built to do, and the output was still wrong.

This guide is for CTOs, product leaders and risk owners running generative AI in production. It gives you a checkable standard for whether you could actually handle one of these incidents: the Recall Test.

Why AI incidents break normal incident response

Conventional incident response assumes failures are loud. A service returns errors, latency spikes, a queue backs up. Monitoring catches it, and the fix is usually a rollback.

AI failures are different in three ways.

**They are silent.** A wrong answer returns a 200 status code in normal time. Nothing in standard observability distinguishes a correct answer from a confident fabrication.

**They are probabilistic.** The same prompt can produce a good answer a hundred times and a bad one on the next attempt. "We could not reproduce it" is not evidence that it did not happen, and it is not evidence that it will not happen again.

**They are attributable to you.** The legal position is moving faster than most product roadmaps. In Moffatt v. Air Canada, a British Columbia tribunal held the airline liable for negligent misrepresentation after its website chatbot gave a customer incorrect information about bereavement fares, and rejected the idea that the chatbot was a separate entity responsible for its own words. As the tribunal put it, the chatbot "is still just a part of Air Canada's website" (Moffatt v. Air Canada, 2024 BCCRT 149, summarised by McCarthy Tétrault). Whatever your model says, your company said.

For some systems there is now a regulatory clock as well. Under Article 73 of the EU AI Act, providers of high-risk AI systems must report serious incidents to market surveillance authorities "not later than 15 days" after becoming aware of them, with shorter deadlines for the most severe cases (EU AI Act, Article 73). You cannot meet a fifteen-day deadline if it takes you three weeks to work out what happened.

The Recall Test

The name comes from product recalls. A manufacturer that discovers a defect has to answer a fixed set of questions fast: how did we find out, what caused it, which units are affected, how do we stop more going out, and how do we fix the ones already shipped. An AI system needs the same answers, and most cannot produce them on demand.

The Recall Test is five checks. Score each one as pass or fail against your live system today, not against your roadmap.

1. Detect: would you know within hours, not weeks?

   Most AI incidents are discovered by a customer, a journalist or a support agent who happens to notice. That is not detection; it is luck.

   Passing this check means you have at least one deliberate signal for bad outputs: user feedback captured against specific responses, automated evaluation of a sample of production traffic, guardrail triggers that are counted and alerted on, or escalation patterns in support tickets that point back to the AI feature. It also means someone is on call for those signals, not just for server health.

2. Trace: can you reconstruct exactly what happened?

   Given a single bad answer, can you show the full input, the system prompt version, the model and version that served it, any retrieved documents, any tool calls and their results, and the raw output before post-processing?

   If any of those are missing, you cannot tell whether the cause was the model, the prompt, the retrieval layer or a downstream integration. For retrieval-based systems, our guide to diagnosing wrong RAG answers walks through how often the fault turns out to be the documents rather than the model.

3. Scope: can you say who else received a bad answer?

   One reported failure is rarely the only one. Passing this check means you can query your logs for every interaction that shared the failing condition: the same prompt version, the same retrieved document, the same tool, the same customer segment, the same time window. Then you can say, with evidence, "this affected 40 conversations" or "this affected one".

   Without scope, every incident becomes either an overreaction (disable everything) or an underreaction (fix the one case and hope).

4. Contain: can you stop it in minutes without a deployment?

   A traditional rollback restores the previous code. An AI incident may need something narrower: turning off one tool, removing one document from the index, pinning a previous prompt version, routing one category of question to a human, or switching the feature off for one market.

   Passing this check means those controls exist as configuration, have been tested, and can be used by the on-call engineer without waiting for a release cycle. A kill switch that has never been flipped is a hypothesis.

5. Correct: can you fix the cause and prove it is fixed?

   Correction has two halves. The first is the technical fix: a prompt change, a corrected document, a new guardrail, a model change. The second is evidence. The failing case, and the variations found during scoping, should become permanent entries in an evaluation set that runs before every future release. Otherwise the same failure returns the next time someone edits the prompt.

   Correction also covers people. If customers acted on a wrong answer, someone has to decide whether to contact them, and that decision should not be invented under pressure.

Scoring your system honestly

Score one point per check, and be strict. "We could probably find it in the logs" fails Trace. "We would pull the feature" fails Contain if pulling the feature requires a release.

• **5 out of 5:** you can run an AI incident like any other production incident. Keep the evaluation set growing.
• **3 to 4:** you can handle a small incident but a public one would stretch you. Fix the failing checks before expanding the feature to more users.
• **0 to 2:** you are running a production AI system without an incident capability. That is a business risk, not just a technical one, and it should be on the risk register.

The most common pattern is passing Detect partially and failing Trace and Scope entirely, because logging was designed for debugging during development rather than for investigation after launch.

What it takes to pass each check

The engineering behind the Recall Test is not exotic, but it has to be designed in rather than bolted on after the first incident.

**For Detect:** feedback controls tied to response IDs, sampled automated evaluation of live traffic, and alerts on guardrail and refusal rates rather than only on errors.

**For Trace:** structured logging of every request with prompt version, model identifier, retrieved context and tool calls, retained for long enough to investigate and handled under the same access controls as other customer data. In an ISO 27001-governed environment like the one described on our security page, those logs are themselves sensitive records and need retention and access rules.

**For Scope:** logs that are queryable, not just stored. If the only way to search them is to grep compressed files, scoping will take days.

**For Contain:** feature flags at the level of tools, document collections, prompt versions and user segments, plus a documented fallback path to a human or a static response.

**For Correct:** an evaluation harness that runs automatically before release, with incident cases added as regression tests.

Our AI security and governance and MLOps work builds these controls alongside the feature itself, because retrofitting them after launch usually means re-plumbing the request path.

Writing the AI incident runbook

Once the controls exist, write them down. A useful AI incident runbook is short and specific.

• **Severity definitions that fit AI.** A wrong answer about store hours and a wrong answer about medication or money are not the same severity, even though both are "incorrect output".
• **Named owners.** Who decides to contain, who talks to customers, who talks to legal, and who decides whether a regulatory report is required.
• **The first hour.** Preserve the trace, run the scope query, apply the narrowest effective containment, and open a record with timestamps. Regulators and courts care about when you knew.
• **Communication templates.** Prepared wording for customers who received a wrong answer, so nobody drafts an apology at midnight.
• **Postmortem questions.** Which Recall Test check failed first, and what changes so it passes next time.

Run a tabletop exercise against it at least once before you need it. The first run always finds a missing permission or a log that was not retained.

Questions leaders ask about AI incident response

Do we need a separate process from our normal incident management?

No. You need AI-specific detection, tracing and containment controls feeding into the incident process you already have. Running a parallel process creates confusion about who is in charge.

Is this only relevant to regulated industries?

The regulatory deadlines apply to specific high-risk systems, but liability for what your AI tells customers does not depend on your sector. Any customer-facing AI feature benefits from passing the test.

How much logging is too much?

Log what you need to pass Trace and Scope, protect it like customer data, and set a retention period your legal and security teams agree with. Logging nothing to avoid privacy risk simply moves the risk to the day of the incident.

Can a vendor model provider handle this for us?

They can provide parts: model version information, content filters, usage logs. They cannot see your prompts, your documents, your tools or your customers. Most of the Recall Test sits in your application layer.

Should we pause our AI rollout until we pass all five checks?

Not necessarily. Many teams fix the failing checks in parallel with a limited rollout. What matters is knowing your score and not expanding exposure faster than your ability to handle a failure.

Run the Recall Test on your AI system with us

If you have a generative AI feature in production, or about to launch, we can run the Recall Test with your team. A senior engineer from our applied AI practice will walk through each check against your actual system, show you which ones fail and why, and outline the smallest set of changes that gets you to a defensible incident capability. If you already pass, we will tell you that too.

Talk to us about an AI incident readiness review. You can see the broader scope of our AI and ML development services and examples of our delivery on our case studies page.