A model that starts leaking training data at 2am is not a normal outage, and your IT runbook will not catch it. AI incidents fail in ways classic monitoring never watches: silent drift, prompt injection, exfiltration through outputs, confident hallucination at scale. Every hour between a model misbehaving and a human confirming it, the affected users, the data footprint, and your regulatory exposure all compound at once. Here is the operating model that closes that gap, from the first anomaly signal to a defensible disclosure decision you can actually stand behind.
AI incidents fail quietly, then all at once
A traditional service outage announces itself. Latency spikes, error rates climb, a pager fires, and the on-call engineer knows within minutes that something is wrong. AI-specific incidents rarely behave that way. A recommendation model drifts three percent per week until, two months later, it is systematically steering a protected class toward worse offers. A support assistant absorbs a poisoned help article and begins confidently citing a refund policy that does not exist. A retrieval system quietly surfaces one customer's contract text inside another customer's answer. None of these trip a CPU alarm. The system is up, fast, and wrong, and the clock on your regulatory and reputational exposure has already been running.
The cost of that blindness is measurable. When the gap between a model starting to misbehave and a human confirming it stretches from minutes to days, three things compound at once: the volume of affected users, the size of the data footprint, and the number of downstream decisions made on bad output. A leakage event caught in one hour might touch 40 users; the same event caught in three days can touch 40,000. Under regimes like GDPR, a qualifying personal-data breach carries a 72-hour notification clock that starts when you become aware, and the EU AI Act adds serious-incident reporting duties for high-risk systems. An AI incident response capability is not a rebranded IT runbook. It is a distinct operating model built around failure modes that classic monitoring was never designed to watch.
Five stages from anomaly signal to disclosure decision
Treat AI incident response as five sequenced stages, each with an owner, an entry trigger, and a target time. The point is not speed for its own sake; it is to make the containment decision before the blast radius grows, and to make the disclosure decision on evidence rather than panic. Assign every AI system a severity tier at design time so the response scales to the stakes, and rehearse the flow quarterly against a live model, not a slide.
| Stage | What you are actually doing | Owner | Target time (Sev-1) |
|---|---|---|---|
| Detect | Catch drift, leakage, jailbreak, or failure via output monitors, embedding-distance alerts, canary prompts, and user reports | ML platform / on-call | Under 30 min to acknowledge |
| Triage | Confirm it is real, classify the failure mode, and set severity by data sensitivity and user reach | Incident commander | Under 1 hr to classify |
| Contain | Roll back to the last known-good version, gate the endpoint, or fail over to a human queue | ML platform | Under 2 hrs to stop the bleed |
| Investigate | Reconstruct inputs, prompt, model version, and outputs; scope who and what was affected | ML + security forensics | Under 24 hrs to scope |
| Disclose | Decide notification obligations to customers, regulators, and partners; issue the postmortem | Legal + comms + eng lead | Within statutory clock (e.g. 72 hrs) |
Worked example. A B2B analytics vendor runs a RAG assistant over customer-uploaded documents. At 09:14 an embedding-distance monitor flags that retrieved chunks for Customer A are landing inside Customer B's session above a set threshold; two support tickets arrive by 09:20 describing "someone else's numbers." The incident commander confirms cross-tenant leakage and sets Sev-1 by 09:40. By 10:50 the team has gated the retrieval endpoint and failed answers over to a "we are verifying" human queue, stopping new exposure. Forensics reconstructs that a mis-scoped index built the prior night dropped the tenant filter, and by end of day scopes exposure to 11 tenants and 320 answers over 14 hours. Legal confirms four of those tenants had a contractual 24-hour notice term, so disclosure to those four goes out that evening with the specific documents named, and a full postmortem follows within 72 hours. Total window from first signal to containment: 96 minutes, because the stages and owners were defined before the incident, not during it.
Build the capability before you need it
- Instrument AI-specific detection: ship output monitors, embedding-distance and drift alerts, canary prompts run on a schedule, and a one-click "report a bad answer" path that pipes straight into your incident queue.
- Assign every model a severity tier at deployment, keyed to data sensitivity and user reach, and write the containment action for each tier in advance (roll back, gate, or human-failover) so the on-call engineer is choosing, not inventing.
- Guarantee rollback by versioning every model, prompt, and index with the metadata your governance layer already stores, so "return to last known-good" is a deploy command and not a rebuild.
- Pre-wire the disclosure decision: maintain a lookup of which systems process which categories of personal or contractual data, mapped to the specific notification clocks and thresholds that apply, so legal decides on facts within hours.
- Run a live game-day every quarter against a real staging model, inject a synthetic drift or leakage event, and measure your actual time-to-contain against the targets in the table.
Where AI incident response quietly breaks down
- Reusing the IT runbook unchanged. It watches uptime and latency, not drift or leakage. Fix: add AI failure modes as first-class incident categories with their own detection signals and containment plays.
- No rollback path. Teams discover mid-incident that the "last good" model was never saved. Fix: version model, prompt, and index together and test a restore drill so rollback is proven, not assumed.
- Disclosure decided by whoever is loudest at midnight. Fix: pre-map data categories to statutory clocks and thresholds so legal applies a rule, and route every AI incident through legal by default rather than by memory.
- Silent failures never surface because the only signal is a user complaint. Fix: deploy proactive monitors and canary prompts so you catch drift before customers do, and treat a spike in "bad answer" reports as a paging event.
- The postmortem blames a person and stops there. Fix: run blameless reviews that produce a systemic control (a new monitor, a tenant-scoping test, a gate) and track it to closure like any other incident action item.
What to have in place this quarter
- A written AI incident playbook with the five stages, named owners, and target times per severity tier.
- At least one automated AI-specific detector live in production (drift, embedding-distance, or canary-prompt) wired to your paging tool.
- A tested rollback for your highest-severity model, proven by an actual restore drill, not a config note.
- A one-page data-to-obligation map that tells legal which clocks apply to which systems.
- One completed game-day with a recorded time-to-contain and at least one control shipped from the debrief.