AI systems fail in ways classic software does not. The same prompt returns a clean answer at 9am and a fabricated citation at noon because temperature, retrieved context, and a silent vendor model swap all changed. A generic postmortem asks which line of code was wrong, finds nothing, and closes with a shrug, so the incident recurs a month later in a new shape. This is a blameless, seven-part review built for probabilistic systems, with a severity scale, an error taxonomy, and one rule: every incident adds a test to your golden set so the failure cannot pass silently again.
Why AI incidents need their own template
Classic postmortems assume deterministic systems: a bug shipped, a query was slow, a disk filled. AI systems break that assumption. The same prompt returns a good answer at 9am and a fabricated citation at noon because temperature, retrieved context, and a silent vendor model update all changed. A generic root-cause template asks "what line of code was wrong" and finds nothing, so the incident closes with a shrug and recurs a month later in a slightly different shape.
An AI-specific review has to capture model and data behavior, guardrail efficacy, and decision impact, not just the stack trace. It has to name the error type precisely, because a hallucination, a retrieval miss, and a jailbreak demand entirely different fixes. And it has to be blameless and lightweight, because a review that takes three days and points fingers will be skipped under delivery pressure. The template below is designed to be filled in under an hour, produce concrete control changes, and feed your evaluation suite so the same failure cannot pass silently again.
There is a second reason the format matters. AI incidents cluster: one loosened refusal, one stale index, one unapproved model bump tends to produce a family of failures, not a single event. A review that only patches the one output you happened to see leaves the class untouched, and the next variant lands next week. The value of a structured review is that it forces you from the instance to the class, from "this answer was wrong" to "which control let this class of answer through," and then binds a fix to that control. Done consistently, incidents stop being noise and become the highest-signal input you have into what to test and harden next.
Seven sections that turn a failure into a control change
The review moves from triage to verified close in seven sections. Each one exists to answer a specific question and to hand the next section what it needs: triage locates and ranks, the context snapshot freezes what was running, and characterization forces a precise error name before anyone proposes a fix. The table shows the section, its purpose, and the concrete artifact it must produce before the review can advance.
| Section | Purpose | Required artifact |
|---|---|---|
| Header and triage | Locate and rank the incident | ID (YYYYMMDD-domain-seq), severity S1 to S4, discovery source |
| Context snapshot | Freeze what was running | Model and prompt-pack version, retrieval sources, HITL gates |
| Impact summary | Size the blast radius | Users affected, cost, latency, SLA breaches, override rate |
| Failure characterization | Name the error precisely | Error type plus a minimal reproducible prompt and input |
| Contributing factors | Explain how controls let it through | Model, retrieval, guardrail, and ops factors, no blame |
| Remediation, verification, and close | Fix the class, then prove it will not recur silently | Tightened guardrail or monitor, a golden-set test, post-fix metrics vs baseline, and a decision |
The sequence is deliberate. Characterization comes before remediation because a fix proposed without a precise error type usually treats the symptom: teams add a keyword filter for a hallucination that was really a retrieval miss, and the miss keeps happening. Contributing factors come before remediation because you cannot harden a control you have not named. And verification is a hard gate, not a formality, because an AI fix that is not proven against a reproducing case and a baseline metric is just a hopeful edit that the next release can quietly undo.
Consider a worked case. A lending assistant tells a customer their application "is approved," which it has no authority to say. Triage tags it 20260614-lending-004, severity S2 (regulatory exposure). The context snapshot shows prompt-pack v7 shipped two days earlier without a re-run of the refusal suite. Failure characterization names it a policy misfire and captures a 12-word prompt that reproduces it every time. Contributing factors: the "never state a decision" rule lived only in the app prompt, which lost precedence after a merge. Remediation moves the rule into the policy pack with precedence, adds a pre-flight classifier, and files one regression test. Verification shows the override rate on that intent falling from 9% to under 1% across the next 1,500 sessions, and the incident closes with a control that generalizes rather than a patch that does not.
Operationalize the template
- Standardize one severity scale (S1 to S4, keyed to customer harm, regulatory exposure, financial impact, and operational disruption) and one error taxonomy of 6 to 8 types across every team, so incidents are comparable.
- Wire the template to your ticketing system as a form or copyable doc so opening a review takes seconds and links remediations to tracked tickets with owners and SLOs.
- Make one golden or eval test per incident a hard gate: no review closes until a reproducing case is added to the suite and passes against the fix.
- Keep a living control registry of every guardrail, test, and monitor, each with an owner and an SLO, and update it in the remediation section of every review.
- Run a monthly governance review that summarizes incidents by type, tracks repeat rate, and reports control effectiveness rather than raw incident counts.
What breaks incident reviews
- Blaming people: reviews become defensive and stop surfacing the real chain. Fix: assume good intent and interrogate systems and controls, never individuals.
- One-off fixes: patching the single bad output without hardening the guardrail guarantees recurrence. Fix: require a control change that covers the whole error class.
- No reproduction: closing on a narrative with no minimal repro means you cannot test the fix. Fix: block close until a reproducing prompt and input are captured.
- Unowned actions: remediations without a named owner and date quietly rot. Fix: every follow-up is a ticket with an owner and an SLO.
- Reviews that never feed evals: the same failure sails through the next release. Fix: enforce the golden-test-per-incident rule as a merge gate.
Five moves to start this week
- Publish the seven-section template as a copyable doc with two worked example incidents.
- Define S1 to S4 and the 6 to 8 error types your org will use.
- Require one golden or eval test per incident before any close.
- Stand up a control registry with owners and SLOs.
- Schedule a monthly incident and control-effectiveness review for governance.