Model risk management was built for models that give the same answer twice, and generative systems break that assumption on day one. The same prompt can produce different output, the model shifts when a vendor ships an update you never scheduled, and the failure modes are confident fabrication and prompt injection, not gradual drift. Classic validate-once-then-monitor discipline does not fit. This guide adapts the three-lines-of-defense model to govern behavior rather than just weights, and shows how one bank cut hallucination incidents by 60 percent. If you run AI in a regulated function, this is the control layer you cannot skip.
Why classic model risk management does not fit generative systems
Traditional model risk management, shaped by supervisory guidance such as the US SR 11-7, assumes a model is a fixed function: the same inputs produce the same output, the model is validated once against a test set, and it is then monitored for statistical drift. Generative systems violate every part of that assumption. Output is non-deterministic, so the same prompt can yield materially different answers on two runs. The model itself changes when a vendor pushes an update you did not schedule and may not even be told about in advance. And the dominant failure modes are not gradual drift but confident fabrication, prompt injection, data leakage, and unsafe tool use. A validation approach that checks accuracy on a static, curated test set will happily pass a system that invents a citation the moment it meets a real customer query it was never tested on.
This does not mean discarding the discipline. The three-lines-of-defense structure, independent validation, and ongoing monitoring all still apply and remain the backbone of credible governance. What changes is what you test for and how often you test it. You govern behavior across a distribution of prompts rather than accuracy on a fixed set. You treat the prompt template and the retrieval corpus as first-class model components, because a change to either alters behavior as surely as a change to the weights. And you treat every vendor model update as a change event that reopens validation, rather than a silent background improvement you can trust on faith. The goal is to govern the behavior of the system as deployed, not just the artifact you licensed.
A model risk framework adapted for generative behavior
Extend each classic control to the generative failure modes and assign it to the right line of defense. The table below maps the risk dimension to the specific generative test, the owner in the three-lines model, and a monitoring cadence. The organizing principle is that prompt version and corpus version are part of the model identity, so a change to either should trigger the same revalidation a weight change would. Read the table as a standing test plan rather than a one-time checklist: the cadence column is what keeps it alive as the system and its vendor evolve.
| Risk dimension | Generative test | Line of defense | Cadence |
|---|---|---|---|
| Accuracy | Grounded QA on held-out prompt set, faithfulness scored | 1st line build team | Every release |
| Hallucination | Citation-check: claims must trace to a source | 2nd line validation | Weekly sample |
| Prompt injection | Red-team adversarial prompt battery | 2nd line validation | Quarterly plus on change |
| Stability | Same-prompt variance and vendor-update diff | 1st line, monitored | Continuous |
| Bias and safety | Toxicity and disparity across cohorts | 2nd line validation | Quarterly |
| Assurance | Audit of logs, versions, and overrides | 3rd line internal audit | Annual |
A bank applying this framework to a customer-service assistant added a citation-check that failed any answer whose claims did not trace to an approved knowledge source, and it logged the prompt version and corpus version on every single response. Hallucination incidents, measured as answers that asserted a fact with no traceable source, fell about 60 percent within two quarters. Just as valuable, when the vendor shipped a model update, the continuous same-prompt diff caught a tone regression and a subtle change in refusal behavior before either reached a customer, and the team held the update in staging until the second line had re-run the core battery. Neither of those catches would have been possible under a validate-once-and-monitor-drift approach, because nothing in the input distribution had changed; only the model had. The broader principle is that in a generative system the thing you are governing is behavior, and behavior can change from three directions at once: the prompt you write, the corpus you retrieve from, and the weights the vendor ships. A control framework that watches only one of those three will miss the other two, and the misses tend to be exactly the failures that reach a customer. Extending the classic three lines to cover all three sources of change, at the cadences in the table, is what keeps the framework honest as the system evolves under you.
Stand up governance that fits non-deterministic systems
- Build a held-out prompt set that mirrors real usage and score faithfulness, not just accuracy, on every release before it ships, so you are testing behavior on the queries the system will actually meet.
- Add a citation or grounding check that fails any output whose claims do not trace to an approved source, and treat an ungrounded confident answer as a defect rather than a stylistic quirk.
- Run an adversarial red-team battery for prompt injection and unsafe tool use quarterly and again on any material change, because the most damaging failures are provoked by inputs, not surfaced by averages.
- Version the prompt template and the retrieval corpus as model components; a change to either reopens validation the same way a change to model weights would, and both must be logged on every response.
- Treat every vendor model update as an unscheduled change event: diff the same-prompt outputs, re-run the core tests, and gate adoption behind sign-off rather than accepting the update silently.
How generative model risk gets missed
- Validating on a static test set once and calling it done. Fix: monitor continuously against live-representative prompts, since output can shift without any code change on your side.
- Ignoring vendor updates that silently change behavior. Fix: subscribe to update notices and gate adoption behind a same-prompt diff and a re-run of the core battery.
- Testing accuracy but never adversarial robustness. Fix: add a prompt-injection and unsafe-tool-use red team, because the worst failures are deliberately provoked, not random.
- Not versioning the prompt or corpus, so a bad output cannot be reproduced or explained. Fix: log prompt version and corpus version on every response as mandatory metadata.
- The first line grading its own homework. Fix: keep validation independent in the second line, with internal audit assuring the whole chain of controls end to end.
Model risk controls to add this quarter
- Assemble a 200-prompt held-out set drawn from real traffic and score faithfulness on it.
- Turn on citation-checking for one high-stakes generative flow and fail every ungrounded answer.
- Run a one-day prompt-injection red team against that flow and log every successful break.
- Add prompt-version and corpus-version fields to your response logs and confirm they populate.
- Write the vendor-update change procedure and assign an owner for the same-prompt diff.