Most AI regressions are not single-file mistakes but set mismatches: a prompt tweak that clashes with a new policy filter, or a routing change that quietly invalidates cached outputs. Version the artifacts separately and every rollback becomes archaeology while every audit becomes a guessing game. Ship them as one immutable tagged set, promoted through the same gates, and you get reproducible behavior, simple audits, and rollback measured in seconds. The pattern is to make the set, not the part, the unit you test, ship, and revert.
Regressions live in the seams
Most AI regressions are not single-file mistakes. They are set mismatches. A prompt tweak that improves one answer clashes with a policy filter shipped the same week. A routing change quietly invalidates a cache full of answers generated under the old prompt. A vendor silently updates the model behind a stable API, and behavior drifts overnight with no code change to point at. In each case the individual artifact was fine. The failure lived in the seam between artifacts that were versioned, deployed, and rolled back independently.
The cost of treating each artifact separately shows up at the worst moment. When an incident hits and you need to roll back, you discover the prompt is on version 14, the policy pack on version 9, the routing table untracked, and the model pinned nowhere. Reconstructing the exact combination that was live at 2:14 that afternoon becomes archaeology, and every minute of it is customer-facing. Teams that version artifacts as one set cut mean time to rollback from tens of minutes to seconds, because rollback is a single tag swap rather than a five-way reconciliation. The fix is simple to state and disciplined to hold: release sets, not parts.
The deeper reason to bundle is that AI behavior is emergent across the whole configuration, not additive across its parts. A prompt does not have a fixed quality score; it has a score only in combination with a specific model, a specific policy pack, and a specific retrieval configuration. Change any one and the others behave differently. That is why per-artifact versioning gives a false sense of control: you can prove each file is at its intended version and still have no idea what the system as a whole will do, because the interaction was never captured anywhere. Versioning the set captures exactly the thing that determines behavior, the combination, and makes it the unit you test, ship, audit, and roll back.
Ship one tagged set
A release set is the full bundle of artifacts that determine model behavior, promoted together under one immutable tag such as ai-suite@2025.08.18. You version the set, not the parts. The tag is applied to every artifact and every runtime switch, and it moves through dev, staging, canary, and production unchanged, with no rebuilds between environments. Semantic versioning of the set tells everyone what kind of change shipped: a major bump for a policy or model change, a minor for a prompt or routing change, a patch for a threshold nudge.
| Artifact in the set | Versioned how | Breaks silently when unpinned | SemVer trigger |
|---|---|---|---|
| model_ref | Vendor ID plus date, pinned in manifest | Upstream weight update drifts behavior | MAJOR |
| policy_pack | Checksummed file in the set | Filter change clashes with prompt intent | MAJOR |
| prompt_pack | Immutable, tagged, no in-prod edits | Edited live, incident unreproducible | MINOR |
| routing_table | Declarative config under the tag | Invalidates caches without warning | MINOR |
| retrieval_cfg | Index and rerank params in manifest | Recall shifts, answers move | MINOR |
| eval_suite | Tagged, gates promotion in CI | Stale evals pass a broken set | PATCH or higher |
A worked example shows the payoff. A team promoted release ai-suite@2025.06.02 to production: a prompt refinement plus a routing change that sent more traffic to a cheaper model. Ninety minutes later, complaints spiked on a specific answer type. Because everything shared one tag, the on-call engineer read the release notes, saw the routing delta, and rolled back to ai-suite@2025.05.19 with a single feature-flag flip. The previous set was still hot, its caches namespaced by tag, so there was no cold start and no mixed outputs. Total time from alarm to restored behavior was under 4 minutes. The postmortem then reproduced the exact failing set in staging from its manifest, found that the cheaper model failed two goldens the eval gate should have caught, and fixed the gate. Under independent versioning, that same incident would have meant untangling which prompt, policy, and routing versions were jointly live, a reconciliation that routinely runs 30 to 45 minutes while customers wait.
Make the set the unit of release
- Create a versioned set manifest: a lockfile listing every artifact URI and checksum, with the model pinned by vendor ID plus date so an upstream weight change cannot slip in unnoticed.
- Tie CI to the tagged eval suite. Promotion from staging requires a pass on answerability, safety, latency, and cost. No pass, no promotion.
- Roll out behind feature flags by segment or tenant, and keep the previous set hot so rollback is a single flag flip with no cold start.
- Namespace every cache by set tag and invalidate on promotion, so answers generated under an old prompt never leak into a new release.
- Enforce policy precedence in a registry that resolves system over policy over tool over app prompts, consistently, per set, so precedence is never ambiguous at runtime.
How set versioning slips
- Artifact drift: a prompt edited in production without bumping the set tag, producing an incident nobody can reproduce. Fix: make prompts immutable and force every change through a new tag.
- Environment snowflakes: rebuilding prompts or dependencies per environment, which creates the classic "works in staging" bug. Fix: promote the same tagged set through every environment with no rebuilds.
- Unscoped caches: old answers leaking into a new release because caches are not namespaced or have no TTL by risk tier. Fix: key caches by set tag and invalidate on promotion.
- Shadow changes: a vendor model updated without pinning or a fitness check, drifting behavior silently. Fix: pin model ID plus date and alert on upstream weight changes.
- Stale eval gates: an eval suite that lags the set it guards, letting a broken release pass. Fix: version the eval suite inside the set and require its pass for promotion.
Adopt the set today
- Tag todays production as ai-suite@YYYY.MM.DD and write the first set manifest with checksums.
- Pin every vendor model by ID plus date; add an alert on upstream weight changes.
- Namespace caches by set tag and add TTLs by use-case risk tier.
- Require a tagged eval-suite pass for staging-to-production promotion; store the scorecard with the release notes.
- Keep the previous set hot behind a feature flag so rollback is one flip.