Evaluation, Safety & Guardrails

Why "it looked good in the playground" is not a strategy

Teams ship LLM features on vibes. Someone tries a dozen prompts in a playground, the outputs read well, and it goes live. Then a customer pastes an adversarial input, or the input distribution shifts, and the feature produces something confidently wrong or unsafe. The gap is that manual spot-checking does not scale and does not catch the tail, which is exactly where LLM failures live. If a feature is right 97% of the time, hand-checking twenty outputs will almost always show zero failures, and the 3% that reaches customers is invisible until it is an incident. The tail is not an edge case; on real traffic it is the whole risk.

A real evaluation practice has two halves. Offline evals run against a fixed set before every change, so you know a prompt tweak or model swap did not regress. Online evals run against live traffic, sampling real outputs and scoring them, because production shows you failures your fixed set never imagined. Neither replaces the other. Offline gives you a fast, deterministic signal in CI, ideally under a couple of minutes so it runs on every pull request; online tells you the truth about what users actually send, which drifts week over week as usage grows and adversaries probe. Run only offline and you optimize against a museum piece. Run only online and you learn about regressions after they have already reached customers.

An eval taxonomy: match the check to the risk

Not every eval belongs in the deploy gate. Cheap deterministic checks run on every commit; expensive human review runs weekly on a sample. The organizing principle is cadence versus cost: run fast checks often and block on them, run slow checks less often and use them to steer. The table below is the taxonomy we use to decide what runs when and what actually stops a release. The rule of thumb is that anything cheap enough to run in seconds and objective enough to trust should fail closed, while anything slow or subjective should alert and inform rather than block automatically.

The most valuable eval set is not synthetic. It is built from real work artifacts: actual user inputs, the outputs that got corrected, and the tickets where the feature failed. Every production incident and every user edit becomes a new row. Over a few months this dataset becomes your moat, because it encodes the exact failure modes your users produce, which no public benchmark contains. Take a support-summarization feature. Its team started with a golden set of 60 real tickets and their human-corrected summaries, scored on a 0-to-3 rubric for factuality and completeness. A model swap that looked like a pure upgrade dropped the golden-set pass rate from 91% to 78%, all on tickets containing tables, a failure mode the vendor benchmark never touched. Because the check ran in CI, they caught it before merge instead of in a customer escalation, and the eight failing table cases became permanent regression rows.

Stand up both loops in a sprint

• Start the golden set from real artifacts: pull 50 to 100 real inputs, capture the corrected outputs, mask sensitive data, and score against those. Grow it every time production surprises you, so the set tracks reality instead of freezing at launch.
• Put assertion checks in the request path so malformed or unsafe output fails closed before it reaches a user, not after. Schema and format checks are cheap, deterministic, and catch a surprising share of real defects for near-zero cost.
• Run an LLM-as-judge with a written rubric on a nightly sample and track pass rate as a first-class metric next to latency and cost, alerting when it drops more than a few points or falls under 90%.
• Treat guardrails as product: an input classifier, an output filter, and a policy layer, each versioned and monitored with its own SLA, not a regex bolted on at the end. Give safety an owner and a backlog like any other feature.
• Red-team on a schedule with a real adversary mindset. Jailbreaks are not exotic; they are a Tuesday, so budget recurring time to break your own system and turn every successful break into a permanent case in the jailbreak suite.

Where eval programs quietly rot

• Judge grading itself: using the same model family to generate and to score, which hides shared blind spots. Fix: use a different model as judge and spot-check the judge against human ratings on a sample every week.
• Static eval sets: a golden set frozen at launch stops reflecting real traffic within weeks. Fix: append new failures continuously and retire stale cases, treating the set as a living asset with an owner.
• Guardrails with no observability: filters that block silently, so you never learn what they caught or wrongly blocked. Fix: log every block with the reason and review false positives, because an over-eager filter quietly kills good outputs and adoption with them.
• Explainability theater: a rationale generated after the fact that does not reflect the actual decision. Fix: prefer traceable inputs (which documents were retrieved, which rule fired) over post-hoc narratives that read well and mean nothing.
• Incidents with no template: each one handled ad hoc, so lessons evaporate. Fix: a standard review with trigger, blast radius, root cause, and the eval case added to prevent recurrence, so the golden set grows with every failure.

The minimum viable eval program

• A golden set of 50-plus real inputs scored in CI on every PR.
• Assertion checks in the request path that fail closed.
• Nightly LLM-as-judge pass rate tracked as a dashboard metric.
• A jailbreak suite that runs on every release and blocks regressions.
• An incident template that ends with a new eval case, every time.

Related advisory guides

Go deeper on this theme with the detailed guides in this collection.