Most evaluation sets are written by the same people who built the model, so they test the clean cases the team already imagined and miss the ones that break in production. That is how a suite scores ninety-two percent in the lab, then falls over against misspelled names, half-finished sentences, and angry-customer phrasing nobody invented. The richer material already sits in your tickets, chat logs, and documents, where the real edge cases live. Mine those artifacts into a golden set of 50 to 100 cases, de-identify without gutting the signal, and stratify sampling so scores finally predict production.
Hand-written eval sets test the wrong distribution
When a team writes its own evaluation set, it writes down the cases it can imagine. Those cases are clean, well-formed, and centered on the happy path, because the people writing them are the people who designed the happy path. The result is a test suite that scores ninety-two percent in the lab and then falls over the first week in production, because production is full of misspelled entity names, half-finished sentences, two questions stapled into one message, and the specific angry-customer phrasing nobody thought to invent. The eval set was not wrong. It was just testing a distribution the product never actually sees.
The material that would fix this already exists. Every support ticket, every chat transcript, every email thread, and every internal document is a record of a real request phrased the way real people phrase things, including the edge cases that only surface under load. The job is not to generate more test cases. It is to mine the ones you already have, protect the sensitive parts, and sample them so the eval set carries the same shape as production traffic. Do that and your test scores start predicting your production behavior, which is the entire point of testing and the thing synthetic suites quietly fail to deliver.
Mine, protect, stratify, annotate, wire in
Turning raw artifacts into a trustworthy eval set is a pipeline with five stages, each of which fails differently if you skip it. The table below is the working spine: what each stage does, the trap it exists to avoid, and a concrete target you can measure against so the work has an end state rather than drifting forever.
| Stage | What it does | Failure it prevents | Concrete target |
|---|---|---|---|
| Source inventory | Catalog artifact types by sensitivity and format | Pulling from one easy source and calling it representative | At least 3 sources, each tagged low, medium, or high sensitivity |
| De-identification | Redact PII, PHI, credentials, contract terms | A leaked customer name ending up in a test fixture in the repo | Automated redaction plus a sampled human check on 10 percent |
| Stratified sampling | Draw cases by frequency, severity, and domain | Over-representing easy tickets because they are the common ones | Each severity tier is at least 15 percent of the set |
| Golden annotation | SMEs label the correct output and the reason | Grading against an answer nobody actually agreed was right | Two annotators per case, disagreements resolved and logged |
| Evaluation harness | Auto-score the set on every pre-release build | A set that gets built once and never runs again | Wired into CI, blocks release on a score regression |
| Refresh cadence | Add new incidents, retire stale ones | A frozen set that stops reflecting current traffic | Quarterly refresh, version and change-log each revision |
The target column matters more than the stage names. A pipeline with vague intentions produces a set nobody trusts; a pipeline with numeric targets produces a set you can defend in a release review. Fifty to a hundred well-chosen, well-labeled cases drawn from real incidents will out-predict a thousand synthetic ones every time, because they carry the messiness the model will actually meet.
Turn artifacts into a defensible golden set
- Start with an artifact ingestion job that pulls from at least three real sources, such as the support ticket queue, the shared inbox, and a document store, and stamps each item with its source and a sensitivity tag before anything else touches it.
- Run automated redaction first, then sample ten percent for a human de-identification check. Redaction that removes names but leaves an account number or a contract clause is a leak waiting to be committed, so verify on a sample before the data leaves the pipeline.
- Stratify the draw so each severity tier and each major domain holds at least fifteen percent of the set. If escalations are two percent of traffic but half your production pain, deliberately over-sample them so the eval set tests where it hurts.
- Have two subject-matter experts annotate each case with the correct output and a one-line rationale, and log every disagreement and its resolution. The rationale is what lets you debug a failed test later instead of just seeing that it failed.
- Wire the finished set into CI so it runs on every candidate build and blocks the release on a score regression, and version the set so a passing score is reproducible against a known revision months later.
How real-artifact eval sets go wrong
- Shipping sensitive data into the test suite. Redaction is skipped or partial, and a customer name or account number lands in a fixture in version control. Fix: redact before storage, never after, and gate the pipeline on a sampled human check so nothing sensitive is committed.
- Over-representing the easy cases. Common tickets are common precisely because they are simple, so a naive random draw fills the set with them and the hard cases vanish. Fix: stratify by severity and frequency and hold a minimum quota for escalations and failures.
- Losing provenance. A case gets edited or its source is forgotten, so a failed test cannot be traced back and the result is not reproducible. Fix: stamp source, extraction date, and transformation version on every case and never mutate a case in place.
- Grading against contested answers. One person decided the "correct" output and nobody checked, so the set encodes one opinion as ground truth. Fix: use two annotators per case, resolve disagreements explicitly, and record the resolution as part of the case.
- Freezing the set at launch. The set is built once, then usage shifts and new failure modes appear that it never covers. Fix: refresh quarterly by adding cases from recent incidents and retiring stale ones, versioning each revision with a change log.
Stand up a real-artifact eval set this quarter
- Pick two artifact sources you already have, run a redacted extraction on both, and confirm the redaction on a ten percent human-checked sample.
- Assemble 50 to 100 golden cases drawn from real incidents, stratified so each severity tier is at least fifteen percent of the set.
- Get two SMEs to label the correct output and a one-line rationale for each case, logging every disagreement and how it was resolved.
- Wire the set into your pre-deployment checks so it runs on every candidate build and blocks release on a score regression.
- Schedule a quarterly refresh that adds new incident cases, retires stale ones, and versions the set with a dated change log.