Most AI teams log tokens and latency, then wonder why they cannot improve the product. Those are perimeter metrics: they tell you the machine ran, not whether it helped. A dashboard full of green latency charts sits happily above a product quietly giving wrong answers, because nothing on it measures correctness. The telemetry that compounds into learning is smaller and more specific: the retrieval IDs behind an answer, the confidence at decision time, whether the user accepted or reworked the output. Log that handful of events and every session becomes a training signal instead of a lost one.
You cannot improve what you only measure at the perimeter
Walk into most AI teams and the observability story is the same: tokens per request, latency percentiles, error rate, cost per day. These are real and worth having, but they are perimeter metrics. They tell you the machine is running. They tell you nothing about whether the answer was right, whether the user believed it, or whether the retrieval that fed it was any good. A dashboard full of green latency charts sits happily above a product that is quietly giving wrong answers, because nothing on that dashboard measures correctness.
Micro-telemetry is the opposite discipline. Instead of aggregate system health, it captures the small, specific events inside a single interaction that reveal whether the system actually learned or helped: which chunks were retrieved and at what score, what confidence the model held when it answered, whether the user accepted the output verbatim or reworked it, and what they did next. Each of these is a labeled example waiting to happen. Log them well and every session your users run becomes free training data for your next evaluation set and your next prompt revision. The economics here are unusually good. You are already serving these interactions; the only added cost is instrumenting the decision points and setting a retention policy. In return you get a stream of real, labeled, in-distribution examples that no synthetic test set could match, because they come from the exact users, phrasings, and edge cases your product actually meets in production.
Log decisions and outcomes, not just throughput
The useful events cluster into a few categories. The test for whether an event belongs is simple: could you use it later to decide whether the system was right, or to build an evaluation case? If yes, log it with enough context to reconstruct the moment. If it only tells you the request happened, it is perimeter noise.
| Event | What it captures | What it enables later | Retention |
|---|---|---|---|
| Retrieval IDs and scores | Which chunks fed the answer, at what similarity | Debugging bad answers, tuning the index | 90 days |
| Model confidence at answer time | How sure the system was before the user saw it | Calibrating the uncertainty threshold | 90 days |
| Accept / edit / reject | Whether the user kept, changed, or dropped the output | The single strongest quality signal you have | 1 year |
| Edit distance on kept outputs | How much the user had to fix a "good enough" answer | Finding near-misses that metrics call wins | 1 year |
| Prompt and model version | Exactly which config produced this result | Attributing regressions to a change | Indefinite |
| Follow-up action | What the user did in the next 60 seconds | Inferring real satisfaction without a survey | 90 days |
The accept-edit-reject event is the workhorse. Consider a drafting feature where the token dashboard reports a healthy 99.8 percent success rate, meaning the calls completed. Micro-telemetry tells a different story: of 1,000 drafts, 620 were accepted unchanged, 310 were kept but heavily edited at a median edit distance of 240 characters, and 70 were rejected outright. The real usable-as-is rate is 62 percent, not 99.8 percent, and the 310 heavily-edited drafts are the exact examples your next evaluation set should contain.
Instrument the decision points, then close the loop
- Attach a provenance record to every generated output: the retrieval IDs, the similarity scores, the prompt version, and the model. This is a governance requirement anyway, and it doubles as the richest debugging trail you will ever have when an answer goes wrong.
- Capture the accept, edit, or reject signal implicitly from user behavior, not a thumbs-up widget. Whether they kept the text, changed it, or deleted it is honest data that a rating prompt, which most users ignore, will never give you.
- Record confidence at answer time and compare it against the accept rate weekly. If high-confidence answers get rejected as often as low-confidence ones, your threshold is miscalibrated and users are being shown false certainty.
- Sample and label a slice of real sessions each week, pulling the heavily-edited and rejected cases first. These near-misses become your evaluation set, so the set reflects real failure rather than synthetic tests you invented.
- Scope every telemetry write by workspace_id and set a retention window per event type. Behavioral signals can live a year to spot drift; raw content should expire faster to respect privacy and control storage cost.
The five ways telemetry fails to teach
- Logging throughput and calling it observability. Tokens and latency cannot tell you if an answer was right. Fix: add at least one outcome event, accept-edit-reject, before you add any more perimeter metrics.
- The ignored feedback widget. Thumbs-up buttons get used by under 2 percent of users and skew toward extremes. Fix: infer satisfaction from behavior, what they kept and what they did next, instead of asking.
- Orphaned events with no context. An "answer generated" log with no retrieval IDs or version is useless for debugging. Fix: every event carries the provenance needed to reconstruct the moment.
- Logging content forever. Storing full prompts and outputs indefinitely creates a privacy liability and a storage bill. Fix: set per-event retention, keep behavioral signals long, expire raw content fast, and scope by workspace.
- Collecting data no one reviews. Telemetry that is never sampled or labeled is just cost. Fix: put a weekly ritual on the calendar to pull edited and rejected cases into the evaluation set, closing the loop.
Turn on learning signal this sprint
- Add an accept-edit-reject event to your primary AI feature, captured from behavior, not a rating prompt.
- Attach retrieval IDs, scores, prompt version, and model to every generated output as provenance.
- Log model confidence at answer time and chart it against the accept rate to check calibration.
- Set a per-event retention policy and scope every write by workspace_id.
- Schedule a weekly session to pull edited and rejected cases into your evaluation set.