Observability: What Matters Beyond Tokens

Token graphs prove the model ran, not that it worked

The first dashboard most teams build for an LLM feature tracks tokens in, tokens out, latency, and error rate. Those are real, but they are the vital signs of the plumbing, not of the product. A retrieval-augmented system can hold latency flat and token counts steady while its answers quietly rot, because an embedding change dropped recall from 82 to 61 percent and nothing in the token graph moved. The failure is invisible at the layer most teams watch.

Observability that matters treats the AI feature as a chain of steps, each with its own health signal: did retrieval find the right context, was the answer grounded in that context, did the tool call succeed, and what did the whole task cost to resolve correctly. When you instrument the chain instead of the endpoint, silent regressions become visible, and you can attribute a bad answer to the step that caused it rather than guessing.

The reason this matters more for AI than for a normal service is that failures here are usually soft. A traditional API either returns a 500 or it does not. An LLM feature returns a fluent, confident, well-formed answer that happens to be wrong, and it does so with the same status code and roughly the same token count as a correct one. Quality is a separate axis from availability, and nothing on the standard infrastructure dashboard measures it.

Instrument the chain, and measure quality per step

Trace every request end to end with a single trace ID that follows it through retrieval, generation, and any tool calls, then attach a quality signal to each span. The point is attribution: when an answer is wrong, you want to know whether retrieval missed, the model ignored good context, or a tool returned garbage. The table lists the signals that earn their place beyond tokens, what they catch, and a workable target to alert on.

The unit that ties these together is cost per resolved task, not cost per call. A cheaper per-call model that triples retries and refusals is more expensive where it counts, and only the resolved-task denominator makes that visible.

Here is how it plays out in practice. A support assistant ships a routine change: the vector index is re-embedded with a newer model to improve quality. Token graphs are flat, latency is flat, the error rate is zero, and the change looks clean. But complaints trickle in a week later. With chain-level observability the story is legible in minutes. The trace view shows retrieval hit rate on the labeled 300-query set fell from 84 to 67 percent the day of the re-embed, because the new vectors were never re-indexed for a subset of product docs. Groundedness on sampled live traffic dropped from 93 to 79 percent as the model, starved of the right context, started filling gaps from memory. Cost per resolved task rose 22 percent because users re-asked and the assistant retried. None of those three signals lives on a token dashboard, yet together they point straight at the re-embed as the cause, turning a vague "answers got worse" into a one-line root cause and a targeted re-index fix.

The broader lesson is that these signals only pay off if they are wired before you need them. Reconstructing a week-old regression from raw logs is nearly impossible once the context that produced each answer is gone, so the trace ID, the labeled evaluation set, and the groundedness sampler have to be standing infrastructure, not something you bolt on during the incident. Teams that treat quality telemetry as a launch requirement, on par with logging and alerting, catch the silent regressions in hours; teams that add it after the first outage spend that outage guessing.

Build the signals that catch failures before users do

• Propagate one trace ID across retrieval, generation, and tools so any answer can be replayed span by span with its inputs.
• Maintain a labeled evaluation set of 200-500 real queries and score retrieval hit rate and groundedness on it on every deploy, not just at launch.
• Sample 1-2 percent of live traffic into an automated groundedness judge, and route low scores to human review.
• Track cost per resolved task, defining "resolved" with an explicit signal such as a thumbs-up, a completed action, or no follow-up within the session.
• Alert on rate of change, not absolute values. A retrieval hit rate falling 6 points in a week is the story, not the raw number.

Where token-only monitoring leaves you blind

• Watching only tokens and latency. Fix: add retrieval hit rate and groundedness so answer-quality regressions become visible.
• No end-to-end trace ID. Fix: propagate one ID through every span so a bad answer can be attributed to the failing step.
• Groundedness measured once at launch. Fix: score it continuously on a labeled set and on sampled live traffic, since embeddings and data drift.
• Counting cost per call instead of per resolved task. Fix: define resolution explicitly and divide spend by it so retries and refusals are priced in.
• Silent tool failures swallowed as generic errors. Fix: instrument schema validation and status codes per tool, and alert below 98 percent success.

Confirm you can see past the token graph

• Every request carries one trace ID across retrieval, generation, and tools.
• Retrieval hit rate and groundedness are scored on a labeled set every deploy.
• A live-traffic sample flows into an automated groundedness judge.
• Cost is reported per resolved task, with resolution explicitly defined.
• Alerts fire on rate of change, and tool-call success is tracked per tool.