Two failures define the first year of an LLM in production: a bill three times the forecast because every request hits the biggest model, and a quality regression nobody can explain because prompt, policy, and model were never versioned together. Both are MLOps problems, not model problems, so a bigger model just treats the symptom. Route easy requests to a small cheap model and escalate only the hard ones. Watch cost, latency, quality, and reliability, not tokens alone. Version prompts, policies, and models as one artifact so you can always answer what changed.
The bill nobody forecast and the outage nobody could explain
Two failures define the first year of an LLM system in production. The first is a cost surprise: the monthly bill is triple the forecast because every request, trivial or hard, hits the biggest model. The second is an unexplained regression: quality dropped last Tuesday and nobody can say whether it was a prompt edit, a model version change, or a shift in user input, because none of it was versioned together. Both are MLOps problems, not model problems. A team that jumps to a bigger model or a longer prompt is treating symptoms; the fix lives in how requests are routed, observed, and versioned.
The discipline that fixes them is the same one that matured for traditional services: routing, observability, and versioning. The twist is that LLM systems have non-deterministic outputs and token-based pricing, so the classic tooling does not map cleanly. You need to watch signals that request-count dashboards never had, and you need to treat a prompt as a deployable artifact with the same rigor as a container image. A one-line prompt edit can move cost, latency, and quality all at once, so it deserves the same review, versioning, and rollback path as a code change, not a quiet edit in a vendor console.
An observability signal map for LLM systems
Token count is the signal everyone watches and the least useful one alone. What actually predicts cost and outages is a small set of signals across four dimensions: cost, latency, quality, and reliability. Watch all four with real thresholds, because a system can be cheap and fast while quietly getting less accurate, and tokens will never tell you that. The map below is what we wire up before a feature reaches meaningful traffic, with concrete alert lines so the dashboard pages someone instead of just decorating a wall.
| Signal | Why it matters | Tool / source | Alert threshold |
|---|---|---|---|
| Cost per request (rolling) | Catches routing regressions before the bill does | Gateway logs, provider usage API | >20% above 7-day baseline |
| P95 end-to-end latency | Tail latency, not mean, is what users feel | Tracing (OpenTelemetry spans) | P95 > 4s on interactive paths |
| Time to first token | Perceived speed on streaming responses | Streaming gateway metric | >800ms on chat surfaces |
| Online eval pass rate | Silent quality drift no cost metric shows | LLM-as-judge on sampled traffic | <90% or a 5-point drop |
| Fallback / retry rate | Provider instability and timeout pressure | Gateway error logs | >2% of requests retried |
| Escalation rate (small to large model) | Router health and cost driver | Router decision logs | >40% escalating (router miscalibrated) |
Two-model routing is the highest-leverage cost pattern. A small fast model handles the majority of traffic, and a cheap classifier or confidence check escalates only genuinely hard requests to the large model. Done well this cuts cost by half or more with no perceptible quality loss, because most real traffic is easy. Work the arithmetic: if a large model costs $10 per million output tokens and a small one costs $0.60, and a length-and-task classifier keeps 70% of traffic on the small model with no measured quality drop, blended cost falls from $10 to about $3.42 per million, a 66% reduction, while the golden-set pass rate holds within a point. The escalation rate in the table is how you know the router is tuned: too high and you are paying for the big model anyway, too low and quality suffers, so you watch it like a thermostat.
Wire the gateway, then the router, then the versioning
- Route through a single gateway so every request has one place to log cost, latency, model version, and prompt version. Without this one chokepoint, observability is impossible and cost attribution is guesswork.
- Start two-model routing with a simple rule (length, task type, or a cheap classifier), measure escalation rate, and tune until quality holds at the lowest cost. Begin conservative, escalating too often, then tighten as the golden set confirms the small model is safe.
- Pick batch or streaming per surface: stream interactive chat for time-to-first-token, batch async jobs for throughput and lower per-token cost, since many providers price batch work well below real-time.
- Version prompts, policies, and model IDs together as one release artifact, so a rollback reverts all three and "what changed" has one answer instead of a three-way guessing game across consoles.
- Run cost postmortems that end in a change: a routing rule, a prompt trim, a cache. A postmortem that only produces a chart changes nothing, so make a merged behavioral change the definition of done.
How LLM ops quietly gets expensive and opaque
- Watching tokens only: a green token dashboard while quality silently drifts. Fix: track online eval pass rate next to cost as a co-equal signal, because cheap and fast can still mean wrong.
- Prompt changes outside version control: edited in a console, untracked, so regressions are unattributable. Fix: prompts live in the repo and deploy like code, through review and with a rollback path.
- One model for everything: paying flagship prices for trivial requests. Fix: two-model routing with a measured escalation rate, which typically cuts spend by half or more.
- Mean-latency dashboards: they hide the tail that actually hurts users. Fix: alert on P95 and time-to-first-token, not averages, since the slow tail is what people notice and remember.
- Cost postmortems with no owner or action: a recurring meeting that produces charts and no change. Fix: every postmortem closes with a merged behavioral change, assigned to a named owner with a date.
The production-readiness shortlist
- A single gateway logging cost, latency, model version, and prompt version per request.
- Two-model routing live with escalation rate on a dashboard.
- P95 latency and time-to-first-token alerts, not mean.
- Prompts, policies, and model IDs versioned as one artifact.
- A cost postmortem template that requires a merged change to close.
Related advisory guides
Go deeper on this theme with the detailed guides in this collection.
Cheap first, smart second, route only when needed.
Answerability, latency budget, and drift, not just spend.
When nightly jobs beat real-time (and vice versa).
From “too expensive” to concrete routing/caching fixes.
Ship sets, not parts; roll forward safely.