Cost Postmortems That Actually Change Things

The bill spikes, everyone stares, nothing changes

The pattern is familiar. Someone forwards the monthly model invoice with the total highlighted, a thread fills with theories, and two weeks later the number is higher. The review produced a chart and a shrug. Nothing was assigned, so nothing shipped. A cost postmortem is the opposite of that ritual: it treats a spend spike like a production incident, with a timeline, a named root cause, and owned actions that land before the next billing cycle closes. The word postmortem matters. It signals that a surprise bill is a defect in the system, not weather, and defects get investigated and fixed.

The reason ordinary reviews fail is that they stop at the aggregate. A total of $52,000 across 4.1 million requests is not actionable. What moves the number is the shape underneath it: the one workflow that retries three times on every failure, the summarizer that pastes the full document into context on each turn, the debug prompt someone left running against production traffic. Cost is never uniform. In most stacks 70 to 80 percent of the overage traces to fewer than five request patterns, and until you name them you are guessing. The average cost per request, say $0.013, hides the fact that a handful of patterns run at ten times that rate while the rest run at a tenth. Averages comfort; distributions inform.

Trace the spend, then assign the fix

A cost postmortem runs in four passes: quantify the delta against the prior baseline, decompose it by request pattern, find the root cause of the top offenders, and assign owned actions with a re-check date. The discipline is refusing to leave the room with a number instead of a decision. Every finding must end in a person, a change, and a date. Here is a worked example from a support-automation team whose bill jumped from $18,000 to $41,000 in a single month, a 128 percent rise that triggered their spend threshold automatically.

The total identified saving was $21,160 per month, enough to bring the bill back under $20,000, and every line has a name attached. Two findings are worth pausing on. The summarizer line, at 44 percent, was not a model problem at all: the code re-sent the entire ticket thread on every turn, so a conversation of 20 messages paid to reprocess the first message 20 times. Capping context at the last six messages cut its token volume by 68 percent with no measurable drop in answer quality. The debug-traffic line is the other tell: 12 percent of a $41,000 bill, roughly $4,900, was pure waste from a test harness nobody remembered was running against a production key. That is the kind of finding a chart-and-shrug review never surfaces, because it never looks below the total. The team re-checked at the next invoice: actual spend landed at $19,400, within 3 percent of the projection, and the postmortem template went into their incident tooling so the next spike runs the same way.

Run it like an incident, not a meeting

• Set a trigger threshold in advance: any month where spend rises more than 20 percent or crosses an absolute ceiling auto-opens a postmortem, so it is never a judgment call about whether the spike was big enough to bother with.
• Pull spend by request pattern, not by team or by day. Tag every call with a workflow label at the source so the decomposition takes minutes rather than a week of log spelunking, and so the same view is one query next time.
• Rank patterns by share of the overage and stop at the ones covering 80 percent. Chasing the long tail burns senior time to recover cents; the 7 percent "everything else" line can wait.
• For each top pattern write a one-line root cause and a one-line fix with a named owner and a ship date inside the current cycle. If a fix cannot ship this cycle, split it until part of it can.
• Schedule the re-check for the next invoice and put projected versus actual saving on the same page, so the loop closes and the team learns whether the fix held or the pattern found a way back.

Where cost postmortems quietly fail

• Stopping at the total. The aggregate is a symptom, not a cause. Fix: always decompose by request pattern before anyone is allowed to propose an action.
• No owner on the action. "We should truncate context" dies in the meeting notes and resurfaces next month unchanged. Fix: every fix line carries a person and a date, or it does not count as a fix.
• Optimizing the wrong tier. Teams tune a 6 percent line because it is interesting while a 44 percent line runs untouched. Fix: rank strictly by share of overage and work top-down.
• Skipping the re-check. The saving was projected, never verified, and the pattern silently returns two cycles later. Fix: book the follow-up against the next invoice before you close the current one.
• Treating cost as separate from reliability. Blind retries inflate the bill and hide real failures at the same time. Fix: read cost and error rates together, since the same guardrail usually improves both.

Before your next invoice

• Tag every model call with a workflow label at the call site so spend is decomposable on demand.
• Set an auto-trigger threshold (percent rise plus absolute ceiling) for opening a postmortem.
• Build a one-query view of spend by request pattern and bookmark it.
• Cap retries and truncate context on your two largest workflows today, before the next spike.
• Move the postmortem template into your incident tooling so it runs the same way every time.