When to Fine-Tune vs. Prompt vs. Tools

Reach for the cheapest lever that clears the bar

Fine-tuning has a gravitational pull. It feels like the real engineering move, the thing serious teams do, so it gets chosen before anyone has tried a better prompt. The result is a six-week labeling project that produces a model you cannot easily update, when a system prompt and a retrieval step would have shipped in three days and stayed correct as the facts changed. The lever you pick is an architecture decision with long consequences for cost, latency, and how fast you can respond to change.

The right choice falls out of three questions: how fast do your facts change, what behavior must you control precisely, and what can you actually measure. Fast-changing facts belong in retrieval, not weights, because you update a source in minutes but retrain a model in days. Deterministic actions belong in tools, not prose, because a function returns an exact number and a model only predicts a likely one. Stable style and narrow, well-labeled tasks are where fine-tuning finally earns its cost. Most real features end up combining prompting with tools and reserve fine-tuning for the one or two places it measurably pays.

The framework below turns those instincts into a testable decision you can run in a week, so the choice rests on metrics rather than on which lever felt most impressive in the planning meeting. The discipline that makes it work is sequencing: you exhaust the cheap, reversible levers first, and you only reach for the expensive, sticky one after the cheap ones have plateaued against a real benchmark. That order is not conservatism for its own sake; it is how you avoid baking a moving target into weights and discovering, three weeks later, that a prompt change would have done the same job for a fraction of the cost.

Five questions, five defaults

Answer each question about the capability you are building. A "yes" points to the lever on the right. The questions are ordered from cheapest and most reversible to most expensive and most sticky, so if several apply you generally start at the top and only descend when the higher levers cannot meet the bar.

The reason the order matters is reversibility. A prompt change ships in minutes and rolls back just as fast. A tool is a bounded, auditable piece of code you can test in isolation. A fine-tune, by contrast, couples your behavior to a training run, a dataset you must maintain, and a retraining cadence every time the base model or the task shifts. That stickiness is exactly why it should be last: you pay its maintenance cost forever, so it has to buy something the cheaper levers demonstrably cannot. When a feature needs fresh facts and one exact calculation and a consistent tone, that is usually RAG plus a tool plus a good prompt, not a fine-tune at all.

Here is the framework as a one-week bakeoff. A team building a claims-triage classifier wants to fine-tune from day one. Instead they build a 75-item golden set with rubric scoring for accuracy, policy adherence, and latency. Prompt-only baselines hit 71% accuracy at 900ms and $0.004 per call. Adding retrieval of the policy manual and one lookup tool lifts accuracy to 88% at 1,300ms and $0.006, clearing the 85% bar. A trial LoRA fine-tune on 1,400 labeled examples reaches 90% but costs a week of labeling, adds a retraining dependency, and buys only 2 points. The call is clear: ship RAG plus tools now, and revisit fine-tuning only if the extra 2 points later justify the maintenance, measured per dollar rather than argued by feel.

Run the decision in a week

• Build a 50 to 100 item golden set from real work, with a rubric scoring accuracy, policy adherence, and latency, before you touch any lever.
• Baseline with prompting: try schema-first, exemplar-first, and policy-first patterns, keep the best, and record its score, latency, and cost per call.
• Add RAG and one or two critical tools, re-run the benchmark, and compare the quality gain against the added latency and cost.
• Only if you have plateaued, trial a small adapter fine-tune on one or two narrow intents with 1k or more labeled examples, and measure the incremental gain per dollar.
• Pick the cheapest architecture that clears the bar, ship it, version prompts, tools, and models together under one release tag, and revisit quarterly.

Where the choice goes wrong

• Premature fine-tuning: tuning before prompts and tool boundaries are stable bakes in a moving target. Fix: exhaust prompting and tools first, then tune what has settled.
• Unlabeled evals: arguing "this looks better" with no rubric or golden set. Fix: score every lever against the same labeled benchmark.
• Tool sprawl: many functions, no observability, so latency balloons and failures are opaque. Fix: keep the tool layer small, idempotent, and logged.
• Static behavior in prompts: hard-coding policies in prompts that belong in tools or governance. Fix: externalize rules that must be enforced and audited.
• Frozen decisions: picking a lever once and never re-testing as facts and volume change. Fix: revisit the bakeoff quarterly with fresh data.

Five moves to start this week

• Publish a one-page decision tree and share it with product and risk.
• Build a 50 to 100 item golden set with a scoring rubric.
• Run a one-week bakeoff: prompt-only, then RAG plus tools, then a trial fine-tune.
• Choose on metrics: the cheapest architecture that clears the bar.
• Version prompts, tools, and models as a set under one release tag.