When to fine-tune, and when not to
Every vendor will tell you what fine-tuning is. Almost none will tell you when to walk away from it. That gap is where the money is lost. Here is the decision, start to finish.
What fine-tuning actually is
A base model has already read most of the public internet. Fine-tuning shows it a few thousand more examples of one specific task, like your dialogue summaries, your ticket resolutions, or your classification calls. It nudges the weights so the model does that task the way your examples do. You are not teaching it new facts. You are teaching it a behavior.
That is a narrow thing, and it is easy to confuse with three cheaper moves that look similar. A good prompt changes behavior with no training at all. RAG hands the model your facts at question time without touching its weights. And a frontier model often already does the task well enough that you change nothing. Fine-tuning only makes sense once you know it is doing something those three cannot.
So before the how, the honest question is the when. Most of the value is in ruling it out.
| What it changes | What it costs | What it's for | |
|---|---|---|---|
| Prompting | The instruction, not the model | Nothing but tokens | Most tasks. Try this first |
| RAG | The facts in context at query time | Inference plus a vector index | Your data, your documents, live info |
| Frontier model | Nothing. You just call it | Per-token API price | Anything it already does well |
| Fine-tuning | The model's weights | Training, hosting, retraining | Proprietary judgment, volume, data that can't leave |
The decision path
Follow it top to bottom. Each step is a reason the previous one failed. If you never fall through, you never fine-tune, and that is the common case, not a failure.
When fine-tuning does earn its keep
Three cases, and they are specific. If none of them describes your task, the demo on the previous page is your situation, not this one.
The model has to reproduce decisions only your people have made. An AP clerk's exception calls. An underwriter's rationale. Which tickets get escalated and why. A frontier model never saw those decisions, and no document holds them. The judgment lives in what your team did, case after case. And here is the part most people miss: the labels usually already exist. Every resolved ticket, every approved invoice, every closed case is a training example nobody had to sit down and label. The data is process exhaust.
This is the real enterprise frontier, and where our judgment-capture work lives.
At a few thousand calls, paying a frontier model per token is the cheapest thing you will do all week. At tens of millions of calls a month, a small model you host can beat frontier pricing even when the quality is identical, which, as the demo shows, it can be. The break-even is a spreadsheet question, not a belief.
Find your break-even in the Inference Cost Modeller →Some records cannot be sent to an external API at all. Regulated data, contractual boundaries, sovereignty rules. When the data has to stay on infrastructure you control, a model you host is not the cheaper option, it is the only option. Fine-tuning is how you make that hosted model good at the task.
When it does not
Generic summarization. Generic classification. Sentiment, extraction, rewriting, translation. Anything a frontier model already does well out of the box. If your task is a common one, someone already trained a very large model on it, and you will not beat that by fine-tuning a small one.
The sharpest tell: if your edge is missing context, where the model would be right if it just knew your facts, that is RAG, not fine-tuning. Fine-tuning is for missing judgment, not missing facts. Confusing the two is the most expensive mistake in this whole area.
The demo you just saw is exactly this case. Fine-tuning a BART model on dialogue summaries worked, and a frontier model matched it with zero training. Generic task, frontier wins. That is the rule, not the exception.
How would you even know it worked?
You do not ship a fine-tuned model because the outputs feel better. Before training, you hold back a set of labeled examples the model never sees. After training, you run both the old and new model against that held-out set and measure the difference. Overlap with the reference summaries, exact-match on a classification, whatever the task actually cares about. If the tuned model does not measurably beat the alternative on examples it was never trained on, you have a nice story and no result.
This is also the honest test for the three cases above. “Proprietary judgment” is only real if a held-out set of your team's past decisions shows the tuned model making them and the frontier model missing them. Measure it, or you are guessing.
Fine-tuning is a narrow tool. Most of the value is in knowing you don't need it, and recognizing the rare case where you do.
Before you fine-tune anything, answer these
- Have you tried a frontier model with a good prompt, and did it actually fail?
- If it failed, was it missing your facts? Then reach for RAG, not fine-tuning.
- Was it missing consistent format or behavior? Try a few examples in the prompt first.
- Does the task need judgment only your people have made, and do the labels already exist as records you keep?
- Are you at a volume where a small hosted model is cheaper than frontier pricing? Model it.
- Must the data stay on your own infrastructure?
- Do you have a held-out, labeled set to prove the tuned model actually wins?
If 4, 5, or 6 is a clear yes and 7 is ready, fine-tuning is on the table. Otherwise you have a prompting or RAG problem wearing a fine-tuning costume.