Anmol ParimooWhen does fine-tuning actually make sense versus prompting
Fine-tuning makes sense when you have a narrow, high-volume, stable task and you want a smaller, cheaper model to match a bigger model's quality on that one thing. For almost everything else, prompting with retrieval gets you there faster and cheaper, and for anything that has to act inside your systems, fine-tuning is the wrong tool entirely. That is the short answer. The rest of this piece is the decision framework behind it, because "just fine-tune it" is the instruction data teams get handed most often, and it is wrong more often than it is right.
If you want to skip to the interactive version, our fine-tune vs agents decision framework runs the same logic and a 24-month cost model on your numbers.
The misconception that starts most of these arguments
Here is the single line that ends the "why can't we just fine-tune it" debate:
Fine-tuning changes how a model responds, not what it can do. A fine-tuned model cannot call your APIs, query your database, or hold state across steps, so for any task that has to take actions in your systems, fine-tuning cannot be the answer.
This is the misconception worth killing on the spot, because it drives real budget. An executive hears "fine-tune a model on our data" and pictures a model that can look up an account, issue a refund, and log the result. That is not what fine-tuning produces. Training on your examples shifts the style, format, and vocabulary of the output. It does not give the model hands. The moment a task needs to read or write anything in your stack, you are talking about an agent, whatever anyone calls it in the meeting.
What each tool actually does
Three tools get conflated in these conversations. They are not competitors on one axis; they solve different problems.
- RAG (retrieval-augmented generation). The model looks things up in your documents at request time and answers using what it retrieved. Nothing is trained. You change behavior by changing the retrieved context.
- Fine-tuning. You train a base model further on your labeled examples so its outputs match a style, format, or domain. Behavior is baked in at training time. It still cannot call tools.
- Agents. You give a model tools and a control loop, so it can call APIs, read and write records, take multi-step actions, and carry state through a task. This is the only one of the three that acts.
| Capability | RAG | Fine-tuning | Agents |
|---|---|---|---|
| Call your APIs or tools | No | No | Yes |
| Hold state across steps | No | No | Yes |
| Answer in your voice and format | Partly | Yes | Yes, using the model underneath |
| Use your latest data | Yes, at request time | No, frozen at training | Yes, live via tools |
| Time to change behavior | Minutes | A training run | Minutes to hours |
| Typical unit cost | Low | Low inference, real training cost | Highest, scales with calls per task |
The table makes the pattern obvious: if the row you care about is "call your APIs" or "hold state," only one column says yes, and it is never fine-tuning.
The disqualifying conditions, as a framework
The useful way to choose is not to argue for your favorite tool. It is to run gates that eliminate the wrong tools in order, so the right one is whatever survives. We call this the MLDeep Fine-Tune vs Agents Framework, and these are the disqualifying conditions that do the work:
- Needs to act in your systems. If the task calls tools, mutates records, or spans multiple steps with state, fine-tuning and plain RAG are both out. You are building an agent.
- Needs current data. If the answer depends on data that changes after any training run, fine-tuning is out. Retrieval or tool calls are in.
- Low volume or unstable. If the task runs rarely or its shape keeps changing, fine-tuning cannot amortize its training and maintenance cost. Prompting is in.
- Narrow, high-volume, stable, no tool calls. This is the one case where fine-tuning genuinely earns its place, and it is narrower than most teams assume.
- Mostly knowledge lookup in your voice. RAG with light prompting covers it, and you can defer everything heavier.
Run those in order and the tool chooses itself. The full ten-gate version, including the accuracy and error-cost gates that decide the close calls, is written up in the MLDeep Fine-Tune vs Agents Framework on the hub, with a one-page PDF you can hand to a skeptical stakeholder.
Ready to fix your data foundation?
Book a free diagnostic call and find out where your stack stands.
Book a CallWhen fine-tuning genuinely wins the job
We are not against fine-tuning; we are against using it where it cannot work. It is the right tool for a real and specific set of jobs: a narrow, high-volume task where a small fine-tuned model matches a frontier model at a fraction of the per-token cost; a strict output format the base model keeps drifting from; a domain vocabulary generic models fumble. And there is a genuinely advanced use, distillation, where you run an agent to prove out a workflow, capture its successful traces, then train a small model on those traces to make the hot path cheaper at scale.
Notice the shared trait: high volume, narrow scope, no need to call tools. If your task is broad, low-volume, or has to act in your systems, fine-tuning is the wrong first move, and no amount of training changes that. If you want the sibling case, when a simple automation beats a model entirely, we cover it in AI agents versus automation.
The cost crossover, and why it is a range not a point
The question the platform team actually loses sleep over is the crossover: at what volume does distilling an agent's hot path onto a fine-tuned model beat running the full agent? The honest answer is a range, not a single number, and here is why. The dominant cost driver in an agent system is calls per task, and that swings by an order of magnitude between a simple lookup and a long multi-step workflow. Any TCO model that hands you a single crossover volume is hiding that swing.
Two things about the bands matter if you are going to put this in front of a CFO. First, lead with the mid number, because the low and high endpoints are not simultaneously achievable. A "low" TCO that takes the optimistic end of twenty independent variables at once, cheapest model, best utilization, fewest retries, lowest error cost, is a fantasy floor, not a real P10. Second, the crossover moves with your error cost and accuracy, which are the two variables most teams never write down. Our model computes the 24-month TCO with honest bands, priced at current 2026 rates from named vendors, Anthropic, OpenAI, RunPod, and Together AI, each linked with the date we read it, so you can verify every input in the assumptions table rather than trust it.
"The tell for a rigged cost model is a single crossover number. Calls per task swing by ten times across real workflows, so if your model does not show that as a range, it is selling you a conclusion, not computing one."
Anmol Parimoo, founder of MLDeep
Why we can make this call
We build the systems this framework describes, not slideware about them. We are a dbt Labs certified partner, and our production stack is the boring, real one: dbt for transformations, Terraform for infrastructure, BigQuery for the warehouse, and Python with LangChain for the agent layer. That matters here because the fine-tune-versus-agents question is usually settled by whether the data foundation and the tool-calling interfaces are solid, not by which model you pick. Most "the model is wrong" complaints we are called in on are data-foundation problems wearing a model costume.
If you want a verdict on your own stack before you commit engineering time, that is what our AI Stack Audit delivers: a clear read on whether your foundation can support production agents, with a 90-day roadmap.
Frequently asked questions
Can a fine-tuned model call tools or APIs?
No. Fine-tuning changes how a model responds, not what it can do. A fine-tuned model has no ability to call an API, query a database, or write to a record on its own. Tool calling is a property of the agent scaffold around a model, the loop and the tool definitions, not of the weights. If a task needs to act in your systems, you need an agent, and the underlying model can be fine-tuned or not depending on cost, but the fine-tuning is never what gives it the ability to act.
When is fine-tuning cheaper than building an agent?
When the task is narrow, high-volume, stable, and does not need to call tools, so a small fine-tuned model can replace a more expensive frontier model on that one job. It is also cheaper as the distilled hot path of an agent you already trust: you run the agent to generate good traces, then train a small model on them to cut per-task cost at scale. For a genuine tool-using workflow, fine-tuning is usually not cheaper, because it cannot do the job at all. Run both paths through the cost model with your volume and error cost before deciding.
What is the distillation crossover?
The distillation crossover is the monthly task volume at which distilling an agent's hot path onto a fine-tuned small model becomes cheaper than running the full agent. It is a range, not a fixed number, because the crossover depends heavily on calls per task, which varies by an order of magnitude across workflows, and on your accuracy and error-cost assumptions. The right way to use it is to find your own crossover band from your own inputs, then decide whether your projected volume clears it with margin.
