Fine-Tuning vs RAG vs Prompting: How to Choose the Right Approach for Your LLM App

The short answer: Start with prompting, add retrieval (RAG) when the model needs knowledge it doesn't have, and fine-tune only when you need behaviour that prompting can't reliably produce: a strict style or format, a narrow skill, or a smaller and cheaper model that matches a bigger one on one task. Fine-tuning is a poor way to teach a model new facts; retrieval is a poor way to teach it a new behaviour. Most production systems end up using prompting and RAG together, and a minority add fine-tuning on top.

Every team building with language models eventually has the same meeting. The prototype mostly works, but not quite. Answers are a little off, the tone isn't right, the model doesn't know about the product launched last month, and someone says the sentence that has launched a thousand GPU bills: "We should fine-tune it on our data."

Sometimes that's exactly right. More often it's an expensive detour that solves a different problem from the one the team actually has. The confusion comes from treating three very different techniques as interchangeable dials for "make the model better at our thing." They aren't. Prompting, retrieval and fine-tuning change different parts of the system, fix different kinds of failure, and fail in different ways.

This guide explains what each technique really changes, the failure each one is good at fixing, what they cost to build and maintain, and a decision process you can run in an afternoon instead of a quarter. If you've ever been in that meeting, it should give you better questions to ask.

Three techniques, three different jobs

The quickest way to stop confusing them is to ask where each one puts information.

• Prompting puts information in the request. Instructions, examples and context travel with every call. Nothing about the model changes.
• Retrieval-augmented generation (RAG) puts information in the request too, but chooses it dynamically: for each question, a search step finds relevant passages from your documents and inserts them into the prompt. The idea was formalised in a 2020 paper by Lewis et al., and it's now the default way to give models access to private or fast-changing knowledge.
• Fine-tuning puts information in the weights. You train the model further on examples of the inputs and outputs you want, and the resulting model behaves differently even with a short prompt.

That single distinction explains most of the trade-offs. Information in the request is easy to change, easy to inspect and costs tokens every time. Information in the weights is cheap per call, but slow and expensive to change, hard to inspect, and blended into everything else the model learned.

Knowledge problems and behaviour problems

Before choosing a technique, diagnose the failure. Almost every "the model isn't good enough" complaint falls into one of two buckets.

Knowledge problems are about what the model knows. It doesn't know your refund policy, your internal API, last week's release notes, or the customer's order history. It might confidently invent an answer, because it has never seen the real one. The fix is to give it the information.

Behaviour problems are about how the model acts. It knows enough, but the output is wrong in shape or manner: too long, the wrong tone, inconsistent formatting, ignoring a convention, using the wrong level of detail for the audience, or failing at a narrow task like classifying tickets into your specific categories. The fix is to change what it does with what it knows.

This distinction matters because the techniques map onto it unevenly. Retrieval is excellent at knowledge problems and does nothing for behaviour problems beyond what the retrieved text implies. Fine-tuning is good at behaviour problems and surprisingly weak at knowledge problems. Prompting can help with both, up to a point, and is by far the cheapest thing to try.

Write down ten real examples of failures before you decide anything. If most of them are "it didn't know X," you have a knowledge problem. If most are "it knew, but answered badly," you have a behaviour problem. Mixed results are common, and they usually mean you need more than one technique.

Prompting: the first tool, not the toy one

Prompting gets dismissed as the beginner option, which is a mistake. Modern models follow detailed instructions well, and a carefully engineered prompt solves a large share of behaviour problems for the price of an afternoon.

Things worth trying before anything heavier:

• Be specific about the output. Length, structure, tone, what to include and what to leave out. "Answer in two or three sentences, then a bulleted list of next steps" beats "be concise."
• Show examples. Two or three examples of ideal input and output pairs (few-shot prompting) often fix formatting and tone problems that instructions alone don't.
• Constrain the format mechanically. If you need JSON, use structured outputs rather than asking nicely. I covered that in the guide to getting reliable JSON from LLMs.
• Give the model a way out. Tell it what to do when it isn't sure, so it doesn't invent an answer.
• Try a stronger model or more reasoning effort. Before training anything, check whether a more capable model already does the job. If it does, you've learned the task is achievable, and you have a quality bar to aim for.

Prompting's limits show up in three ways. The prompt grows long and fragile, with every new edge case adding another paragraph. The model follows the instructions most of the time but not consistently enough. Or the long prompt makes every request expensive and slow. Those are the signals that a heavier technique might pay off. "We haven't really tried" is not one of them.

RAG: when the model needs to know things

Retrieval is the right answer to a knowledge problem in almost every case. The model gets the relevant facts at the moment it needs them, the facts can be updated by editing a document rather than retraining anything, and you can show users the sources behind an answer.

Retrieval is the strong choice when:

• The knowledge changes often: documentation, prices, policies, inventory, news.
• The knowledge is large, more than you can paste into every prompt.
• Different users should see different information, so access control matters.
• You need citations or an audit trail of where an answer came from.
• Being wrong is costly, and you want answers grounded in a specific source of truth.

Its weaknesses are mostly engineering ones. Retrieval adds moving parts (chunking, indexing, search, ranking) and every one of them can fail quietly. The most common cause of a bad RAG answer is that the right passage never reached the model, not that the model misread it. The guide on why RAG chatbots give bad answers walks through the common failures and fixes. Retrieval also adds tokens to every request, which shows up on the bill; the guide on cutting LLM API costs covers how to keep that in check.

One more option belongs here: long context. If your knowledge base is small and stable, say a few hundred pages, you may be able to put all of it in the prompt and skip retrieval entirely. With prompt caching, repeated long prompts can be surprisingly affordable. It's worth testing before you build an indexing pipeline for a corpus that would fit in a single request.

Fine-tuning: when you need the model to behave differently

Fine-tuning trains an existing model further on your examples. With modern techniques you rarely update every weight. Methods like LoRA (Hu et al., 2021) train small additional matrices instead, which cuts the number of trainable parameters dramatically and makes fine-tuning feasible on modest hardware for open-weight models. Several API providers also offer hosted fine-tuning for some of their models; OpenAI's fine-tuning guide is one example of what that looks like.

Fine-tuning earns its cost in a fairly specific set of situations:

• Consistent style or format that prompting can't hold reliably, for example a house writing style across thousands of outputs, or a rigid output structure on a model that lacks constrained decoding.
• A narrow, repetitive task with lots of labelled examples: classification into your own categories, extraction from a specific document type, routing tickets to the right team.
• Shrinking the prompt. If every request carries a two-thousand-token prompt full of instructions and examples, a fine-tuned model can learn that behaviour and run with a short prompt, which reduces cost and latency.
• Distillation. Training a smaller, cheaper model on high-quality outputs from a larger one, so the small model matches the large one on your specific task. Check your provider's terms before doing this with their outputs; some restrict it.
• Domain language the base model handles poorly, such as a specialised notation or jargon-heavy field, where examples teach patterns that instructions can't describe.

The trap: using fine-tuning to teach facts

This is the most common and most expensive misunderstanding. It feels intuitive that training a model on your documentation will make it "know" your documentation. In practice, models learn new facts from fine-tuning slowly and unreliably. A study by Ovadia and colleagues, "Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMs", found that retrieval consistently outperformed unsupervised fine-tuning for injecting knowledge, and that models struggled to learn new facts through fine-tuning unless they saw many variations of the same fact.

Worse, a model fine-tuned on facts doesn't know which of its answers came from your data and which it made up, so it can't cite sources, and when the facts change you have to train again. If your goal is "the model should know our product," the answer is retrieval. If your goal is "the model should answer the way our best support agent answers," fine-tuning becomes interesting, ideally on top of retrieval that supplies the facts.

Comparing the three side by side

Notice that fine-tuning is the only option that ties you to a specific model version. When a better base model comes out, prompts and retrieval pipelines usually carry over with light tuning. A fine-tuned model has to be trained again on the new base, and evaluated again from scratch. Given how quickly models improve, that maintenance cost is easy to underestimate. Teams regularly find that a newer base model with a good prompt beats their fine-tuned older one.

The hidden cost of fine-tuning: data

Training runs get the attention, but the expensive part of fine-tuning is the dataset. You need examples of exactly the behaviour you want, consistently, in quantity. That means:

• Collecting inputs that represent real traffic, including the awkward cases.
• Producing ideal outputs for each one, written or carefully reviewed by people who know what good looks like.
• Making them consistent. If two reviewers disagree about the right answer, the model learns the disagreement. Inconsistent data produces an inconsistent model.
• Keeping a held-out evaluation set the model never trains on, so you can tell whether it improved or just memorised.

How many examples you need depends heavily on the task and the method, so don't trust a universal number. A narrow formatting behaviour might be learned from a few hundred good examples; a complex skill can need far more. The practical approach is to start with a modest set, measure on your held-out data, and add more only where the evaluation shows weakness.

There's a quieter benefit here. Building a clean, labelled dataset forces you to define precisely what good output looks like. Teams that do this work often discover that a sharper prompt, written with the same clarity, gets most of the improvement they wanted from fine-tuning.

A decision process you can run this week

Rather than debating in the abstract, run the techniques in order, each one gated by evidence.

1. Build a small evaluation set first. Fifty to a hundred real inputs, with what a good answer looks like for each. Without this, you can't tell whether any change helped. The guide on testing LLM features with evals shows how to set one up quickly.
2. Get the best result you can with prompting. Clear instructions, a few examples, structured output where it applies, and the most capable model you can afford. Record the scores.
3. Diagnose what's still failing. Sort the remaining failures into knowledge and behaviour problems.
4. For knowledge failures, add retrieval (or long context, if the knowledge is small and stable). Measure again.
5. For behaviour failures that survive good prompting, consider fine-tuning. Estimate the data effort honestly before committing.
6. If quality is fine but cost or latency isn't, consider fine-tuning or distilling a smaller model, then compare cost per successful task, not just per token.

The important property of this process is that each step is cheaper than the next and produces evidence for the next decision. Teams that skip straight to fine-tuning lose that evidence, and often can't say afterwards whether the expensive step was what helped.

Combining them in practice

In mature systems the question is rarely "which one," it's "which combination." Some common patterns:

• Prompting plus RAG is the default for assistants over documentation, support knowledge bases and internal search. It covers most production use cases.
• RAG plus a fine-tuned model works when the answers need your facts and a particular voice or structure. Retrieval supplies the facts; fine-tuning shapes how they're presented. Train the model on examples that include retrieved context, so it learns to use the passages rather than ignore them.
• A fine-tuned small model as a router or classifier in front of a larger general model. The small model handles the cheap, repetitive decision (which team, which intent, which tool), and the large model handles open-ended work.
• Distillation for scale. A large model with a careful prompt generates high-quality outputs; those become training data for a smaller model that serves high-volume traffic. The large model remains the reference you evaluate against.

Evaluating a fine-tuned model before you ship it

A fine-tuned model that scores well on its training data proves almost nothing. Before it replaces anything in production, it should pass three kinds of checks.

Target task on held-out data. Compare the fine-tuned model against your best prompted baseline on examples it never saw during training. The comparison has to be fair: same inputs, same scoring, and the baseline should be your strongest prompt on your most capable affordable model, not the weak first draft you started with. If the fine-tuned model doesn't clearly win here, stop.

Everything else it still needs to do. Fine-tuning shifts a model's behaviour, and that shift can leak into areas you didn't train on. A model tuned to write terse support replies may become terse everywhere, including in places where users need detail. A model tuned on one output format may start forcing that format onto unrelated requests. Keep a small regression set covering the other things your feature handles (out-of-scope questions, refusals, edge cases, different languages if you support them) and run it every time.

Real traffic, gradually. Offline results are an estimate. Route a small percentage of real requests to the fine-tuned model, compare user feedback, escalation rates and cost per completed task against the existing version, and only then widen the rollout. Keep the previous version deployable, because rolling back a fine-tune is much easier than diagnosing one under pressure.

It's also worth recording what the fine-tune cost end to end: the hours spent collecting and labelling data, the training runs, the evaluation work and the expected maintenance when the base model changes. That number is what you compare against the alternative of a better prompt on a newer model, and it's the number most teams never write down.

Common mistakes to avoid

• Fine-tuning before trying a better prompt. The single most common waste of time and money.
• Fine-tuning on documentation to add knowledge. Use retrieval for facts.
• No held-out evaluation. Without it, a fine-tuned model that memorised the training set looks like a success.
• Ignoring regressions. Fine-tuning for one behaviour can degrade others. Evaluate on general tasks the system still needs to handle, not just the target task.
• Treating RAG as plug-and-play. Retrieval quality needs measuring and tuning like any other search system.
• Forgetting the upgrade path. Plan for how you'll move to the next base model before you invest heavily in a fine-tune of the current one.

Frequently asked questions

Should I use fine-tuning or RAG for my chatbot?

If the chatbot needs to answer questions about your products, documents or policies, use RAG: it supplies current facts, can cite sources and is easy to update. Consider fine-tuning only if, after good prompting and retrieval, the bot's tone, format or narrow task performance is still not good enough. Many production chatbots use RAG alone.

Can fine-tuning teach a model new knowledge?

Only weakly. Research comparing the two found that retrieval consistently outperformed unsupervised fine-tuning for adding knowledge, and that models learn new facts from fine-tuning unreliably. Fine-tuning is better at changing behaviour, style and format than at teaching facts, which is why retrieval is the usual choice for knowledge.

Is prompt engineering enough for production?

Often, yes, especially combined with structured outputs and retrieval. Prompting is limited when instructions become long and fragile, when the model follows them inconsistently, or when a long prompt makes every request expensive. Those are the signals to consider fine-tuning; skipping prompt work entirely is not.

How many examples do I need to fine-tune a model?

It depends on the task and method. A narrow formatting or style behaviour can sometimes be learned from a few hundred high-quality, consistent examples, while complex skills need many more. Start with a modest set, measure on held-out data the model never trained on, and add examples where the evaluation shows weaknesses.

Can I combine RAG and fine-tuning?

Yes, and it's a common pattern when you need both your facts and a particular behaviour. Retrieval supplies the up-to-date knowledge, and fine-tuning shapes how the model uses and presents it. Train the fine-tuned model on examples that include retrieved passages, so it learns to rely on them.

The takeaway

"Fine-tune it on our data" sounds like the serious engineering option, and prompting sounds like the hobbyist one. The reality is close to the opposite: prompting and retrieval solve most problems faster, cheaper and more transparently, and fine-tuning is a specialised tool for behaviour that the other two can't produce.

So diagnose before you build. Sort failures into knowledge and behaviour, measure against a real evaluation set, and climb the ladder one rung at a time. When you do reach for fine-tuning, you'll know exactly what it's for, and you'll be able to prove it worked.