Explainer · RAG vs fine-tuning
RAG vs Fine-Tuning: Which One Your LLM Actually Needs
The most expensive sentence in applied AI is "let\u2019s just fine-tune it" — said about a knowledge problem. The second most expensive is "retrieval will fix it" — said about a behavior problem. This page separates the two, with the costs and maintenance each one really carries, so the decision can be made on evidence instead of fashion.
| Dimension | RAG (retrieval-augmented generation) | Fine-tuning |
|---|---|---|
| What it changes | The context the model sees at answer time | The model\u2019s weights — its default behavior |
| Knowledge freshness | As fresh as your index; update a document and the next answer uses it | Frozen at training time; new facts need a new training run |
| Citations | Native — answers can point to the retrieved source | None — the model cannot say where a "fact" in its weights came from |
| Consistency of format & tone | Depends on prompting every time; drifts under edge cases | Baked in; holds under edge cases because it is the default behavior |
| Upfront cost | Retrieval infrastructure: ingestion, embeddings, index, evaluation | Dataset production (demonstrations, preference data) plus a training run that is usually the smallest line |
| Running cost | Index storage, embedding and retrieval per query, freshness operations | Tuned-model hosting (can bill hourly, always-on) plus re-tuning as the domain shifts |
| Maintenance burden | Document pipelines and retrieval quality — a data-operations problem | Dataset versioning, eval sets and retraining cadence — a modeling problem |
| Characteristic failure | Retrieves the wrong passage and answers confidently from it | States stale or invented facts fluently, with no way to cite or patch them |
| Data it needs from you | Your documents, kept current | Examples of ideal behavior: demonstrations and graded outputs, written or reviewed by domain experts |
When RAG wins
- The knowledge changes. Prices, policies, inventory, account states, regulations: anything with a newer version coming. Retrieval reads the current version; a fine-tune remembers an old one.
- Answers must show their work. Support, legal, medical and financial answers that cite sources are a retrieval feature, not a model feature.
- The corpus is huge or access-controlled. Retrieval respects permissions at query time; weights cannot un-know a document a user should not see.
- You are still learning what users ask. Retrieval logs show the real questions; they become the eval set — and eventually the fine-tuning dataset — for whatever you build next.
When fine-tuning wins
- Output shape must never drift. Structured outputs, house formats, regulated phrasing, consistent tool-calling: behavior that has to hold on the worst input, not just the average one.
- The domain has a way of reasoning. How your analysts structure a review, how your support team triages, how your clinicians document. Demonstrations teach it; prompts only request it.
- A smaller model must do a bigger model\u2019s job. The classic cost play: fine-tune a small open model on a frontier model\u2019s (or your experts\u2019) demonstrations for one narrow task, then serve it cheaply — the training prices on our fine-tuning services page show how inexpensive the run itself has become.
- Latency and prompt size matter. Behavior in the weights is behavior you stop paying to describe in every prompt.
The hybrid is the enterprise answer
Enterprise AI agents — the financial-services case is the cleanest example — almost always land on both, with a strict division of labor. Retrieval owns the facts: client data, product terms, filings and market information live in systems that update continuously, and the agent\u2019s answers can be traced to them. Fine-tuning owns the conduct: the agent formats its analysis the house way, applies compliance language verbatim, calls tools in the approved sequence, and declines what it must decline — because it was trained on demonstrations of exactly that conduct, written and graded by people accountable for it. Teams building that loop need two supply chains, not one: document pipelines for retrieval, and human feedback services for the demonstrations and graded outputs the conduct is trained on. The enterprise fine-tuning guide covers the boundary, retention and ownership questions that come with it.
A decision checklist, in order
- Write down the failures you are trying to fix, from real outputs. Knowledge failures or behavior failures?
- Build the eval set first — 100–500 real cases with graded ideal answers. Without it, every later claim is vibes.
- Ship prompting + retrieval. Measure against the eval set.
- If behavior failures remain, collect demonstrations for exactly those cases and fine-tune the smallest model that could plausibly hold the behavior.
- Re-measure. Keep the hybrid only if both halves earn their maintenance.
Frequently asked questions
RAG vs fine-tuning: which is better?
Neither, in the abstract — they fix different problems. RAG (retrieval-augmented generation) gives a model access to facts it can look up at answer time: current, citable, changeable without retraining. Fine-tuning changes the model’s behavior: format, tone, terminology, reasoning patterns and tool use, baked into the weights. If the model gives wrong answers because it lacks your facts, that is a retrieval problem. If it gives wrong-shaped answers despite having the facts in front of it, that is a behavior problem — and behavior is what fine-tuning is for.
What is the best practice for RAG vs fine-tuning in enterprise AI agents, such as financial services?
The hybrid, with clear job separation. Retrieval carries everything that changes or must be cited — account data, policy documents, filings, rate sheets — because it can be updated the moment the source changes and every answer can point at its source. Fine-tuning carries everything that must be consistent — the agent’s output format, its compliance phrasing, its tool-calling discipline, the house way of structuring an analysis. Financial-services teams feel the split most sharply: an agent that quotes last quarter’s numbers from its weights is a liability, and an agent whose disclosures drift in format is a different liability. RAG fixes the first; fine-tuning fixes the second.
Is fine-tuning cheaper than RAG?
The training run usually is; the system usually is not. A focused fine-tune at published prices can cost tens to hundreds of dollars in training compute (dataset tokens × epochs at $0.34–$25 per million), but it commits you to dataset production, evaluation and re-tuning as the world changes, plus hosting for the tuned model. RAG has no training bill but a permanent infrastructure and maintenance surface: ingestion, chunking, embeddings, index freshness and retrieval evaluation. Compare total cost of ownership over a year of changes, not the price of the first demo.
Can you combine RAG and fine-tuning?
Yes — that is the standard production architecture for serious deployments. Fine-tune a model on your domain’s demonstrations so it reasons, formats and uses tools the way your team does; retrieve the facts it should reason over at runtime. The fine-tune makes retrieval output usable (the model knows how to read your documents and what to do with them), and retrieval keeps the fine-tune honest (the model never has to memorize anything that can change).
When should I not fine-tune?
When the knowledge you want to add changes faster than you can retrain; when you need answers to cite their sources; when you do not yet have evaluation data proving the base model fails on behavior rather than knowledge; and when a better prompt plus retrieval already passes your evals. Fine-tuning is also the wrong first move before you have a held-out eval set — without one, you cannot tell a better model from a merely different one.
Sources
Cost figures are the providers’ published list prices, read November 2026.
- Together AI pricing — what a fine-tuning run costs at published per-token rates
- Vertex AI pricing — model tuning prices and tuned-endpoint billing (Google Cloud)
- OpenAI API pricing — the fine-tuning wind-down, and why platform choice now matters
- Fireworks AI pricing — training and serving rates for fine-tuned open models
Facts on this page were drawn from the public sources above, checked November 2026. If you spot an error, tell us at the corrections address in the footer — we fix sourced errors and note the change.
Last updated: November 8, 2026