RAG vs Fine Tuning: Company Knowledge and Behavior

Use RAG when the model lacks the right company facts; consider fine-tuning when it has those facts but repeatedly uses the wrong style or decision rules. RAG vs fine tuning is a choice between supplying evidence and teaching behavior, and the 3 business cases below show why a cheap training job is not automatically the cheaper system.

Documentation, model availability and list prices verified on 16 October 2026. The workloads and arithmetic below are illustrative planning examples, not measured customer results.

RAG vs Fine Tuning: Choose the Failure You Need to Fix

Retrieval fixes access to knowledge; fine-tuning targets recurring behavior. That distinction tells you where to spend effort before it tells you which service to buy.

Retrieval-augmented generation, or RAG, fetches relevant material and puts it into the model's working context before it answers. Fine-tuning continues training a pretrained model on examples of the responses or decisions you want, changing its learned parameters. Think of a support employee: retrieval supplies the current reference binder; fine-tuning is coaching with corrected examples. Neither substitutes for clear instructions.

The distinction is a useful default, not a law that training cannot teach facts. A fine-tune can learn factual associations. It still does not give you a dependable database of current policies, document permissions and source references. AWS's comparison separates these knowledge and customization roles.

Run the evidence test: take a failed question and manually supply the correct, current passage. If the answer becomes acceptable, repair retrieval. If it still fails, inspect the instructions, task difficulty and response behavior. Fine-tuning becomes a candidate only after that simpler baseline has been improved.

For builders, this separates the document pipeline from the training pipeline. For operators, it identifies who owns the problem: the knowledge owner or the person approving examples. For buyers, it supplies a useful acceptance test: ask the vendor to show which failure the extra component fixes.

RAG vs Fine-Tuning vs Prompt Engineering

Start with the smallest intervention that passes your task's acceptance test. Prompt engineering means improving the instructions, examples and context supplied with a request. It remains part of both retrieval and tuning systems.

RAG vs Prompt Engineering

Use a prompt when the necessary material is already in the request. Add retrieval when the application must find that material among company sources. A better instruction cannot reveal a cancellation clause the model never receives; a search index cannot resolve contradictory instructions about how to apply it.

Prompt Engineering vs Fine-Tuning

Keep changing brand rules and occasional tasks in prompts. Consider tuning when a stable, repeated behavior still fails despite clear instructions and representative examples. Google's tuning guidance recommends establishing the prompt baseline first.

Fine-Tuning vs Transfer Learning

Fine-tuning is a form of transfer learning: you adapt capabilities learned during earlier training to your task. It is not another source-retrieval technique. For this decision, the relevant question is whether you need to change learned behavior at all.

Choose a Supported Model Before You Build

Check both model support and account eligibility before preparing a managed fine-tuning project. A model name in an old tutorial is insufficient.

OpenAI's self-serve fine-tuning platform closed to organizations that had never used it on 7 May 2026. From 2 July 2026, organizations that had not run a fine-tuned model in the preceding 60 days also lost the ability to create jobs. Its current notice says active existing customers lose new-job creation on 6 January 2027; existing fine-tuned inference lasts only until the underlying model is deprecated. This is not the default route for a new company project. OpenAI deprecation notice.

Google Gemini 3.1 Flash-Lite is a candidate whose current Google Cloud documentation explicitly supports supervised fine-tuning, meaning training from labeled examples. Google's current documentation calls the platform Gemini Enterprise Agent Platform; the former Vertex AI tuning documentation redirects there. The model supplies a concrete budget below, not a claim that it will win your task. Supported tuning models.

The deployment constraint matters: this model's tuning runs in us-central1 or europe-west4, while tuned serving uses the us or eu multi-region endpoints. Budget those endpoints rather than copying the cheaper global base-model rate. Google also excludes supervised fine-tuning from its service-level guarantee. Tuning limitations.

Price basis for the examples: standard non-global text inference is $0.275 input, $0.0275 cached input and $1.65 output per million tokens; tuning is $3 per million training tokens, including repeated epochs, and tuned inference is 1.5 times the base rate, giving $0.4125 input and $2.475 output. These are Google Cloud list prices, not Gemini app subscription prices. Google pricing.

An epoch is one pass through the training dataset. All examples assume one model call per item and billable output totals that include any reasoning tokens. They exclude tax, engineering, labeling, evaluation calls, retries, source-document storage, document scanning and additional search services. Those items need their own budget; a training invoice is not an implementation quote.

Three Business Cases: What You Build and What You Pay

Support Answers From Company Documents: Start With RAG

A software company's support assistant needs current evidence before it needs training. If a customer asks whether cancellation takes effect immediately, the answer must follow the current policy and any applicable contract exception.

Build an ingestion process that reads approved documents, divides them into useful passages, and preserves the source URL, effective date, product and access permissions. At request time, restrict the search to material the customer may see, retrieve the relevant passages, and ask the model to answer from that evidence with citations. Replace changed passages and remove deleted ones. A source changing does not instantly update your index.

OpenAI text-embedding-3-small turns text into numerical representations used for similarity search. It costs $0.02 per million input tokens and produces 1,536 dimensions by default. Embedding the documents makes them searchable; it does not train the answer-writing model. Model price, embedding dimensions.

Cloudflare Vectorize is the vector database in this example, storing those representations and finding similar ones. Its wall is evidence quality: a nearby passage can be outdated or irrelevant. Your application must enforce access and verify that the selected text supports the answer.

Build charge: assume 10,000 passages averaging 500 tokens after chunking. Initial embedding is 5 million tokens × $0.02 = $0.10. That excludes parsing and the engineering work above.

The index stores 15.36 million dimensions, exceeding Vectorize's 5 million free allowance. The paid allocation includes 10 million stored dimensions and 50 million queried dimensions per month; overages are $0.05 per 100 million stored and $0.01 per million queried. Cloudflare's published formula gives 30.72 million queried dimensions for this index and 10,000 monthly queries, inside that allocation. The stored overage is $0.00268/month. Allowances are assumed unused elsewhere in the account. Vectorize pricing.

Monthly component budget: replace 1,000 passages and embed 10,000 questions averaging 100 tokens, costing $0.03. Generate 10,000 answers, each with 2,500 total input tokens and 400 output tokens, costing $13.475. Add the $5 Workers Paid account minimum and the storage overage: approximately $18.51/month, plus the initial embedding charge. Worker execution is assumed to remain within its included allowances. Workers pricing.

The operational test is whether the answer uses the correct effective policy, cites a passage that supports it, and declines to invent an exception. Also test deleted documents and a customer attempting to access another customer's contract. Tune later only if correct evidence still produces a repeatable behavior failure.

A House Writing Style: Try Examples and Caching Before Training

A stable editorial voice can justify tuning; a brand still arguing about its voice cannot. Consider a company producing product emails that must sound direct, avoid exaggerated claims, and end with a concrete next step.

Start with a style guide and approved examples in the prompt. If the result repeatedly misses subtle editorial choices, build training pairs: the original brief and facts as input, the editor-approved draft as output. An archive of finished emails alone does not show which input should produce which response.

For an illustrative brief announcing Saturday support, an approved target might be: “You can now reach support on Saturdays. Open chat when you need help.” The target demonstrates the voice; the brief provides the facts.

Separate style from factual content. Train the preference for short openings; supply this week's product details at request time. Keep untouched briefs for evaluation, and ask editors to judge unsupported claims, tone, required content and the amount of rewriting needed. Do not treat a text-similarity score as proof of good brand writing.

Build charge: 1,000 pairs averaging 1,600 total tokens × 3 epochs = 4.8 million training tokens, or $14.40 for one job. These are budget assumptions, not a recommendation that this dataset size or epoch count will pass evaluation.

Monthly model usage: 10,000 drafts, each with 1,200 input tokens and 400 output tokens, cost $14.85 on the tuned model. The same base model with 4,800 additional style/example tokens per request costs $23.10 without cache hits. At the steady-state limit where every repeated prefix hits the implicit cache, that prompt-based baseline falls to $11.22, plus cache-miss and warmup costs. These calculations use the Google rates above.

That reverses the spending argument. Tuning may still earn its place by reducing editorial rework, but it is not automatically the cheaper way to avoid repeatedly paying for a long style prompt. Measure cache usage and editor acceptance before committing to a training pipeline.

A Ticket Classifier: Tune Stable Decisions, Retrieve Changing Rules

A ticket classifier is a strong tuning candidate when its labels are stable and its recurring mistakes are measurable. Suppose the allowed labels are billing, access, bug and review.

The message “I cannot open my invoice” illustrates the hard part. It contains a billing word, but the responsible queue might be access. Agree on that boundary with support leaders, then label representative messages, including ambiguous ones. Keep related messages from the same conversation together when splitting training and evaluation data, and include later tickets to expose changes in language or products.

The output must be an allowed label, but valid formatting alone is not success. Measure missed urgent cases, wrong-queue assignments, unnecessary review and the downstream reassignment burden separately. The model's own statement that it is confident is not a calibrated routing threshold.

Build charge: 5,000 labeled examples averaging 300 total tokens × 3 epochs = 4.5 million training tokens, or $13.50 for one job. Monthly model usage: 100,000 tickets at 250 input and 20 billable output tokens each costs approximately $15.26 tuned, compared with $10.18 for the same short prompt on the base model. The tuned model must earn that premium through better decisions. Google pricing.

Use application code to reject invalid labels and send ambiguous cases to review. Google's tuning documentation warns that adding controlled generation, which constrains output to a schema, at inference can create a mismatch with training, so evaluate the exact combination you intend to deploy. Known tuning issues.

If queue ownership changes frequently, keep the current routing policy outside the weights. Retrieve it or supply it in the prompt. If the label boundaries themselves keep changing, postpone tuning until the operation is stable enough to teach.

What Long Context and Prompt Caching Change in 2026

A small, stable document collection can now be worth sending directly before you build a retrieval service. Long context and caching were not invented in 2026; they change the baseline you should compare against today.

A context window is the amount of material the model can work with in a request. Gemini 3.1 Flash-Lite, released on 7 May 2026, lists a 1,048,576-token context window. That is capacity, not a guarantee that every exception in a large handbook will be used correctly. Leave room for instructions, the question and generation, then test the difficult passages. Model specification.

RAG vs CAG

Cache-augmented generation, or CAG, describes preloading knowledge and reusing cached context instead of performing document retrieval for every question. It is useful when the relevant corpus fits, changes infrequently, and has the same access permissions for everyone receiving it. Prompt caching reuses processing of repeated input; it does not put those facts into model weights or guarantee a fresh answer.

For an assumed 80,000-token handbook reused across 10,000 requests, the handbook's input component alone is $220 without caching or $22 at the all-hit cached rate. New questions, output, cache misses and cache creation still cost money. Keeping an explicit cache of that handbook for 720 hours adds $57.60 in storage, using the listed $1 per million token-hours rate. Google cache pricing.

Implicit caching has no storage charge, but hits are not guaranteed. Put shared material first, keep its prefix consistent, and inspect cachedContentTokenCount in responses. This model needs at least 4,096 tokens of eligible context, so a short ticket prompt should not be budgeted as cached. Caching documentation.

The boundary is practical. Use direct long context for a bounded handbook you can refresh and authorize as a whole. Use retrieval when the company corpus is larger, changes in pieces, or contains documents with different permissions. Cache a common instruction prefix inside a RAG system when it helps; these choices can coexist.

Do not assume every cache feature carries across to a fine-tune. Google's current tuned-model caching documentation says explicit caching is unsupported for fine-tuned models, while implicit support depends on the model version.

Use Both When Facts Change and Behavior Still Fails

Combine retrieval and tuning when each fixes a separately demonstrated failure. A support assistant might retrieve the current cancellation clause while a tuned model follows the company's preferred response structure: answer, evidence, permitted next action, or escalation.

Train with examples that resemble the evidence the model will receive in production. Include missing, irrelevant and conflicting passages so the desired behavior includes asking for clarification or declining to invent an answer. At runtime, access checks still run before restricted material reaches the model.

Evaluate retrieval and response behavior separately. Correct citations, permission boundaries and source refresh remain application responsibilities; a fine-tune does not enforce them for you.

If finding the evidence requires follow-up searches, the agentic RAG guide explains when a bounded retrieval loop earns its complexity. Once a stable behavior gap justifies training, how to fine-tune an LLM covers dataset preparation, checkpoints and evaluation.

What Is Overhyped?

Neither “RAG always wins” nor “fine-tuning makes the model know our company” is a sound build specification. Retrieval can fail through stale sources, poor passage selection or missing context. Tuning can learn contradictory examples and reproduce an obsolete rule more consistently.

Cheap training compute is also easy to overvalue. The invoice excludes the work of deciding what good looks like, resolving label disagreements and keeping a test set useful. Larger context windows remove a capacity constraint; they do not establish correct reasoning over every supplied fact.

The claim worth demanding is narrower: on your held-out work, this component fixes a named failure while preserving the other requirements. If the existing prompt already passes, leave the system alone.

When to Use RAG vs Fine-Tuning

Act now on missing evidence; wait on training until you can name and measure the behavior gap. Operators with changing policies should fix source ownership and refresh. Editors with settled examples and recurring rewrites should compare prompting against tuning. A classifier with changing labels should wait. A working prompt-based workflow is unaffected by the availability of another optimization technique.

Make this the Monday move:

1. 1

   Save the failures before changing the system

   Select representative support questions, writing briefs or tickets. Record the approved answer or decision, and keep these evaluation cases outside the training set.

2. 2

   Supply the missing evidence manually

   Repeat the task with the correct passage and clear instructions. Classify the remaining error: missing knowledge, unclear rule, wrong behavior or a task the model cannot yet handle.

3. 3

   Price the smallest candidate

   Compare the current prompt, a cached or long-context prompt, retrieval, and only then a supported fine-tune where needed. Count actual billable tokens, cache hits, retries and human correction work.

4. 4

   Ship against the task's acceptance test

   Choose the simplest design that passes. Assign an owner for document refresh or training-example review, record the deployed versions, and preserve a rollback path.

Frequently Asked Questions

What is fine-tuning in AI?

Fine-tuning adapts a pretrained model using additional training examples. For company workflows, the useful target is usually a repeatable response style or task behavior.

How expensive is it to fine-tune a model?

The writing example's single Google training job is $14.40 under its stated token and epoch assumptions. Labeling, evaluation, serving and engineering are separate costs, and unsuccessful experiments add more jobs.

What is better than RAG?

A direct prompt can be simpler when it already contains all necessary evidence. Fine-tuning can help a stable behavior problem. Neither replaces retrieval when the system must find current, authorized facts.

What are the key differences between a fine-tuning model and a RAG-based approach?

Fine-tuning changes learned parameters through training. RAG changes the evidence supplied at request time. Updating a source index, retraining a model and proving that an answer has a valid citation are different jobs.

Is fine-tuning debunked?

No, if you mean adapting machine-learning models. Its value is task-dependent and should be demonstrated against a prompt baseline using examples excluded from training.

How was Chatgpt fine-tuned?

OpenAI's original ChatGPT announcement, published in 2022, described supervised dialogue examples followed by reinforcement learning from human feedback. That historical process is separate from today's business API tuning eligibility.

What did Stephen Hawking say about fine-tuning?

That is a question about cosmological fine-tuning, a different meaning of the term. It does not concern adapting a language model. The business decision here is about retrieval and training, not a claim about the universe.

Is fine-tuning still a thing?

Yes. Google's current documentation lists supported models, including the Gemini model used here. Provider availability changes, so verify the specific model and your account's eligibility before preparing a job.

Which model is best for fine-tuning?

Choose a supported base model that already handles the core task and passes your deployment constraints. Gemini 3.1 Flash-Lite is a priced example here, not a tested universal winner; your held-out cases decide.

For practical decisions about building with AI, join the newsletter.