AWS Builder Center

Builder Center

Fine-Tuning vs RAG on AWS: How to Choose

Fine-Tuning vs RAG on AWS: How to Choose

Fine-tuning makes a model smarter. RAG makes it informed. A practical 4-question framework with real AWS costs and code included.

Fine-Tuning vs RAG on AWS — How to Choose the Right AI Strategy (With a Decision Framework)

TL;DR: Fine-tuning makes a model smarter. RAG makes a model informed. They solve different problems, cost very differently, and the wrong choice will waste months of work. Here's a practical framework for choosing — based on real decisions I've had to make at a Premier AWS Partner.


The Question I Get Asked Most

Every time I talk about AI at STC, someone asks: "Should we fine-tune the model or just use RAG?"
It sounds like a technical question. It's actually a product question, a cost question, and a data governance question all rolled into one. This post gives you a decision framework you can apply immediately.


What Each Actually Does


RAG
Fine-Tuning
What it changes
What the model knows at query time
The model's weights (its skills and style)
Analogy
Giving an employee a reference manual
Sending an employee to a training course
Updates
Instant — sync new docs to S3
Requires a new training job (hours to days)
Data needed
Documents (any format)
Labeled examples (Q&A pairs, completions)
AWS service
Bedrock Knowledge Bases
Bedrock Custom Model / SageMaker
Typical cost
Low ($5–50/month at low volume)
High ($200–2000+ per training run)


The Decision Framework

(See the decision diagram in this post)
Ask these four questions in order:

Question 1: Is this a knowledge problem or a behaviour problem?

Knowledge problem: The model gives wrong answers because it doesn't have your data. → Example: "What is our company's refund policy?" — the model can't know this. → Use RAG.
Behaviour problem: The model gives answers in the wrong format, wrong tone, or wrong domain style. → Example: "Write a medical discharge summary" — the model can write one, but not in your hospital's exact template and clinical shorthand. → Use fine-tuning.
If you're not sure, start with RAG. It's faster and cheaper to validate.


Question 2: How often does your data change?

Changes frequently (weekly/daily — pricing, policies, inventory): → RAG wins. Re-syncing an S3 bucket takes minutes. Retraining a model takes hours and costs money every time.
Stable (medical codes, legal precedents, domain vocabulary that rarely changes): → Fine-tuning becomes more viable since you're not constantly retraining.


Question 3: Do you have labelled training data?

Fine-tuning requires hundreds to thousands of high-quality examples in the format:
{"prompt": "Write a discharge summary for...", "completion": "Patient: John D., DOB..."}
Creating this data is expensive and time-consuming. If you don't have it, fine-tuning is blocked before you even start.
RAG requires only raw documents — PDFs, Word docs, HTML pages. Much easier to assemble.


Question 4: What's your latency and cost tolerance?

RAG: Each query retrieves chunks first, then generates. Adds ~200–500ms latency. Cost scales with query volume.
Fine-tuning: The model has knowledge baked in — no retrieval step. Faster at inference, but you pay upfront for training and ongoing for a dedicated endpoint.
For most enterprise use cases at scale, RAG is cheaper unless query volume is extremely high (millions of queries/day).


AWS Implementation Side-by-Side

RAG on AWS (Amazon Bedrock Knowledge Bases)

import boto3



client = boto3.client('bedrock-agent-runtime')



# Query your documents — no training needed

response = client.retrieve_and_generate(

    input={"text": "What is the Q1 2026 refund policy?"},

    retrieveAndGenerateConfiguration={

        "type": "KNOWLEDGE_BASE",

        "knowledgeBaseConfiguration": {

            "knowledgeBaseId": "YOUR_KB_ID",

            "modelArn": "arn:aws:bedrock:us-east-1::foundation-model/anthropic.claude-3-sonnet-20240229-v1:0"

        }

    }

)



print(response['output']['text'])

# → "According to the Q1 2026 policy (source: policies/refunds-q1.pdf), 

#    customers may return items within 30 days..."
Setup time: 1–2 hours Data required: Your documents in S3 Update process: Re-sync S3 bucket


Fine-Tuning on AWS (Amazon Bedrock Custom Models)

import boto3



bedrock = boto3.client('bedrock')



# 1. Prepare your training data in JSONL format and upload to S3

# Each line: {"prompt": "...", "completion": "..."}



# 2. Create a fine-tuning job

response = bedrock.create_model_customization_job(

    jobName='clinical-notes-finetune-v1',

    customModelName='clinical-notes-claude-v1',

    roleArn='arn:aws:iam::YOUR_ACCOUNT:role/BedrockFineTuningRole',

    baseModelIdentifier='anthropic.claude-3-haiku-20240307-v1:0',

    customizationType='FINE_TUNING',

    trainingDataConfig={

        's3Uri': 's3://your-bucket/training-data/clinical-notes.jsonl'

    },

    outputDataConfig={

        's3Uri': 's3://your-bucket/fine-tuning-output/'

    },

    hyperParameters={

        'epochCount': '3',

        'batchSize': '8',

        'learningRateMultiplier': '1.0'

    }

)



job_arn = response['jobArn']

print(f"Fine-tuning job started: {job_arn}")

# Takes 2-8 hours depending on data size
Setup time: Days to weeks (data preparation is the bottleneck) Data required: 500–10,000 labelled examples Update process: Full retraining job


Real Examples: Which to Choose

Use RAG when:

  • Internal knowledge base / document Q&A ✅
  • Customer support bot that needs current product info ✅
  • Contract review assistant (search across legal docs) ✅
  • Any time your data changes more than monthly ✅

Use Fine-Tuning when:

  • You need a specific output format consistently (discharge summaries, code in a house style) ✅
  • Domain-specific vocabulary matters (medical, legal, financial jargon) ✅
  • You want the model to develop a branded voice/tone ✅
  • Latency is critical and you can't afford the retrieval step ✅

Use Both (RAG + Fine-Tuned Model):

For complex enterprise deployments: fine-tune for style/behaviour, RAG for live knowledge. Example: A fine-tuned model that writes in your company's template and retrieves the latest policy docs.


Cost Comparison: Real Numbers

RAG (Bedrock Knowledge Bases)
  • OpenSearch Serverless: ~$700/month minimum (consider Aurora pgvector for dev/test: ~$50/month)
  • Titan Embeddings V2: $0.02 per 1,000 tokens (ingestion, one-time)
  • Claude 3 Sonnet inference: ~$0.003 per query
  • Low-volume total: ~$5–50/month
Fine-Tuning (Bedrock Custom Models)
  • Claude Haiku fine-tuning: ~$0.004 per 1,000 tokens of training data
  • 10,000 examples × avg 500 tokens = $20 per training run
  • Provisioned throughput for inference: ~$600–1,200/month (dedicated endpoint required)
  • Viable only at high query volume or strong business case


My Recommendation for Most Teams

Start with RAG. It gets you to production in hours, not weeks. If you hit a ceiling — the model consistently writes in the wrong format, uses wrong terminology, or you need sub-100ms responses at scale — then invest in fine-tuning.
At STC, every AI project we've shipped has started as RAG. Two have moved to a hybrid approach after proven business value.


Resources


I'm Meghana Rajagopal, lead Cloud & AI Delivery at STC (AWS Premier Partner) and AWS ML Engineer Associate. I write about practical AWS AI/ML solutions and architecture. Connect with me on  LinkedIn  
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article