RAG in Production: 7 Things That Break After Your First 1,000 Users.

Retrieval-Augmented Generation (RAG) looks simple on a whiteboard:

User → Retrieve relevant documents → Send them to an LLM → Generate an answer

In production, however, the hard part is rarely calling the LLM.

The real engineering challenges appear when your system has more documents, more users, more queries, and less room for mistakes.

A RAG system that works perfectly with 100 documents can behave very differently with millions of chunks and thousands of concurrent requests.

This article covers 7 common problems that can break a RAG system in production, along with practical ways to design around them using AWS services.

A Simple RAG Architecture on AWS

A production RAG system can be represented as:

Supporting services can include:

· Amazon S3 — document storage
· Amazon Bedrock — foundation models, embeddings, and managed GenAI capabilities
· Amazon OpenSearch Service — vector and hybrid search
· Amazon CloudWatch — logs, metrics, and monitoring
· AWS IAM — authentication and authorization
· Amazon Bedrock Knowledge Bases — managed RAG capabilities for supported architectures

The exact architecture depends on the application's requirements, but the underlying problems remain surprisingly similar.

1. Bad Chunking

The first problem often happens before retrieval even begins.

Suppose you have a 50-page company policy document.

You could split it into arbitrary chunks:
Chunk 1 → first 500 characters
Chunk 2 → next 500 characters
Chunk 3 → next 500 characters

This is easy to implement, but it may destroy the meaning of the original document.

Imagine this sentence:

«Employees can work remotely for up to 20 days per quarter.»

If the sentence is split across two chunks, the retrieval system may receive incomplete information.

Why this matters

The quality of your RAG system is heavily influenced by the quality of the chunks entering the retrieval system.

Good chunking should consider:

· Document structure
· Paragraph boundaries
· Headings
· Tables
· Semantic relationships
· Chunk size
· Overlap
· Metadata

A financial report, legal document, and source-code repository should not necessarily use the same chunking strategy.

Key takeaway

«If your chunks contain too much or too little context, even a powerful LLM cannot reliably retrieve the right answer.»

AWS takeaway

For managed RAG workflows, Amazon Bedrock Knowledge Bases can help handle document ingestion and retrieval workflows while allowing developers to focus on application logic.

2. Poor Retrieval

Now assume your documents are chunked perfectly.

Your system still has another problem:

Can it actually find the right information?

Consider a user asking:

«"What was our cloud infrastructure spending in Q2?"»

Your vector search might return documents containing words such as:

cloud
infrastructure
spending
Q2

But semantic similarity does not guarantee that the returned chunk contains the exact answer.

This is where retrieval strategy becomes important.

Vector Search

Vector search represents text as embeddings and finds semantically similar content.

Hybrid Search

For many applications, combining:

· semantic/vector search
· keyword search
· metadata filtering
· reranking

can produce better results than relying on vector similarity alone.

For example:

Key takeaway

«RAG quality is often limited more by what you retrieve than by which LLM generates the answer.»

AWS takeaway

Amazon OpenSearch Service can support vector and search-based retrieval patterns, while Amazon Bedrock can provide models for embedding and reranking workflows.

3. Embedding Model Mismatch

Embeddings are one of the least visible but most important parts of a RAG system.

An embedding converts text into a numerical representation:

Similar concepts should ideally produce nearby vectors.

But there is no universally perfect embedding model.

A model optimized for general English text may not perform equally well for:

· Medical documents
· Legal documents
· Financial reports
· Code
· Multilingual content
· Technical documentation

For example, a developer documentation search system may require embeddings that understand technical terminology much better than a generic conversational dataset.

Key takeaway

«Your embedding model should match the language, domain, and type of information your application actually searches.»

AWS takeaway

Amazon Bedrock provides access to embedding models and other foundation-model capabilities, allowing teams to choose models appropriate for their application instead of building the entire ML infrastructure themselves.

4. Hallucinations

This is probably the most misunderstood RAG problem.

A common assumption is:

«"If I give the LLM documents, it cannot hallucinate."»

Unfortunately, that is not true.

RAG reduces hallucination risk by giving the model relevant context, but the model can still:

· Misinterpret information
· Combine unrelated facts
· Generate unsupported claims
· Ignore retrieved context
· Answer when evidence is insufficient

Consider:

A production system should explicitly handle the second case.

Possible safeguards

You can introduce:

· Source citations
· Confidence thresholds
· Retrieval-score thresholds
· Prompt constraints
· Grounding checks
· Output validation
· Guardrails
· Human review for high-risk workflows

A good RAG application should be comfortable saying:

«"I don't have enough information to answer that."»

That is often better than confidently generating a wrong answer.

Key takeaway

«Retrieval does not automatically make an LLM truthful; the application still needs grounding and validation.»

AWS takeaway

Amazon Bedrock provides capabilities such as Knowledge Bases and Guardrails that can be incorporated into architectures where grounded and controlled responses are important.

5. Slow Retrieval

A RAG system is a pipeline.

That means latency accumulates.

For example:

Stage Latency
API request 50 ms
Query embedding 100 ms
Vector search 200 ms
Reranking 250 ms
LLM generation 1,500 ms
Total 2,100 ms

Two seconds might be acceptable for some applications.

But imagine adding:

· Multiple retrieval calls
· More reranking
· Larger prompts
· Multiple LLM calls
· External APIs
· Complex authorization

Suddenly, your response can take several seconds.

Where to optimize

Start measuring each stage separately:

Then optimize the actual bottleneck.

Possible techniques include:

· Caching
· Reducing unnecessary retrieval calls
· Limiting retrieved chunks
· Optimizing indexes
· Streaming responses
· Choosing appropriate models
· Parallelizing independent operations

Key takeaway

«A highly accurate RAG pipeline is useless if users have to wait several seconds for every response.»

AWS takeaway

Amazon CloudWatch can help monitor application metrics and latency so you can identify which stage of the RAG pipeline is actually slowing the system down.

6. Rising Costs

A prototype can be cheap.

A production RAG application can become surprisingly expensive.

Your cost can come from several places:

And there is another hidden factor:

Prompt size.

If every query sends 20 large chunks to the LLM, token consumption can grow rapidly.

For example:

5 retrieved chunks × 500 tokens
= 2,500 context tokens

20 retrieved chunks × 500 tokens
= 10,000 context tokens

The second approach may not provide 4× better answers, but it can generate substantially more work and cost.

Cost optimization

Useful strategies include:

· Retrieve fewer but better chunks
· Cache repeated queries
· Cache embeddings
· Use smaller models where appropriate
· Monitor token usage
· Use asynchronous processing for non-interactive workloads
· Set budgets and alerts
· Avoid unnecessary LLM calls

Key takeaway

«At scale, token usage, embeddings, vector search, and repeated retrieval can become more expensive than the LLM itself.»

AWS takeaway

AWS Cost Explorer, AWS Budgets, and service-level monitoring can help teams understand and control the cost of production workloads.

7. No Evaluation

This is the problem that makes all the other problems difficult to detect.

You deploy your RAG application.

Users say:

«"Sometimes the answers aren't great."»

But what does "not great" actually mean?

Is the problem:

· Retrieval?
· Chunking?
· Embeddings?
· Prompting?
· The LLM?
· Latency?
· Missing documents?

Without evaluation, you're guessing.

Measure the pipeline

A useful evaluation system should track multiple dimensions.

Retrieval quality — Did we retrieve the correct information?

Answer quality — Did the generated answer actually answer the question?

Grounding — Is the answer supported by the retrieved context?

Latency — How long did the complete request take?

User feedback — Did the user consider the answer useful?

A simple production dashboard might track:

Metric Value
Retrieval Recall 91%
Grounded Answers 94%
Answer Success Rate 89%
P95 Latency 1.8s
Average Tokens/Request 3,200
User Satisfaction 4.3/5

These numbers turn "the AI feels bad" into an engineering problem that can actually be investigated.

Key takeaway

«If you do not measure retrieval quality and answer quality, you are optimizing the system blindly.»

AWS takeaway

Amazon CloudWatch can provide operational monitoring, while application-level evaluation can be implemented through custom metrics, logs, dashboards, and evaluation pipelines.

Putting Everything Together

A production RAG architecture should not simply look like:

User → Vector DB → LLM

A more realistic system looks like:

The important idea is that RAG is not one model call.

It is a complete distributed application.

A Practical Production Checklist

Before calling your RAG application production-ready, ask:

Data

☐ Are documents correctly parsed?
☐ Is chunking appropriate for the document type?
☐ Is metadata attached to chunks?
☐ Are outdated documents removed or versioned?

Retrieval

☐ Is vector search sufficient?
☐ Would hybrid search improve results?
☐ Are metadata filters being used?
☐ Is reranking necessary?

Generation

☐ Does the LLM receive only useful context?
☐ Can the system refuse when evidence is insufficient?
☐ Are answers grounded in retrieved information?
☐ Are citations available where appropriate?

Performance

☐ Is every pipeline stage measured?
☐ Is P95/P99 latency monitored?
☐ Are expensive operations cached?
☐ Can independent operations run in parallel?

Cost

☐ Are token costs tracked?
☐ Are embedding costs tracked?
☐ Is vector storage monitored?
☐ Are budgets and alerts configured?

Evaluation

☐ Do you have a test dataset?
☐ Are retrieval metrics measured?
☐ Are answer-quality metrics measured?
☐ Are production user signals captured?

The Bigger Lesson

The biggest mistake is treating RAG as:

«"Put documents into a vector database and connect an LLM."»

That is enough for a prototype.

Production requires much more.

You need to think about:

Every stage can become a bottleneck.

And every stage can affect the final user experience.

Conclusion

RAG is powerful because it allows LLM applications to work with information that was not part of their original training data.

But production RAG is fundamentally a systems engineering problem.

The best implementation is not necessarily the one with the biggest model.

It is the one that:

· Retrieves the right information
· Uses the right context
· Produces grounded answers
· Responds quickly
· Controls costs
· Protects data
· Measures its own performance
· Improves continuously

The goal isn't simply to make an LLM answer questions.

The goal is to build a system users can trust.

Better RAG → Better Answers → Real Impact

Key Takeaways

1. Good retrieval starts with good data and chunking.
2. Retrieval quality matters as much as generation quality.
3. Choose embeddings based on your actual domain.
4. RAG reduces hallucinations but does not eliminate them.
5. Measure every stage of the pipeline to control latency.
6. Optimize token usage and retrieval to control costs.
7. If you cannot measure your RAG system, you cannot reliably improve it.

Further Reading

For a production implementation, explore:

· Amazon Bedrock
· Amazon Bedrock Knowledge Bases
· Amazon OpenSearch Service
· Amazon S3
· Amazon CloudWatch
· AWS IAM
· AWS Cost Explorer
· AWS Budgets

The most important lesson is to start simple, measure everything, and add complexity only when the data proves you need it.