AWS Builder Center

Advanced RAG Patterns

Most RAG systems work fine in demos and fall apart in production. This talk walks through every stage where basic RAG breaks from poorly-formed queries to flat vector indices to hallucinated answers and shows how to fix each one. We cover query transformation, semantic routing, RAPTOR indexing, ColBERT late interaction, Chain-of-Note post-processing, and agentic correction loops like CRAG and Self-RAG. Leave with a decision framework you can apply to your system the same week.

Advanced RAG patterns

Most RAG systems work fine in demos and fall apart in production. This happens for a predictable reason: basic RAG has five distinct points of failure, and most teams only discover them after shipping. This post walks through every stage where things break and shows how to fix each one systematically — from query transformation and intelligent routing, to ColBERT-style retrieval, Chain-of-Note post-processing, and agentic correction loops. By the end you'll have a complete mental model and working code patterns for building RAG that holds up when real users ask real questions.

Contents

  1. Why Basic RAG Fails at Scale
  2. Query Transformation
  3. Query Routing
  4. Advanced Indexing (Multi-Rep + RAPTOR)
  5. Advanced Retrieval (DPR + ColBERT)
  6. Re-ranking & Chain-of-Note
  7. Agentic RAG: CRAG & Self-RAG
  8. Evaluation Framework
  9. Production Architecture Blueprint
  10. Implementation Patterns

1 · Why Basic RAG Fails at Scale

The Basic Pipeline Has 5 Failure Points

1
User Query → Query Embedding → Similarity Search → Retrieved Chunks → LLM Answer
StageWhat goes wrongWhy it matters
User QueryAmbiguous, poorly wordedGarbage in, garbage out
Query EmbeddingGeneric model misrepresents intentWrong retrieval space entirely
Similarity SearchSemantically close ≠ answer-bearingTop-K chunks miss the point
Retrieved ChunksIrrelevant or missing contextLLM gets confused or hallucinates
LLM AnswerIgnores context, uses prior knowledgeWrong answer delivered confidently
The core issue is that people build basic RAG and it works fine on demos. Then it hits production, user queries get weird, and suddenly the retrieval is returning chunks that are vaguely related but useless for actually answering the question. The system looks smart but isn't.
Advanced RAG means intercepting at every one of those stages. Each arrow in the pipeline is an opportunity to add intelligence.

2 · Query Transformation

The Problem

Users don't write good queries. They write what they're thinking, not what the vector database needs to hear. A single poorly-phrased query will retrieve the wrong documents even if the answer is sitting right there in your corpus.

Technique 1: Multi-Query Generation

Generate N paraphrased versions of the original query, retrieve separately for each, merge all results before ranking.
1
2
3
4
5
6
7
from langchain.retrievers.multi_query import MultiQueryRetriever

retriever = MultiQueryRetriever.from_llm(
retriever=vectorstore.as_retriever(),
llm=llm
)
docs = retriever.get_relevant_documents("What is the revenue of Company X in 2005?")
You typically see a 10–15% recall improvement on ambiguous queries. The cost is extra LLM calls, which is worth it for any query where precision matters.

Technique 2: RAG-Fusion with Reciprocal Rank Fusion (RRF)

Generate multiple related queries, retrieve for each, then merge the ranked lists using RRF. The key insight is that a document appearing consistently across multiple ranked lists is probably genuinely relevant.
1
2
3
4
5
6
7
def reciprocal_rank_fusion(results: list[list], k: int = 60) -> list:
scores: dict = {}
for ranked_list in results:
for rank, doc in enumerate(ranked_list):
doc_id = doc.page_content
scores[doc_id] = scores.get(doc_id, 0) + 1 / (k + rank + 1)
return sorted(scores.items(), key=lambda x: x[1], reverse=True)
score(d) = Σ 1 / (k + rank_i(d))
Expect 15–20% improvement on queries that have multiple valid angles. The k=60 default is from the original paper and works well in practice.

Technique 3: Decomposition

Break a complex question into sequential sub-questions. Answer each one, then feed those answers as context for the next.
1
2
3
4
# Use case: "Is Berlin warmer today than in 2005?"
# Sub-Q1: What is Berlin's current average temperature?
# Sub-Q2: What was Berlin's average temperature in 2005?
# Synthesis: Compare the two answers
This is probably the most underused technique. Analytical questions are almost never stated verbatim in your documents. If you do a direct vector search for "Is Berlin warmer today than in 2005?" you'll get chunks about Berlin's climate in general, not the specific comparison you need. Decomposition forces the system to retrieve the atomic facts and reason over them.

Technique 4: HyDE (Hypothetical Document Embeddings)

Generate a hypothetical answer to the query, embed that hypothetical answer, then search for real documents similar to it. You're essentially asking: what would a document that answers this question look like?
1
2
3
4
5
6
7
from langchain.chains import HypotheticalDocumentEmbedder

hyde_embeddings = HypotheticalDocumentEmbedder.from_llm(
llm=llm,
base_embeddings=embeddings,
custom_prompt=hyde_prompt
)
It's powerful for domain-specific corpora where query language differs from document language. The tradeoff is that hallucinated hypotheses can steer retrieval in the wrong direction. Ground your prompts tightly.

Technique 5: Step-Back Prompting

Before searching, abstract the query to a higher-level principle. Retrieve for that abstraction, then answer the specific question with that broader context.
1
2
3
Original: "What happens to a gas when pressure increases at constant temperature?"
Step-Back: "What is Boyle's Law?"
Retrieve Boyle's Law docs → use them to answer the original question
This works particularly well in scientific, legal, and technical domains where specific questions are really asking about general principles. The model stops chasing surface-level keyword matches and starts understanding what the question is really about.

3 · Query Routing

The Problem

Real systems don't have one data source. They have a SQL database, a vector store, maybe a graph database, APIs, and a private knowledge base. Sending every query to every source is expensive and noisy. Sending it to the wrong source gives you nothing useful.

Logical Routing via Function Calling

Give the LLM a structured schema and let it decide which store to query. This works well when data sources have clear, non-overlapping domains.
1
2
3
4
5
6
7
8
9
10
from pydantic import BaseModel

class RouteQuery(BaseModel):
datasource: Literal["python_docs", "js_docs", "golang_docs"]

structured_llm = llm.with_structured_output(RouteQuery)

def route(question: str) -> str:
result = structured_llm.invoke(question)
return result.datasource

Semantic Routing

Embed descriptions of each data source and route by embedding similarity. No LLM call needed, just a cosine similarity check.
1
2
3
4
5
6
7
8
9
10
11
from langchain.utils.math import cosine_similarity

route_layer = {
"sql_db": embed("Structured data, financial records, transactions"),
"vector_store": embed("Semantic search over unstructured text"),
}

def semantic_route(query: str) -> str:
q_embed = embed(query)
scores = {k: cosine_similarity(q_embed, v) for k, v in route_layer.items()}
return max(scores, key=scores.get)

Tool and API Routing

Data sources don't have to be static. You can route to live APIs where the LLM generates the parameters, the tool executes, and the output feeds back into the RAG context.
1
2
3
# LLM decides: "This needs real-time data"
# → calls weather_api(city="Berlin", date="today")
# → response becomes part of retrieved context
This is where RAG starts blending into agentic systems, which we'll cover in section 7.

4 · Advanced Indexing

The Problem

Standard chunking is brutal. You split a 50-page document into 512-token chunks, embed them, and hope the right chunk surfaces at retrieval time. Flat vector indices don't understand document hierarchy at all. The chunk that retrieves well often isn't the chunk that answers the question completely.

Multi-Representation Indexing

The insight here is simple: index small, return large. Use LLM-generated summaries or propositions for the vector index (they embed cleanly) but store the full source documents separately. Retrieval hits the summary, generation gets the full document.
1
2
3
4
5
6
7
8
9
10
11
12
from langchain.storage import InMemoryByteStore
from langchain.retrievers.multi_vector import MultiVectorRetriever

summaries = [llm.invoke(f"Summarize: {doc}") for doc in docs]

store = InMemoryByteStore()
retriever = MultiVectorRetriever(
vectorstore=vectorstore, # searches summaries
byte_store=store, # returns full docs
id_key="doc_id",
)
retriever.docstore.mset(list(zip(doc_ids, docs)))
This solves a real problem: dense embeddings of long documents lose specificity because they're averaging over too much content.

RAPTOR

Build a tree. At the leaves are your raw chunks. At each level above, cluster the nodes and have an LLM summarize each cluster. Keep going until you have a single root node that captures the entire document's essence.
1
2
3
4
Level 0: Raw chunks (leaves)
Level 1: LLM summaries of chunk clusters
Level 2: Summaries of level-1 summaries
Root: Global document summary
At query time, search all levels simultaneously. Simple queries hit the leaves. High-level conceptual queries hit the root. Multi-hop questions find intermediate nodes.
Cluster with Gaussian Mixture Models, summarize with an LLM. The paper reports 20%+ gains on multi-hop questions. Implementation: arxiv 2401.18059 

Query Construction for Structured Sources

If your vector store has metadata, use it. Convert natural language conditions into structured filters before searching.
1
2
3
4
5
6
# "What Python tutorials were published after 2023?"
metadata_filter = {"language": "python", "publish_date": {"$gt": "2023-01-01"}}

retriever = vectorstore.as_retriever(
search_kwargs={"filter": metadata_filter}
)
This is a massive win on corpora with rich metadata. Don't let the LLM search broadly and then filter — construct the filter first, then search a smaller, cleaner subset.

5 · Advanced Retrieval: DPR & ColBERT

The Problem

General-purpose embedding models were trained to capture semantic similarity, not answer-bearing relevance. These are different things. A passage about "temperature measurement instruments" is semantically similar to a query about "thermometer history" but might be completely useless for answering it. The model collapses everything into one vector and loses the nuance.

Dense Passage Retrieval (DPR)

Train separate encoders for queries and documents on actual Q&A pairs. The key difference from standard embedding models is the training objective: contrastive loss on real question-answer pairs means the model learns what "answers this question" looks like, not just what's semantically similar.
1
2
3
query encoder: Q → q_vec
document encoder: D → d_vec
score = dot(q_vec, d_vec)
In-batch negatives during training force the model to distinguish answer-bearing passages from merely related ones.

ColBERT: Late Interaction

Standard retrieval compresses an entire document into one vector. ColBERT keeps one vector per token and does comparison at the token level.
1
2
3
4
Query tokens: [q1, q2, q3, q4] → embeddings [x1, x2, x3, x4]
Document tokens: [d1..d8] → embeddings [y1..y8]

MaxSim score = Σᵢ max_j cosine(xᵢ, yⱼ)
For every query token, find the most similar document token and take that score. Sum across all query tokens. This means a query like "Gross Revenue of all products of Company X in 2005" gets scored on each key term independently — gross revenue, all products, company X, year 2005 — rather than as a blended average.
1
2
3
4
5
from ragatouille import RAGPretrainedModel

RAG = RAGPretrainedModel.from_pretrained("colbert-ir/colbertv2.0")
RAG.index(collection=docs, index_name="my_index")
results = RAG.search(query="Gross revenue Company X 2005", k=10)
ColBERT achieves over 2x recall improvement over BM25 on NaturalQuestions. The tradeoff is storage: you're storing token vectors for every document rather than one vector. For most production use cases that's fine.

6 · Re-ranking & Chain-of-Note Post-processing

Re-ranking with Multiple Retrievers

Don't pick one retrieval method. Run BM25, dense retrieval, and ColBERT in parallel, then merge with RRF.
1
2
3
4
5
6
from langchain.retrievers import EnsembleRetriever

ensemble = EnsembleRetriever(
retrievers=[bm25_retriever, dense_retriever, colbert_retriever],
weights=[0.3, 0.5, 0.2]
)
After the ensemble merge, consider a cross-encoder re-ranker. Cross-encoders process the query and document together in a single pass, which is much more accurate than bi-encoder similarity but too slow to run across your entire corpus. Run it on the top-K candidates from your ensemble.
cross-encoder/ms-marco-MiniLM-L-6-v2 is a solid free option. Cohere Rerank is the easy production choice.

Chain-of-Note (CoN)

This is one of the most practical techniques in the whole list. Instead of dumping all retrieved chunks directly into the context window, make the LLM read each chunk and write notes about it first.
1
2
3
4
5
6
7
8
9
10
11
12
CON_PROMPT = """
Given the question: {question}

Read the following passage and write concise notes:
1. Is this passage relevant to the question?
2. What specific facts from this passage help answer the question?
3. What is missing?

Passage: {passage}

Notes:
"""
Three things happen with this approach:
  • Fully relevant passage: the notes extract the key facts cleanly
  • Partially relevant passage: the notes acknowledge the gaps, and the LLM can supplement from its own knowledge without hallucinating
  • Irrelevant passage: the LLM flags it and you can discard it or trigger a web search
The hallucination reduction is real. When you force the model to explicitly evaluate each chunk before synthesizing, it's much harder for it to confabulate details that aren't there. The paper also shows it handles conflicting information across documents much better than vanilla RAG.

7 · Agentic RAG: CRAG & Self-RAG

CRAG: Corrective RAG

The idea is to add a retrieval evaluator, an LLM grader that scores each retrieved document and decides whether it actually warrants being used. If the documents are bad, CRAG either refines the query or falls back to web search.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
from langgraph.graph import StateGraph

def grade_documents(state):
docs = state["documents"]
question = state["question"]
scored = [grader_llm.invoke({"doc": d, "question": question}) for d in docs]
return {
"documents": [d for d, s in zip(docs, scored) if s.binary_score == "yes"],
"web_search": any(s.binary_score == "no" for s in scored)
}

def web_search(state):
if state["web_search"]:
results = tavily_search.invoke(state["question"])
state["documents"] += results
return state
Decision logic:
  • CORRECT: retrieved docs are good, proceed to generation
  • AMBIGUOUS: refine the query and supplement with web search
  • INCORRECT: discard everything and do a full web search
This is a big deal for production systems where document staleness is a concern. Your internal corpus might be six months old. CRAG lets you automatically fall back to fresh web content when local retrieval fails.

Self-RAG

Self-RAG trains the LLM to insert special reflection tokens during generation. These tokens make the model explicitly decide whether to retrieve more content and score its own outputs.
Reflection TokenMeaning
[Retrieve]Should I retrieve more passages?
[ISREL]Is this passage relevant to the query?
[ISSUP]Is my claim supported by the passage?
[ISUSE]Is this output useful overall?
1
2
3
4
# Self-RAG generates multiple candidate segments
# Each scored by [ISREL][ISSUP][ISUSE] tokens
# Best-scored segment selected as output
# Loops back to retrieve more if ISUSE = "no"
When to choose Self-RAG vs CRAG: if you need per-sentence grounding and are doing long-form generation, Self-RAG gives finer control. For most production retrieval pipelines where you want a corrective mechanism without training a custom model, CRAG is simpler and more practical.

8 · RAG Evaluation Framework

The RAG Triad (TruLens)

Most teams measure end-to-end accuracy and miss where the system is actually failing. The RAG Triad isolates three independent failure modes:
MetricWhat it measuresWhat failure it catches
Context RelevanceIs retrieved context relevant to the query?Retrieval noise
GroundednessIs the answer grounded in the context?Hallucination
Answer RelevanceDoes the answer address the query?Off-topic generation
You can have perfect retrieval and still hallucinate. You can be grounded and still give an off-topic answer. Measuring them separately tells you exactly where to focus improvement effort.

RAGAS: Automated Pipeline Evaluation

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
from ragas import evaluate
from ragas.metrics import (
faithfulness,
answer_relevancy,
context_precision,
context_recall,
)
from datasets import Dataset

result = evaluate(
Dataset.from_dict({
"question": questions,
"answer": answers,
"contexts": contexts,
"ground_truth": ground_truths,
}),
metrics=[faithfulness, answer_relevancy, context_precision, context_recall],
)
print(result)
Run this on a representative test set before and after each change to the pipeline. The RAGAS scores will tell you whether a retrieval optimization actually improved generation quality or just looked better in isolation.

What to Track in Production

1
2
3
Retrieval: Hit Rate@K · MRR · NDCG · Context Precision / Recall
Generation: Faithfulness · Answer Relevancy · Hallucination Rate
System: Latency P50/P95 · Cost-per-query · Throughput

LLM-as-Judge for Qualitative Scoring

When you don't have ground truth labels, use a stronger LLM to score outputs:
1
2
3
4
5
6
7
8
9
10
11
JUDGE_PROMPT = """
Rate the following answer on a scale of 1-5 for:
- Faithfulness to context (1=hallucinated, 5=fully grounded)
- Relevance to question (1=off-topic, 5=directly answers)

Question: {question}
Context: {context}
Answer: {answer}

Provide scores as JSON: {"faithfulness": N, "relevance": N, "reasoning": "..."}
"""
This isn't perfect but it's surprisingly good for catching obvious failures and gives you a cheap way to score large test sets without human annotation.

9 · Production Architecture Blueprint

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
┌─────────────────────────────────────────────────────────────────────┐
│ ADVANCED RAG SYSTEM │
│ │
│ INGESTION RETRIEVAL GENERATION │
│ Raw Docs Multi-Query Re-rank (RRF) │
│ ↓ Routing Chain-of-Note │
│ Chunking ┌─────────┐ LLM Answer │
│ ↓ │ Dense │ │
│ Multi-Rep Index │ BM25 │ → Merge │
│ (Summary + Full) │ ColBERT │ Fuse │
│ ↓ └─────────┘ Re-rank │
│ RAPTOR Tree │
│ ↓ AGENTIC LAYER │
│ Vector DB CRAG evaluator │
│ + DocStore Self-RAG loop │
│ Web fallback │
│ │
│ EVALUATION: RAGAS · TruLens · LLM-as-Judge · Latency tracking │
└─────────────────────────────────────────────────────────────────────┘

Technology Stack

LayerOptionsWhat I'd recommend
Vector DBChroma, FAISS, Pinecone, WeaviatePinecone or Weaviate for production
EmbeddingOpenAI, Mistral, Cohere, BGEBGE-M3 or Mistral Embed for cost
RetrievalDense, BM25, ColBERT via RAGatouilleEnsemble for best recall
Re-rankerCross-encoder, Cohere RerankCohere Rerank for simplicity
OrchestrationLangChain, LlamaIndex, LangGraphLangGraph for anything agentic
EvaluationRAGAS, TruLens, DeepEvalRAGAS for automation

10 · Implementation Patterns

Pattern 1: Full Advanced RAG Chain (LangChain)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
from langchain.retrievers.multi_query import MultiQueryRetriever
from langchain.retrievers import EnsembleRetriever, BM25Retriever
from langchain_core.runnables import RunnablePassthrough
from langchain_core.output_parsers import StrOutputParser

multi_query = MultiQueryRetriever.from_llm(retriever=dense.as_retriever(), llm=llm)
ensemble = EnsembleRetriever(retrievers=[bm25, multi_query], weights=[0.4, 0.6])

con_prompt = ChatPromptTemplate.from_template("""
Write reading notes for the context, then answer the question.

Context: {context}
Question: {question}

Notes:
Answer:
"""
)

chain = (
{"context": ensemble | format_docs, "question": RunnablePassthrough()}
| con_prompt
| llm
| StrOutputParser()
)

Pattern 2: CRAG with LangGraph

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
from langgraph.graph import END, StateGraph

workflow = StateGraph(GraphState)
workflow.add_node("retrieve", retrieve)
workflow.add_node("grade_documents", grade_documents)
workflow.add_node("generate", generate)
workflow.add_node("web_search_node", web_search)

workflow.set_entry_point("retrieve")
workflow.add_edge("retrieve", "grade_documents")
workflow.add_conditional_edges(
"grade_documents",
decide_to_generate,
{"web_search": "web_search_node", "generate": "generate"},
)
workflow.add_edge("web_search_node", "generate")
workflow.add_edge("generate", END)

app = workflow.compile()

Pattern 3: ColBERT with RAGatouille

1
2
3
4
5
6
7
8
9
10
11
12
13
from ragatouille import RAGPretrainedModel

RAG = RAGPretrainedModel.from_pretrained("colbert-ir/colbertv2.0")

RAG.index(
collection=[doc.page_content for doc in docs],
document_ids=[str(i) for i in range(len(docs))],
index_name="production_index",
max_document_length=256,
split_documents=True,
)

results = RAG.search(query="What is the gross revenue of Company X in 2005?", k=10)

Decision Tree: Where to Start

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
Is the query simple and well-formed?
├── YES → Standard dense retrieval may suffice
└── NO → Apply Query Transformation first
├── Ambiguous → Multi-Query + RAG-Fusion
├── Complex multi-step → Decomposition
└── Domain mismatch → HyDE / Step-Back

Do you have multiple data sources?
└── YES → Add Query Routing (Logical + Semantic)

Is recall still poor?
└── YES → Upgrade retrieval: DPR → ColBERT → Ensemble + RRF

Are hallucinations still occurring?
└── YES → Add Chain-of-Note post-processing

Is retrieved context unreliable?
└── YES → Add CRAG evaluator + web fallback

Need per-sentence grounding?
└── YES → Self-RAG with reflection tokens

Are you measuring quality?
└── Use RAGAS + TruLens RAG Triad continuously

Conclusion

Advanced RAG isn't one technique it's a layered set of interventions, each targeting a specific failure mode in the pipeline. The good news is you don't have to implement everything at once. Start with query transformation since it gives the biggest bang for the least engineering effort, then layer in better retrieval (ensemble + ColBERT), then add Chain-of-Note to cut hallucinations, and only add CRAG or Self-RAG once you're confident the retrieval itself is solid. Evaluate at every step with RAGAS so you actually know what's improving. The teams that build reliable RAG in production are the ones who treat each stage as an independent engineering problem rather than hoping the LLM will compensate for poor retrieval. It won't.

References

ResourceLink
RAPTOR Paperhttps://arxiv.org/pdf/2401.18059.pdf
CRAG Paperhttps://arxiv.org/abs/2401.15884
ColBERT Paperhttps://arxiv.org/pdf/2004.12832.pdf
RAGAShttps://github.com/explodinggradients/ragas
RAGatouille (ColBERT)https://github.com/bclavie/RAGatouille
LangGraph CRAGhttps://github.com/langchain-ai/langgraph/blob/main/examples/rag/langgraph_crag.ipynb
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article