Advanced RAG Patterns
Most RAG systems work fine in demos and fall apart in production. This talk walks through every stage where basic RAG breaks from poorly-formed queries to flat vector indices to hallucinated answers and shows how to fix each one. We cover query transformation, semantic routing, RAPTOR indexing, ColBERT late interaction, Chain-of-Note post-processing, and agentic correction loops like CRAG and Self-RAG. Leave with a decision framework you can apply to your system the same week.
Advanced RAG patterns
Most RAG systems work fine in demos and fall apart in production. This happens for a predictable reason: basic RAG has five distinct points of failure, and most teams only discover them after shipping. This post walks through every stage where things break and shows how to fix each one systematically — from query transformation and intelligent routing, to ColBERT-style retrieval, Chain-of-Note post-processing, and agentic correction loops. By the end you'll have a complete mental model and working code patterns for building RAG that holds up when real users ask real questions.
Contents
- Why Basic RAG Fails at Scale
- Query Transformation
- Query Routing
- Advanced Indexing (Multi-Rep + RAPTOR)
- Advanced Retrieval (DPR + ColBERT)
- Re-ranking & Chain-of-Note
- Agentic RAG: CRAG & Self-RAG
- Evaluation Framework
- Production Architecture Blueprint
- Implementation Patterns
1 · Why Basic RAG Fails at Scale

The Basic Pipeline Has 5 Failure Points
1
User Query → Query Embedding → Similarity Search → Retrieved Chunks → LLM Answer| Stage | What goes wrong | Why it matters |
|---|---|---|
| User Query | Ambiguous, poorly worded | Garbage in, garbage out |
| Query Embedding | Generic model misrepresents intent | Wrong retrieval space entirely |
| Similarity Search | Semantically close ≠ answer-bearing | Top-K chunks miss the point |
| Retrieved Chunks | Irrelevant or missing context | LLM gets confused or hallucinates |
| LLM Answer | Ignores context, uses prior knowledge | Wrong answer delivered confidently |
The core issue is that people build basic RAG and it works fine on demos. Then it hits production, user queries get weird, and suddenly the retrieval is returning chunks that are vaguely related but useless for actually answering the question. The system looks smart but isn't.
Advanced RAG means intercepting at every one of those stages. Each arrow in the pipeline is an opportunity to add intelligence.
2 · Query Transformation

The Problem
Users don't write good queries. They write what they're thinking, not what the vector database needs to hear. A single poorly-phrased query will retrieve the wrong documents even if the answer is sitting right there in your corpus.
Technique 1: Multi-Query Generation
Generate N paraphrased versions of the original query, retrieve separately for each, merge all results before ranking.
1
2
3
4
5
6
7
from langchain.retrievers.multi_query import MultiQueryRetriever
retriever = MultiQueryRetriever.from_llm(
retriever=vectorstore.as_retriever(),
llm=llm
)
docs = retriever.get_relevant_documents("What is the revenue of Company X in 2005?")You typically see a 10–15% recall improvement on ambiguous queries. The cost is extra LLM calls, which is worth it for any query where precision matters.
Technique 2: RAG-Fusion with Reciprocal Rank Fusion (RRF)
Generate multiple related queries, retrieve for each, then merge the ranked lists using RRF. The key insight is that a document appearing consistently across multiple ranked lists is probably genuinely relevant.
1
2
3
4
5
6
7
def reciprocal_rank_fusion(results: list[list], k: int = 60) -> list:
scores: dict = {}
for ranked_list in results:
for rank, doc in enumerate(ranked_list):
doc_id = doc.page_content
scores[doc_id] = scores.get(doc_id, 0) + 1 / (k + rank + 1)
return sorted(scores.items(), key=lambda x: x[1], reverse=True)score(d) = Σ 1 / (k + rank_i(d))Expect 15–20% improvement on queries that have multiple valid angles. The k=60 default is from the original paper and works well in practice.
Technique 3: Decomposition
Break a complex question into sequential sub-questions. Answer each one, then feed those answers as context for the next.
1
2
3
4
# Use case: "Is Berlin warmer today than in 2005?"
# Sub-Q1: What is Berlin's current average temperature?
# Sub-Q2: What was Berlin's average temperature in 2005?
# Synthesis: Compare the two answersThis is probably the most underused technique. Analytical questions are almost never stated verbatim in your documents. If you do a direct vector search for "Is Berlin warmer today than in 2005?" you'll get chunks about Berlin's climate in general, not the specific comparison you need. Decomposition forces the system to retrieve the atomic facts and reason over them.
Technique 4: HyDE (Hypothetical Document Embeddings)
Generate a hypothetical answer to the query, embed that hypothetical answer, then search for real documents similar to it. You're essentially asking: what would a document that answers this question look like?
1
2
3
4
5
6
7
from langchain.chains import HypotheticalDocumentEmbedder
hyde_embeddings = HypotheticalDocumentEmbedder.from_llm(
llm=llm,
base_embeddings=embeddings,
custom_prompt=hyde_prompt
)It's powerful for domain-specific corpora where query language differs from document language. The tradeoff is that hallucinated hypotheses can steer retrieval in the wrong direction. Ground your prompts tightly.
Technique 5: Step-Back Prompting
Before searching, abstract the query to a higher-level principle. Retrieve for that abstraction, then answer the specific question with that broader context.
1
2
3
Original: "What happens to a gas when pressure increases at constant temperature?"
Step-Back: "What is Boyle's Law?"
Retrieve Boyle's Law docs → use them to answer the original questionThis works particularly well in scientific, legal, and technical domains where specific questions are really asking about general principles. The model stops chasing surface-level keyword matches and starts understanding what the question is really about.
3 · Query Routing
The Problem
Real systems don't have one data source. They have a SQL database, a vector store, maybe a graph database, APIs, and a private knowledge base. Sending every query to every source is expensive and noisy. Sending it to the wrong source gives you nothing useful.

Logical Routing via Function Calling
Give the LLM a structured schema and let it decide which store to query. This works well when data sources have clear, non-overlapping domains.
1
2
3
4
5
6
7
8
9
10
from pydantic import BaseModel
class RouteQuery(BaseModel):
datasource: Literal["python_docs", "js_docs", "golang_docs"]
structured_llm = llm.with_structured_output(RouteQuery)
def route(question: str) -> str:
result = structured_llm.invoke(question)
return result.datasourceSemantic Routing
Embed descriptions of each data source and route by embedding similarity. No LLM call needed, just a cosine similarity check.
1
2
3
4
5
6
7
8
9
10
11
from langchain.utils.math import cosine_similarity
route_layer = {
"sql_db": embed("Structured data, financial records, transactions"),
"vector_store": embed("Semantic search over unstructured text"),
}
def semantic_route(query: str) -> str:
q_embed = embed(query)
scores = {k: cosine_similarity(q_embed, v) for k, v in route_layer.items()}
return max(scores, key=scores.get)Tool and API Routing
Data sources don't have to be static. You can route to live APIs where the LLM generates the parameters, the tool executes, and the output feeds back into the RAG context.
1
2
3
# LLM decides: "This needs real-time data"
# → calls weather_api(city="Berlin", date="today")
# → response becomes part of retrieved contextThis is where RAG starts blending into agentic systems, which we'll cover in section 7.
4 · Advanced Indexing
The Problem
Standard chunking is brutal. You split a 50-page document into 512-token chunks, embed them, and hope the right chunk surfaces at retrieval time. Flat vector indices don't understand document hierarchy at all. The chunk that retrieves well often isn't the chunk that answers the question completely.

Multi-Representation Indexing
The insight here is simple: index small, return large. Use LLM-generated summaries or propositions for the vector index (they embed cleanly) but store the full source documents separately. Retrieval hits the summary, generation gets the full document.
1
2
3
4
5
6
7
8
9
10
11
12
from langchain.storage import InMemoryByteStore
from langchain.retrievers.multi_vector import MultiVectorRetriever
summaries = [llm.invoke(f"Summarize: {doc}") for doc in docs]
store = InMemoryByteStore()
retriever = MultiVectorRetriever(
vectorstore=vectorstore, # searches summaries
byte_store=store, # returns full docs
id_key="doc_id",
)
retriever.docstore.mset(list(zip(doc_ids, docs)))This solves a real problem: dense embeddings of long documents lose specificity because they're averaging over too much content.
RAPTOR
Build a tree. At the leaves are your raw chunks. At each level above, cluster the nodes and have an LLM summarize each cluster. Keep going until you have a single root node that captures the entire document's essence.
1
2
3
4
Level 0: Raw chunks (leaves)
Level 1: LLM summaries of chunk clusters
Level 2: Summaries of level-1 summaries
Root: Global document summaryAt query time, search all levels simultaneously. Simple queries hit the leaves. High-level conceptual queries hit the root. Multi-hop questions find intermediate nodes.
Cluster with Gaussian Mixture Models, summarize with an LLM. The paper reports 20%+ gains on multi-hop questions. Implementation: arxiv 2401.18059
Query Construction for Structured Sources
If your vector store has metadata, use it. Convert natural language conditions into structured filters before searching.
1
2
3
4
5
6
# "What Python tutorials were published after 2023?"
metadata_filter = {"language": "python", "publish_date": {"$gt": "2023-01-01"}}
retriever = vectorstore.as_retriever(
search_kwargs={"filter": metadata_filter}
)This is a massive win on corpora with rich metadata. Don't let the LLM search broadly and then filter — construct the filter first, then search a smaller, cleaner subset.
5 · Advanced Retrieval: DPR & ColBERT
The Problem
General-purpose embedding models were trained to capture semantic similarity, not answer-bearing relevance. These are different things. A passage about "temperature measurement instruments" is semantically similar to a query about "thermometer history" but might be completely useless for answering it. The model collapses everything into one vector and loses the nuance.

Dense Passage Retrieval (DPR)
Train separate encoders for queries and documents on actual Q&A pairs. The key difference from standard embedding models is the training objective: contrastive loss on real question-answer pairs means the model learns what "answers this question" looks like, not just what's semantically similar.
1
2
3
query encoder: Q → q_vec
document encoder: D → d_vec
score = dot(q_vec, d_vec)In-batch negatives during training force the model to distinguish answer-bearing passages from merely related ones.
ColBERT: Late Interaction
Standard retrieval compresses an entire document into one vector. ColBERT keeps one vector per token and does comparison at the token level.
1
2
3
4
Query tokens: [q1, q2, q3, q4] → embeddings [x1, x2, x3, x4]
Document tokens: [d1..d8] → embeddings [y1..y8]
MaxSim score = Σᵢ max_j cosine(xᵢ, yⱼ)For every query token, find the most similar document token and take that score. Sum across all query tokens. This means a query like "Gross Revenue of all products of Company X in 2005" gets scored on each key term independently — gross revenue, all products, company X, year 2005 — rather than as a blended average.
1
2
3
4
5
from ragatouille import RAGPretrainedModel
RAG = RAGPretrainedModel.from_pretrained("colbert-ir/colbertv2.0")
RAG.index(collection=docs, index_name="my_index")
results = RAG.search(query="Gross revenue Company X 2005", k=10)ColBERT achieves over 2x recall improvement over BM25 on NaturalQuestions. The tradeoff is storage: you're storing token vectors for every document rather than one vector. For most production use cases that's fine.
6 · Re-ranking & Chain-of-Note Post-processing

Re-ranking with Multiple Retrievers
Don't pick one retrieval method. Run BM25, dense retrieval, and ColBERT in parallel, then merge with RRF.
1
2
3
4
5
6
from langchain.retrievers import EnsembleRetriever
ensemble = EnsembleRetriever(
retrievers=[bm25_retriever, dense_retriever, colbert_retriever],
weights=[0.3, 0.5, 0.2]
)After the ensemble merge, consider a cross-encoder re-ranker. Cross-encoders process the query and document together in a single pass, which is much more accurate than bi-encoder similarity but too slow to run across your entire corpus. Run it on the top-K candidates from your ensemble.
cross-encoder/ms-marco-MiniLM-L-6-v2 is a solid free option. Cohere Rerank is the easy production choice.Chain-of-Note (CoN)
This is one of the most practical techniques in the whole list. Instead of dumping all retrieved chunks directly into the context window, make the LLM read each chunk and write notes about it first.
1
2
3
4
5
6
7
8
9
10
11
12
CON_PROMPT = """
Given the question: {question}
Read the following passage and write concise notes:
1. Is this passage relevant to the question?
2. What specific facts from this passage help answer the question?
3. What is missing?
Passage: {passage}
Notes:
"""Three things happen with this approach:
- Fully relevant passage: the notes extract the key facts cleanly
- Partially relevant passage: the notes acknowledge the gaps, and the LLM can supplement from its own knowledge without hallucinating
- Irrelevant passage: the LLM flags it and you can discard it or trigger a web search
The hallucination reduction is real. When you force the model to explicitly evaluate each chunk before synthesizing, it's much harder for it to confabulate details that aren't there. The paper also shows it handles conflicting information across documents much better than vanilla RAG.
7 · Agentic RAG: CRAG & Self-RAG

CRAG: Corrective RAG
The idea is to add a retrieval evaluator, an LLM grader that scores each retrieved document and decides whether it actually warrants being used. If the documents are bad, CRAG either refines the query or falls back to web search.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
from langgraph.graph import StateGraph
def grade_documents(state):
docs = state["documents"]
question = state["question"]
scored = [grader_llm.invoke({"doc": d, "question": question}) for d in docs]
return {
"documents": [d for d, s in zip(docs, scored) if s.binary_score == "yes"],
"web_search": any(s.binary_score == "no" for s in scored)
}
def web_search(state):
if state["web_search"]:
results = tavily_search.invoke(state["question"])
state["documents"] += results
return stateDecision logic:
- CORRECT: retrieved docs are good, proceed to generation
- AMBIGUOUS: refine the query and supplement with web search
- INCORRECT: discard everything and do a full web search
This is a big deal for production systems where document staleness is a concern. Your internal corpus might be six months old. CRAG lets you automatically fall back to fresh web content when local retrieval fails.
Paper: arxiv 2401.15884
Self-RAG
Self-RAG trains the LLM to insert special reflection tokens during generation. These tokens make the model explicitly decide whether to retrieve more content and score its own outputs.
| Reflection Token | Meaning |
|---|---|
[Retrieve] | Should I retrieve more passages? |
[ISREL] | Is this passage relevant to the query? |
[ISSUP] | Is my claim supported by the passage? |
[ISUSE] | Is this output useful overall? |
1
2
3
4
# Self-RAG generates multiple candidate segments
# Each scored by [ISREL][ISSUP][ISUSE] tokens
# Best-scored segment selected as output
# Loops back to retrieve more if ISUSE = "no"When to choose Self-RAG vs CRAG: if you need per-sentence grounding and are doing long-form generation, Self-RAG gives finer control. For most production retrieval pipelines where you want a corrective mechanism without training a custom model, CRAG is simpler and more practical.
8 · RAG Evaluation Framework

The RAG Triad (TruLens)
Most teams measure end-to-end accuracy and miss where the system is actually failing. The RAG Triad isolates three independent failure modes:
| Metric | What it measures | What failure it catches |
|---|---|---|
| Context Relevance | Is retrieved context relevant to the query? | Retrieval noise |
| Groundedness | Is the answer grounded in the context? | Hallucination |
| Answer Relevance | Does the answer address the query? | Off-topic generation |
You can have perfect retrieval and still hallucinate. You can be grounded and still give an off-topic answer. Measuring them separately tells you exactly where to focus improvement effort.
RAGAS: Automated Pipeline Evaluation
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
from ragas import evaluate
from ragas.metrics import (
faithfulness,
answer_relevancy,
context_precision,
context_recall,
)
from datasets import Dataset
result = evaluate(
Dataset.from_dict({
"question": questions,
"answer": answers,
"contexts": contexts,
"ground_truth": ground_truths,
}),
metrics=[faithfulness, answer_relevancy, context_precision, context_recall],
)
print(result)Run this on a representative test set before and after each change to the pipeline. The RAGAS scores will tell you whether a retrieval optimization actually improved generation quality or just looked better in isolation.
What to Track in Production
1
2
3
Retrieval: Hit Rate@K · MRR · NDCG · Context Precision / Recall
Generation: Faithfulness · Answer Relevancy · Hallucination Rate
System: Latency P50/P95 · Cost-per-query · ThroughputLLM-as-Judge for Qualitative Scoring
When you don't have ground truth labels, use a stronger LLM to score outputs:
1
2
3
4
5
6
7
8
9
10
11
JUDGE_PROMPT = """
Rate the following answer on a scale of 1-5 for:
- Faithfulness to context (1=hallucinated, 5=fully grounded)
- Relevance to question (1=off-topic, 5=directly answers)
Question: {question}
Context: {context}
Answer: {answer}
Provide scores as JSON: {"faithfulness": N, "relevance": N, "reasoning": "..."}
"""This isn't perfect but it's surprisingly good for catching obvious failures and gives you a cheap way to score large test sets without human annotation.
9 · Production Architecture Blueprint
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
┌─────────────────────────────────────────────────────────────────────┐
│ ADVANCED RAG SYSTEM │
│ │
│ INGESTION RETRIEVAL GENERATION │
│ Raw Docs Multi-Query Re-rank (RRF) │
│ ↓ Routing Chain-of-Note │
│ Chunking ┌─────────┐ LLM Answer │
│ ↓ │ Dense │ │
│ Multi-Rep Index │ BM25 │ → Merge │
│ (Summary + Full) │ ColBERT │ Fuse │
│ ↓ └─────────┘ Re-rank │
│ RAPTOR Tree │
│ ↓ AGENTIC LAYER │
│ Vector DB CRAG evaluator │
│ + DocStore Self-RAG loop │
│ Web fallback │
│ │
│ EVALUATION: RAGAS · TruLens · LLM-as-Judge · Latency tracking │
└─────────────────────────────────────────────────────────────────────┘Technology Stack
| Layer | Options | What I'd recommend |
|---|---|---|
| Vector DB | Chroma, FAISS, Pinecone, Weaviate | Pinecone or Weaviate for production |
| Embedding | OpenAI, Mistral, Cohere, BGE | BGE-M3 or Mistral Embed for cost |
| Retrieval | Dense, BM25, ColBERT via RAGatouille | Ensemble for best recall |
| Re-ranker | Cross-encoder, Cohere Rerank | Cohere Rerank for simplicity |
| Orchestration | LangChain, LlamaIndex, LangGraph | LangGraph for anything agentic |
| Evaluation | RAGAS, TruLens, DeepEval | RAGAS for automation |
10 · Implementation Patterns
Pattern 1: Full Advanced RAG Chain (LangChain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
from langchain.retrievers.multi_query import MultiQueryRetriever
from langchain.retrievers import EnsembleRetriever, BM25Retriever
from langchain_core.runnables import RunnablePassthrough
from langchain_core.output_parsers import StrOutputParser
multi_query = MultiQueryRetriever.from_llm(retriever=dense.as_retriever(), llm=llm)
ensemble = EnsembleRetriever(retrievers=[bm25, multi_query], weights=[0.4, 0.6])
con_prompt = ChatPromptTemplate.from_template("""
Write reading notes for the context, then answer the question.
Context: {context}
Question: {question}
Notes:
Answer:
""")
chain = (
{"context": ensemble | format_docs, "question": RunnablePassthrough()}
| con_prompt
| llm
| StrOutputParser()
)Pattern 2: CRAG with LangGraph
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
from langgraph.graph import END, StateGraph
workflow = StateGraph(GraphState)
workflow.add_node("retrieve", retrieve)
workflow.add_node("grade_documents", grade_documents)
workflow.add_node("generate", generate)
workflow.add_node("web_search_node", web_search)
workflow.set_entry_point("retrieve")
workflow.add_edge("retrieve", "grade_documents")
workflow.add_conditional_edges(
"grade_documents",
decide_to_generate,
{"web_search": "web_search_node", "generate": "generate"},
)
workflow.add_edge("web_search_node", "generate")
workflow.add_edge("generate", END)
app = workflow.compile()Pattern 3: ColBERT with RAGatouille
1
2
3
4
5
6
7
8
9
10
11
12
13
from ragatouille import RAGPretrainedModel
RAG = RAGPretrainedModel.from_pretrained("colbert-ir/colbertv2.0")
RAG.index(
collection=[doc.page_content for doc in docs],
document_ids=[str(i) for i in range(len(docs))],
index_name="production_index",
max_document_length=256,
split_documents=True,
)
results = RAG.search(query="What is the gross revenue of Company X in 2005?", k=10)Decision Tree: Where to Start
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
Is the query simple and well-formed?
├── YES → Standard dense retrieval may suffice
└── NO → Apply Query Transformation first
├── Ambiguous → Multi-Query + RAG-Fusion
├── Complex multi-step → Decomposition
└── Domain mismatch → HyDE / Step-Back
Do you have multiple data sources?
└── YES → Add Query Routing (Logical + Semantic)
Is recall still poor?
└── YES → Upgrade retrieval: DPR → ColBERT → Ensemble + RRF
Are hallucinations still occurring?
└── YES → Add Chain-of-Note post-processing
Is retrieved context unreliable?
└── YES → Add CRAG evaluator + web fallback
Need per-sentence grounding?
└── YES → Self-RAG with reflection tokens
Are you measuring quality?
└── Use RAGAS + TruLens RAG Triad continuouslyConclusion
Advanced RAG isn't one technique it's a layered set of interventions, each targeting a specific failure mode in the pipeline. The good news is you don't have to implement everything at once. Start with query transformation since it gives the biggest bang for the least engineering effort, then layer in better retrieval (ensemble + ColBERT), then add Chain-of-Note to cut hallucinations, and only add CRAG or Self-RAG once you're confident the retrieval itself is solid. Evaluate at every step with RAGAS so you actually know what's improving. The teams that build reliable RAG in production are the ones who treat each stage as an independent engineering problem rather than hoping the LLM will compensate for poor retrieval. It won't.
References
| Resource | Link |
|---|---|
| RAPTOR Paper | https://arxiv.org/pdf/2401.18059.pdf |
| CRAG Paper | https://arxiv.org/abs/2401.15884 |
| ColBERT Paper | https://arxiv.org/pdf/2004.12832.pdf |
| RAGAS | https://github.com/explodinggradients/ragas |
| RAGatouille (ColBERT) | https://github.com/bclavie/RAGatouille |
| LangGraph CRAG | https://github.com/langchain-ai/langgraph/blob/main/examples/rag/langgraph_crag.ipynb |
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article