I Built a RAG Pipeline. Then I Realized Retrieval Wasn't the Hard Part.
A practical journey into why AI applications can fail even when the LLM is working perfectly.
Why a working RAG pipeline can still give the wrong answer
You upload documents, generate embeddings, store them in a vector database, connect a retriever to an LLM—and it works.
Until someone asks a precise question whose answer is buried deep in a long document.
The model responds confidently, but incorrectly.
Sometimes the problem isn't the LLM or the prompt. The evidence never reached the model.
RAG is not just about generating answers. It's about retrieving the right evidence before generation begins.
A RAG pipeline, at a glance
INGESTION
Documents (S3) → Extraction → Chunking → Embeddings → Vector Store
You upload documents, generate embeddings, store them in a vector database, connect a retriever to an LLM—and it works.
Until someone asks a precise question whose answer is buried deep in a long document.
The model responds confidently, but incorrectly.
Sometimes the problem isn't the LLM or the prompt. The evidence never reached the model.
RAG is not just about generating answers. It's about retrieving the right evidence before generation begins.
A RAG pipeline, at a glance
INGESTION
Documents (S3) → Extraction → Chunking → Embeddings → Vector Store
QUERY
User Question → Retrieval → Reranking → Context → Amazon Bedrock → Answer + Sources
On AWS, a practical implementation could use Amazon S3 for documents, Amazon Bedrock for embeddings and generation, OpenSearch or PostgreSQL with pgvector for retrieval, and API Gateway + Lambda to expose the application.
The architecture is simple. Making it reliable is the engineering challenge.
1. Chunking can quietly destroy context
Consider this sentence:
Failover is initiated when the primary node fails to respond to three consecutive health checks within a 15-second interval.
If a chunk boundary splits that sentence, retrieval may return an incomplete fragment. The information exists—but its meaning is harder to recover.
Chunk size and overlap are therefore more than tuning knobs: they influence what context survives retrieval.
def chunk_text(text, chunk_size=500, overlap=100):
chunks = []
start = 0
User Question → Retrieval → Reranking → Context → Amazon Bedrock → Answer + Sources
On AWS, a practical implementation could use Amazon S3 for documents, Amazon Bedrock for embeddings and generation, OpenSearch or PostgreSQL with pgvector for retrieval, and API Gateway + Lambda to expose the application.
The architecture is simple. Making it reliable is the engineering challenge.
1. Chunking can quietly destroy context
Consider this sentence:
Failover is initiated when the primary node fails to respond to three consecutive health checks within a 15-second interval.
If a chunk boundary splits that sentence, retrieval may return an incomplete fragment. The information exists—but its meaning is harder to recover.
Chunk size and overlap are therefore more than tuning knobs: they influence what context survives retrieval.
def chunk_text(text, chunk_size=500, overlap=100):
chunks = []
start = 0
while start < len(text):
end = start + chunk_size
chunks.append(text[start:end])
start += chunk_size - overlap
end = start + chunk_size
chunks.append(text[start:end])
start += chunk_size - overlap
return chunks
This is a simple character-based example, not a production tokenizer-aware chunker. For structured documents, recursive, semantic, or parent-child chunking may preserve meaning better.
2. Similarity is not relevance
A vector search may return passages about monitoring, node architecture, and health checks. Only one may contain the exact threshold the user asked for.
Semantic similarity finds related content; relevance asks whether that content helps answer the query.
One way to improve coverage is hybrid search: combine vector search with keyword retrieval such as BM25, then fuse and rerank the results.
Query
/ \
Vector Search BM25
\ /
Result Fusion
|
Reranker
|
Top-k Context
Reciprocal Rank Fusion is one method for combining rankings:
[ RRF(d)=\sum_{r\in R}\frac{1}{k+\operatorname{rank}_r(d)} ]
The objective isn't to retrieve the most text. It's to retrieve the most useful evidence.
3. Evaluate retrieval before blaming the LLM
A response can sound convincing and still be unsupported.
Evaluate the pipeline in two stages:
Retrieval: Recall@K checks whether relevant evidence appears in the top K results; MRR measures how highly the first relevant result ranks.
Generation: Faithfulness checks whether the answer is supported by context; answer relevance checks whether it addresses the question.
If retrieval failed, prompt tweaks may only hide the symptom.
Debug the stage that failed—not the stage that's easiest to change.
4. Reliable systems know when not to answer
If the knowledge base doesn't contain an incident's exact downtime, the system shouldn't invent a number from a similar report.
A generation instruction can require the model to use supplied evidence, cite sources where possible, and say when the answer is unavailable. This doesn't eliminate hallucinations, but it makes the desired behavior explicit.
For production, I would also separate ingestion from query-time work and consider asynchronous processing with SQS, document versioning, access-aware metadata filters, retries, and CloudWatch monitoring.
A RAG application is a data pipeline, a search system, and a generation system working together.
What I'd do differently next time
I'd begin with a small evaluation set: realistic questions, expected answers, and the source passages needed to answer them.
Then I'd compare chunking strategies, vector versus hybrid retrieval, and reranking—measuring each change before tuning the prompt.
The most capable LLM cannot reliably use evidence it never received.
Did we build a system that can find the evidence before asking the model to reason about it?
Your turn
What has been the hardest part of your RAG implementation: chunking, retrieval quality, hallucinations, latency and cost, or evaluation?
I'd love to hear what broke—and what you changed to fix it.
This is a simple character-based example, not a production tokenizer-aware chunker. For structured documents, recursive, semantic, or parent-child chunking may preserve meaning better.
2. Similarity is not relevance
A vector search may return passages about monitoring, node architecture, and health checks. Only one may contain the exact threshold the user asked for.
Semantic similarity finds related content; relevance asks whether that content helps answer the query.
One way to improve coverage is hybrid search: combine vector search with keyword retrieval such as BM25, then fuse and rerank the results.
Query
/ \
Vector Search BM25
\ /
Result Fusion
|
Reranker
|
Top-k Context
Reciprocal Rank Fusion is one method for combining rankings:
[ RRF(d)=\sum_{r\in R}\frac{1}{k+\operatorname{rank}_r(d)} ]
The objective isn't to retrieve the most text. It's to retrieve the most useful evidence.
3. Evaluate retrieval before blaming the LLM
A response can sound convincing and still be unsupported.
Evaluate the pipeline in two stages:
Retrieval: Recall@K checks whether relevant evidence appears in the top K results; MRR measures how highly the first relevant result ranks.
Generation: Faithfulness checks whether the answer is supported by context; answer relevance checks whether it addresses the question.
If retrieval failed, prompt tweaks may only hide the symptom.
Debug the stage that failed—not the stage that's easiest to change.
4. Reliable systems know when not to answer
If the knowledge base doesn't contain an incident's exact downtime, the system shouldn't invent a number from a similar report.
A generation instruction can require the model to use supplied evidence, cite sources where possible, and say when the answer is unavailable. This doesn't eliminate hallucinations, but it makes the desired behavior explicit.
For production, I would also separate ingestion from query-time work and consider asynchronous processing with SQS, document versioning, access-aware metadata filters, retries, and CloudWatch monitoring.
A RAG application is a data pipeline, a search system, and a generation system working together.
What I'd do differently next time
I'd begin with a small evaluation set: realistic questions, expected answers, and the source passages needed to answer them.
Then I'd compare chunking strategies, vector versus hybrid retrieval, and reranking—measuring each change before tuning the prompt.
The most capable LLM cannot reliably use evidence it never received.
Did we build a system that can find the evidence before asking the model to reason about it?
Your turn
What has been the hardest part of your RAG implementation: chunking, retrieval quality, hallucinations, latency and cost, or evaluation?
I'd love to hear what broke—and what you changed to fix it.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article