The Document Was Right There: Why Your RAG Answers Wrong
The model isn't the problem. When a RAG system answers wrong with the right document in the knowledge base, the failure is in chunking, retrieval, or ranking, here's how to tell which one.
The first time I debugged a RAG system that gave a wrong answer, I did what I think most people do: I blamed the model. I swapped it for a bigger one. I rewrote the prompt three times. I added "answer only from the provided context" in bold caps, as if shouting at a language model ever worked.
The answer was still wrong. And the document with the correct information was sitting right there in the knowledge base. I could search for it by hand and find it in two seconds.
That's when it clicked: the model never saw the document. It answered from whatever fragments of text got handed to it, and those fragments were the wrong ones. The generation step was working perfectly. It was faithfully summarizing garbage.
Most RAG debugging goes wrong because we picture the system as three boxes documents, model, answer when there are really five places for it to fail. Here are the three that break most often, and how to tell which one got you.
Failure 1: Chunking cut the answer in half
Before anything is embedded, your documents get sliced into chunks. This step is invisible, boring, and responsible for a large share of bad answers.
Picture a refund policy with a table: item categories down the left, return windows across the top. Fixed-size chunking with a 300-token limit cuts that table in the middle. Now one chunk holds "Electronics 14 days restocking fee applies" with no header saying what those numbers refer to, and another holds the header row with no data. Neither chunk answers "how long do I have to return a laptop?" and neither is wrong, exactly. They're just amputated.
Amazon Bedrock Knowledge Bases gives you several strategies for this, and picking one is a real design decision:
- Default chunking splits at roughly 300 tokens while respecting sentence boundaries.
- Fixed-size chunking lets you set the token count and an overlap percentage. Overlap is the cheap insurance policy it means a sentence near a boundary shows up in both neighbors.
- Hierarchical chunking indexes small child chunks for precision, then hands the model the larger parent chunk for context. You search narrow and read wide.
- Semantic chunking uses an embedding model to find where the topic actually shifts and splits there, instead of at an arbitrary token count. It costs more, because you're paying a model to do the splitting.
- No chunking treats each file as one chunk, which is the right call when you've already split your content into small, self-contained documents.
My rule of thumb now: if your content has structure tables, numbered clauses, Q&A pairs, API references fixed-size chunking will eventually slice through something that needed to stay together. Hierarchical is usually the better default for structured documents.
Failure 2: "Similar" is not "relevant"
Here's the part that took me longest to accept. Vector search doesn't find the correct chunk. It finds the nearest chunk in embedding space, which is a different thing wearing a convincing disguise.
Ask "can I return an item after 30 days?" and semantic search will happily hand you three chunks dense with return-policy vocabulary from the wholesale policy, the EU policy, and a deprecated 2023 policy. All highly similar. All useless. Meanwhile the chunk that actually answers the question phrases it as "eligibility window" and never uses the word "return" at all.
Two things help:
Hybrid search. Bedrock Knowledge Bases lets you set
overrideSearchType to HYBRID, which combines vector embeddings with raw text matching instead of using embeddings alone. This is what saves you on product codes, error numbers, proper nouns, and acronyms the exact tokens where semantic similarity is close to meaningless and literal matching is everything. Note the constraint: hybrid is supported on Amazon OpenSearch Serverless vector stores with a filterable text field, and on some other stores; plain SEMANTIC is the fallback everywhere else.Language match. Titan Text Embeddings V2 supports 100+ languages, but AWS explicitly warns that cross-language retrieval a knowledge base in one language, queries in another returns sub-optimal results. If your docs are in Portuguese and your users ask in English, that mismatch is a silent accuracy tax you'll never see in the logs.
Failure 3: the right chunk showed up in fifth place
Sometimes retrieval works and ranking doesn't. The correct chunk is in your result set just not near the top, and not inside the window you passed to the model.
This is what rerankers are for. A reranker takes the query and the retrieved documents, scores each document against the query with a model built specifically for relevance, and reorders them. Bedrock offers Amazon Rerank 1.0 (
amazon.rerank-v1:0) and Cohere Rerank 3.5 (cohere.rerank-v3-5:0) through the Rerank API, and you can wire them into Retrieve and RetrieveAndGenerate. Worth knowing: Amazon Rerank 1.0 isn't available in us-east-1, so if that's your region, Cohere is your option.Reranking also lets you invert the usual trade-off. Instead of stuffing 20 chunks into context and hoping, you retrieve 20, rerank, and pass the top 3. Fewer tokens, lower cost, better answer.
How to actually find which one broke
Stop debugging the answer. Debug the retrieval.
Call
Retrieve instead of RetrieveAndGenerate and look at the raw chunks that come back, with their scores. You are asking one question: is the correct information in this list at all?- Not there → retrieval failed. Look at chunking and search type.
- There, but ranked low → ranking failed. Add a reranker.
- There, ranked first, and the answer is still wrong → now it's the prompt or the model.
I've run this check dozens of times. The third case is the rarest by a wide margin.
The uncomfortable lesson is that RAG quality is mostly an information retrieval problem, and information retrieval was a mature discipline decades before any of us typed our first prompt. The foundation model is the least interesting part of the pipeline. It's the part that works.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article