RAG still answering wrong after you updated the docs? Check the librarian, not the temperature
A follow-up to my AIF-C01 concepts article: why an updated knowledge base can still give outdated answers, and how to separate freshness, retrieval and grounding problems.
In my last article, I explained RAG as a model that checks the manual before answering. Fernando Azevedo left a great follow-up question in the comments: what happens if the manual is updated, but the model still reads an old or irrelevant page?
That question deserves its own article. Thanks for the idea, Fernando! 🙌
Meet the three characters of RAG
To understand where RAG breaks, I like to imagine a library with three characters:
- The library is your knowledge base: the documents, split into chunks and stored as embeddings in a vector store.
- The librarian is the retrieval step: it searches the library and brings back the pages that look most relevant to the question.
- The writer is the foundation model: it reads the pages the librarian brought and writes the answer.
Here's the key point: the writer only knows what the librarian hands over. If the librarian brings an old page, the writer will answer confidently from that old page. The answer sounds right, but it's wrong.
Problem 1: the new page isn't on the shelf yet (freshness)
You updated the document in Amazon S3, but the library didn't notice. In Amazon Bedrock Knowledge Bases, changing the files in your data source isn't enough: you need to sync the data source so the new content is chunked, embedded and indexed.
It's like printing a new edition of a book and leaving it in a box at the library entrance. Until someone puts it on the shelf, the librarian keeps finding the old edition.
A common variation: both versions are on the shelf. If
policy-v1.pdf and policy-v2.pdf both live in the data source, the librarian may bring the old one simply because it looks more similar to the question.Fix: sync after every update, remove or archive outdated documents, and use metadata (like a date or version) to filter results.
Problem 2: the librarian picks the wrong page (retrieval relevance)
Sometimes the right page exists and is indexed, but it doesn't rank first. Retrieval is based on similarity, not truth. A page that repeats the same words as the question can beat the page that actually answers it.
Things that influence this: how documents are split into chunks, how many results come back, and how clear the question is.
Fix: test retrieval on its own. Bedrock Knowledge Bases has a
Retrieve API that returns only the chunks, without generating an answer. If the chunks are wrong, the problem is in the library or the librarian, not the writer.Problem 3: the writer ignores the page (grounding)
Now imagine the librarian brought the correct page, but the writer mixed it with what it "remembers" from training, or added details that aren't there. That's a grounding problem: the answer isn't supported by the source.
Fix: instruct the model to answer only from the provided context and to say when it doesn't know, ask for citations, and use the contextual grounding check in Amazon Bedrock Guardrails, which can flag responses that aren't grounded in the source or relevant to the question.
Why lowering the temperature won't save you
Temperature controls how the writer chooses words: low means predictable, high means creative. It has no effect on which page the librarian brings.
If the retrieved page is outdated, a low temperature just makes the model repeat the outdated answer more consistently. Consistent is not the same as correct.
A simple debugging checklist
- Is the new document indexed? Check if the data source was synced after the update.
- Is the old version still there? Remove it or filter by metadata.
- What did retrieval return? Look at the chunks before looking at the answer.
- Does the answer match the chunks? If not, it's a grounding issue.
- Only then tune generation settings like temperature.
💡 On the exam: answers outdated even after updating the documents point to data freshness (sync the knowledge base). Answers not supported by the source point to grounding checks. Temperature is about creativity versus consistency, never about which document was retrieved.
Quick summary
- Library (knowledge base): is the new content indexed? Is the old one gone?
- Librarian (retrieval): did it bring the right page?
- Writer (model): did it stick to the page?
- Temperature: changes the writing style, not the source.
Your turn
Have you ever had a RAG system answer from an old document? Where was the problem: the library, the librarian or the writer? Tell me in the comments! 📚
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article