My 5-check habit for RAG answers (after a week of getting them wrong)

For a week I kept losing practice points on RAG questions, and it was not because I did not know the theory.

I knew retrieval pulls chunks from a knowledge base, and the model answers from those chunks. What I did not have was a feel for what goes wrong in practice. So I built a tiny test: ten questions against a small set of docs, and I graded every answer by hand.

Here are the five checks that stuck with me.

1. Did it answer from the retrieved chunks, or from memory? When the chunks were thin, the model filled the gaps with confident guesses. Faithfulness first, that is what the exam calls it.
2. Did the retrieval actually fetch the right chunk? Two questions failed because the top results were near-misses. That is context recall, and it is a retrieval problem, not a model problem.
3. Were the chunks relevant, or just keyword-matching noise? One answer quoted a chunk that shared words with my question but said nothing useful. Context precision matters.
4. Did the answer change when I rephrased the question? Same meaning, different words, worse answer. A good retriever should not be that fragile.
5. Did I trust the answer too fast because it sounded good? Smooth writing is not proof. I now read the answer, then check the quoted chunks before I believe it.

What changed for me: I stopped asking "is the model smart enough" and started asking "is the retrieval good enough." Most RAG failures I have seen are retrieval failures wearing a model costume.

If you are studying for AIF-C01, try this on one small doc set before your next practice test. It is thirty minutes of work, and it makes the eval questions feel obvious.

Which of these five checks do you think you would skip first? Be honest.