Fix the Corpus Before You Touch the Prompt
I have not built my RAG yet. This is the plan: fix the data before I touch the retriever, the prompt, or the model.
I am building a Hybrid-Search RAG with citation verification. I have not built it yet. Today. So this post is the plan, and the trap I am trying not to walk into.
The failure I am actually scared of is not my retriever is bad. It is this: I spend two days tuning chunk overlap, rerankers,
k, and hybrid weights -> and at the end I discover the answer was never in my documents. Not badly indexed. Never there. Every hour I spent on the vector store was an hour I should have spent reading the source.1
2
3
4
5
6
7
8
9
10
11
12
13
14
question
|
v
+---------------------------------------+
| does the answer exist in the corpus? | <-- most RAG bugs stop and die here
+---------------------------------------+
| no | yes
v v
fix the corpus retrieve -> rank -> generate
(CMS is hand-edited, |
PDF is a wall of text) v
^ verify every claim
| against the passage
+---------------------------------+That loop on the left is boring. That is why it is skipped.
First: fix the number that is not one
I do not need a vector database to know my corpus is broken. I already know it.
The phone-accessory shop menu is edited by hand in a CMS. Prices live in the CMS. Stock lives in the CMS. Opening hours live on a sign I painted last summer. Three sources of truth, and all three already disagree. I know because I checked, once, in August, and then closed the tab and went back to CSS.
Then there is the diet-plan PDF. Forty pages, no headings, no tables, two fonts. Basically a wall of text. Chunking that document well is not a retrieval problem. Chunking that document well means rewriting it.
Here is the part that stings, because I already paid for this lesson in production. My contact-form validator checked
nom, message and reference -> and forgot telephone. No error thrown. No log line. A full week of requests with no phone number, and I only found out because a client said so.A missing field is silent. That is the whole lesson. An unvalidated CMS export is the same bug with a bigger blast radius: I will embed a wrong price and the model will quote it with a citation and a straight face.
Tuning the retriever vs. fixing the corpus
| Dimension | Tuning the retriever | Fixing the corpus |
|---|---|---|
| Time cost | Hours to days. Feels productive. | Hours. Mostly reading. |
| What it fixes | Which of the documents you already have get ranked higher. | Whether the right answer exists at all, and whether it is current. |
| How you know it worked | Recall@k goes up on your eval set. | Your eval set stopped needing excuses. You can answer a question you did not write the eval for. |
| Cost | Almost nothing. Embeddings are pennies. | My time. The most expensive line item in the project. |
| Failure signature | Metrics wobble. Results look plausible. | Metrics look perfect and the answers are still wrong. |
| Reversible? | Yes, instantly. | Yes, but only once. |
The last row is the one that decided it for me. Retriever params are free to change. Data entry is not.
Phase 0, before I embed anything
- One row per claim. Not one document per menu page. Price: 4500 DA. Stock: in, 12 units. Closed Tuesday. Three rows, three sources, three timestamps.
- One owner per fact. The CMS is not automatically the truth. I decide what is, and I write that decision down.
- A freshness stamp on every row. If I cannot say when a price was last true, I do not get to quote it.
- Conflict detection before embedding. Two rows that disagree is a data-entry bug. Log it. Fix it. Do not let the model arbitrate.
- Rewrite the wall-of-text PDF by hand. Not the whole thing. The parts I would actually be asked about.
Embedding first means I embed the mess with extra steps.
The one AWS call I am confident enough to write
I will use Amazon Bedrock for generation and embeddings. I am not writing the ingestion code yet. I am writing the query path, because that is the part with the interesting failure modes.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
import os
import boto3
# Why a placeholder and not a real ID: model IDs are region-specific and they change.
# Check the Bedrock Model List for YOUR region before you copy anything from a blog post
# (including this one).
MODEL_ID = os.environ.get("BEDROCK_MODEL_ID", "REPLACE_ME")
REGION = os.environ.get("AWS_REGION", "eu-west-3")
client = boto3.client("bedrock-runtime", region_name=REGION)
SEARCH_TOOL = {
"toolSpec": {
"name": "search_documents",
"description": "Search the shop knowledge base for prices, stock and opening hours.",
"inputSchema": {
"json": {
"type": "object",
"properties": {
"query": {"type": "string", "description": "Natural language search query."},
"top_k": {"type": "integer", "description": "How many passages to return."},
"filter": {
"type": "object",
"description": "Optional metadata filter, e.g. source=menu.",
"additionalProperties": True,
},
},
"required": ["query"],
}
},
}
}
def log_usage(label, response):
usage = response["usage"]
# Why: cost is a design constraint. If tokens are invisible, they are not budgeted.
print(
f"[{label}] in={usage['inputTokens']} "
f"out={usage['outputTokens']} "
f"total={usage['inputTokens'] + usage['outputTokens']}"
)
def ask(question, history):
return client.converse(
modelId=MODEL_ID,
messages=history,
toolConfig={
"tools": [SEARCH_TOOL],
# Why pin it: let the model decide whether to retrieve and it will
# sometimes answer from its own weights instead. You get a confident
# sentence with no citation, and no signal that retrieval never ran.
"toolChoice": {"tool": {"name": "search_documents"}},
},
inferenceConfig={"maxTokens": 600, "temperature": 0},
)Then the loop: if
stopReason == "tool_use", run the search, append the toolUse block and a matching toolResult, call converse again. Retrieval runs first, always. Not because the tool is smart, but because the model is confidently wrong and cheap.temperature=0 too. I want the same question to give the same wrong answer twice so I can debug it.Citation verification is not a string compare
The naive version is: does the answer text appear in the source? If yes, ship it. This proves almost nothing.
It proves the sentence was near the passage. It does not prove the passage supports the claim, that the number is the current number, or that the model did not merge two facts into one sentence that appears nowhere.
What I actually want:
- Entailment per claim, not per document. Split the answer into claims. Ask: does this passage state this? If no, the citation is decoration.
- Numeric equality for prices. Not fuzzy.
4500is not450and not4500 DA including delivery. - Freshness check. If the cited row is older than the staleness window I set, the answer is wrong even when it is perfectly supported.
- Abstention as an output. No passage above a similarity floor means I don't know. Not a best guess.
So verification is a second model call, and it costs tokens. I will say that out loud now instead of pretending it is free.
The one mistake I expect to make
I will ship the auto-retrieval version first, because it looks cleaner in a demo. I will let the model choose whether to call the tool, I will see nice answers in my tests, and I will spend three weeks finding the questions where it answered from memory with a citation-shaped sentence and no retrieval behind it.
Prevention rule: pin the tool, and log every retrieval call with its query, its hit count, and its latency. Treat
hits == 0 as a corpus bug until I have proof otherwise. If I cannot see the retrieval, I cannot debug the answer.Ironically this is the same lesson as the
telephone field. Silence is not success. Silence is an unhandled case.What $30 actually buys me
Embedding is the cheap part. A 40-page PDF at roughly 500 tokens a page is about 20k tokens of text. At 800-token chunks that is around 25 chunks. Twenty-five embeddings cost me fractions of a cent. Generation costs more than retrieval does, because I am re-sending passages every turn and running a second verification pass on every answer.
The real tradeoffs:
- Bigger chunks -> fewer vectors, cheaper ingestion, fewer tokens in context. Worse precision. A 2000-token chunk around a price table drags in opening hours and the model starts averaging them.
- Smaller chunks -> better precision, more vectors, and eventually the answer gets split across two chunks that each look incomplete.
- Re-ingestion -> the cost that sneaks up. Every hand edit to the CMS means re-embedding changed rows only. Hash the content, skip unchanged rows. A full re-index on every menu change is how you burn a credit balance by accident.
- The verifier -> roughly doubles generation tokens. Cheaper than being wrong in public.
And the expensive one, which is not on any invoice: my time reading my own client's data and writing down which source wins. That is the budget line that actually decides whether this project ships.
Prevention rules so far
- A backup you never restored is not a backup. Same energy: an eval set I never ran is not an eval set.
- Ship the boring thing first. A CSV of hand-checked facts beats a vector store for a very long time.
- If I cannot log it, I cannot fix it.
- One source of truth per fact, and it gets a timestamp.
Next post: the Phase 0 corpus of my own client's menu, cleaned by hand, with the conflicts I found listed. If it goes badly, I will write that too.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article