5 Lessons from Building a PDF-Grounded RAG Chatbot (While It's Still in Progress)

I'm a backend developer at Zyroo, and since August I've been building a RAG pipeline: a chatbot where a user uploads a PDF and asks questions about it. It's not finished, and that's exactly why I'm writing this. Most RAG tutorials walk the happy path on clean, well-formatted text. Real PDFs broke nearly every assumption I started with. Here's what actually helped.

## 1. PDFs aren't documents — they're printing instructions

Text extracted from a PDF comes out in layout order, not reading order. Multi-column pages read straight across columns. Headers and footers repeat on every page. Tables shred into fragments. None of this is visible until you actually read what your extractor hands you.

The lesson: your ingestion step deserves as much engineering as your retrieval step. A cleanup pass — stripping repeated headers and footers, repairing reading order, handling broken table rows — made a bigger difference to answer quality than anything I did later in the pipeline. I now treat ingestion as a real stage with its own checks, not a one-liner you run once and forget.

## 2. Chunk by structure, not just by character count

The default advice — fixed-size chunks with generous overlap — gave me answers that started mid-sentence in the middle of a table. What worked better was splitting on document structure first: headings, paragraphs, table boundaries — then falling back to size limits within those units.

Overlap still matters, but it matters where it's placed. Overlap across a section boundary is mostly wasted context. Overlap inside a coherent section helps the model resolve pronouns and cross-references. Let the document tell you where the cuts go instead of imposing an arbitrary grid on it.

## 3. Retrieval quality is the whole game — measure it before you touch the LLM

Early on, every bad answer looked like a language model problem. It wasn't. Most were retrieval problems: the right chunk never made it into the top-k results, or it was there but buried under irrelevant ones.

The fix was embarrassingly simple. I wrote a small set of questions by hand, each with a known answer from the PDF, and after every pipeline change I checked one thing: was the relevant chunk retrieved? No fancy eval framework, no scores — just a list of questions and a yes/no per question. When retrieval improved on that set, answer quality improved. When I "fixed" things without that check, I was guessing. Build the check before you need it; you'll need it constantly.

## 4. Top-k is a budget, not a number

It's tempting to stuff more chunks into the prompt — more context, more knowledge. In practice, more chunks meant more noise the model had to sort through, plus real cost on every query. I started thinking of context as a budget: enough chunks to cover the question, few enough that the signal survives.

The telling moment was when answers got worse after I widened the window. That was the signal to tighten retrieval — better chunking, filtering, re-ranking — instead of widening the funnel. If adding context degrades answers, your retrieval is the problem, not your context size.

## 5. Ground every answer in its source, or users won't trust it

For a chatbot over someone's documents, a confident wrong answer is worse than "I couldn't find that." Returning the source chunk alongside each answer did two things at once: users could verify what they were told, and I could see exactly which retrieval decisions produced good or bad answers — which fed straight back into lesson 3's question set. Citations aren't just a trust feature; they're your debugging lens.

## The unglamorous work is the work

If you're building RAG over PDFs, the unglamorous parts — ingestion cleanup, chunking strategy, a small hand-built eval set — matter more than the model you pick. The LLM is the easy part. Get the pipeline honest first, then let the model do its job.