How Semantic Chunking Fixed Silent Retrieval Failures in My RAG Study Assistant

*A personal project post-mortem — the bug wasn't the model, the prompts, or the vector database. It was how I cut up the documents.*

## Why I built it

CFA Level II material is dense and spread across thousands of pages, and generic chatbots are unreliable study partners: they hallucinate formulas and misquote curriculum text with complete confidence. So I built my own study assistant around retrieval-augmented generation (RAG), so every answer would be grounded in the actual source material, with citations.

The stack, briefly: the curriculum goes into a ChromaDB vector store; a LangChain retrieval layer assembles relevant passages into the model's context; a LangGraph multi-agent workflow sits on top, with separate agents for question generation, explanation, and verification; and an LLM-as-a-Judge step scores each generated explanation against the retrieved passages, so answer quality is measured rather than assumed. The app runs in Streamlit, containerized with Docker, and I deployed it on AWS EC2. I designed and implemented all of it myself, and used it as my own study tool from June to September 2026.

## The failure: answers that were there, but never found

The most significant failure was quiet. The assistant would tell me the material didn't cover a question — or answer from a near-miss passage — when the content was in the corpus the whole time.

I confirmed this properly instead of guessing: I logged the top-k retrieved chunks for test questions I knew the answers to, then located the answer text manually in the source. The answers existed. Retrieval just wasn't surfacing them.

## The root cause: paragraphs are typography, not meaning

My first chunking strategy split documents by paragraph. It sounds reasonable and it is the default in a lot of tutorials — but a paragraph is a typographical unit, not a semantic one, and CFA questions target concepts.

Two failure modes followed:

- **Split concepts.** A formula could sit in one paragraph while the conditions for using it were in the next. Neither chunk matched a question's embedding well enough to rank, so both were effectively invisible.
- **Diluted concepts.** A long paragraph covering three topics produced one chunk whose embedding was an average of all three — which polluted the top-k results for several topics at once without answering any of them well.

Two common "fixes" didn't work, for the same underlying reason: prompt tuning changed nothing, because a model cannot use context it never receives; and raising top-k just added more near-miss noise around the still-missing right chunk.

## The fix: semantic chunking, AI-assisted and human-verified

I replaced paragraph splitting with semantic chunking. An LLM proposed segment boundaries at concept level, so each chunk is self-contained: definition, conditions, and worked example kept together.

I want to be precise about my role here, because I think it's the part that actually made this work: I treated the AI's segmentation as a draft, not as ground truth. I sampled the output chunks and checked that no concept was cut mid-explanation and no chunk had silently merged two topics; I fixed boundaries where the model got it wrong; and I re-ran my test questions to compare retrieval before and after. AI did the tedious first pass. Verification was mine.

## How I proved it worked

An improvement you can't measure is an anecdote, so the evaluation mattered as much as the fix:

- I built a pool of **100 test questions** from the curriculum, each with a known source passage, so retrieval success was objectively checkable.
- For the before/after comparison, I **randomly sampled 20** of the 100 and used the same 20 for both configurations — random sampling, not the questions that made the change look good.
- I changed **only the chunking**. Same questions, same embeddings, same model, same top-k.
- I tracked **two independent scores**: my LLM-as-a-Judge groundedness score, and standard RAG evaluation metrics (context recall and faithfulness).

After the switch to semantic chunking, both scores improved noticeably — and the improvement was concentrated exactly where the diagnosis predicted: on the questions that had previously failed retrieval entirely. I didn't preserve the exact before/after figures, so I won't quote invented ones; what I kept is the design — a fixed random sample, a single changed variable, and two independent metrics moving in the same direction. That's what makes me confident the fix was real.

## A second finding: models fail differently on the same retrieval

The same evaluation sample let me compare GPT-4, GPT-4o, and GPT-4o-mini under identical retrieval. The newer models scored lower, and when I inspected their failures, the largest category wasn't hallucination — it was **over-refusal**: answering "I don't know" or claiming the context was insufficient on questions where the retrieved chunks demonstrably contained the answer.

I tried to prompt my way out of it first. I collected prompt-improvement suggestions, adapted the ones that made sense, and re-ran the evaluation. Formatting and instruction-following improved across all three models, but the refusal gap didn't close. So I standardized on GPT-4 for both generation and judging, accepting the cost and speed trade-off: for a study tool, a refusal when the material is present is still a wrong answer.

## What I took from this project

1. **Chunking is a design decision, not a preprocessing default.** In a RAG system it deserves the same care as model choice — it decides what the model ever gets to see.
2. **Evaluate before you optimize.** Logging retrieved chunks against known answers found the bug faster than any amount of prompt tuning would have.
3. **AI assistance works best with a verifier.** The LLM segmented thousands of pages far faster than I could have — and its output was only trustworthy because I sampled, corrected, and re-tested it.

The project is on GitHub with full setup and Docker deployment instructions, and I shared it with my CFA study circle — the pattern (ground the model in real sources, then evaluate it instead of trusting it) is the part I hope transfers to anyone building something similar.

**A question for other builders:** if you've built a RAG system, what chunking strategy worked for you — and how did you test it? I'd genuinely like to compare notes, especially on documents where formulas, tables, or conditions carry the meaning.