Why I Picked GPT-4 Over Its Successors for My RAG Study Assistant: The Over-Refusal Problem
Follow-up to my RAG study assistant post: when I benchmarked GPT-4o and GPT-4o-mini against GPT-4 on grounded CFA questions, the newer models scored lower — not from hallucination, but from over-refusing questions the retrieved context already answered. Here is the evaluation, why prompting did not close the gap, and the trade-off I accepted.
Follow-up to my first post on fixing silent retrieval failures in a RAG-based CFA study assistant. Same project, different lesson — and it wasn't the one I expected.
The setup
After switching from paragraph splitting to semantic chunking, my CFA Level II study assistant was working well: both of my evaluation scores — an LLM-as-a-Judge groundedness score I built myself, and standard RAG metrics (context recall and faithfulness) — were climbing, and the worst failure mode (retrieval silently missing answers that were in the corpus) was under control.
Then I did what seemed like an obvious next step: try the newer, cheaper, faster models. If GPT-4o or GPT-4o-mini could match GPT-4's answer quality, there was no reason to keep paying GPT-4 prices on every question. So I ran a direct comparison.
The comparison
The evaluation design was the same one I used for the chunking fix, because fair comparisons need a fixed setup:
- A pool of 100 test questions built from the CFA curriculum, each with a known source passage so retrieval success was objectively checkable.
- A random sample of 20 questions used across all configurations — the same 20 every time, no cherry-picking.
- Identical retrieval for all three models — same embeddings, same top-k, same assembled context. The only variable was the model generating and being judged: GPT-4, GPT-4o, or GPT-4o-mini.
- Two independent scores on every run: my LLM-as-a-Judge score and the RAG metrics (context recall and faithfulness).
I ran each model to the same stopping rule I used throughout the project: I kept iterating — adjusting boundaries and re-running — until both scores crossed 90. Getting there took roughly a dozen hand-fix iterations on chunk boundaries along the way, by my estimate; I didn't log each one precisely, so I won't dress this up as more rigorous than it was.
The surprise: newer models scored lower
GPT-4o and GPT-4o-mini both came in below GPT-4 on the judge score and on the RAG metrics. Given how much better the newer models are on general benchmarks, I assumed I'd made a measurement error somewhere. I hadn't. The difference was real — and it was behavioral, not statistical noise.
When I inspected the failures question by question, the largest single category was not hallucination. It was over-refusal: the model answering "I don't know" or claiming the retrieved context was insufficient — on questions where the retrieved chunks demonstrably contained the answer.
Let that sink in, because the pattern is subtle: given a partial or densely written passage, the newer models defaulted to abstention instead of synthesizing what was in front of them. The content was there. The model just declined to use it.
Why a refusal counts as a wrong answer
I should be fair here: in many products, cautious refusal is a feature. A model that declines to answer when it's genuinely unsure is behaving responsibly, and I'd rather have a conservative assistant than a confident liar.
But context matters. This was a study tool. The user asked the question because they don't know the answer — and the system had already done the hard part: it retrieved the right material. Refusing at that point doesn't protect anyone; it just wastes the retrieval and leaves the student stuck. In a grounded question-answering setting, an unjustified refusal is functionally the same as a wrong answer, except it's wearing a lab coat.
I tried to prompt my way out of it first
Before concluding the models were different, I treated my own prompts as the suspect. I collected prompt-improvement suggestions from another AI tool, adapted the ones that made sense, and re-ran the full evaluation.
The result was instructive: formatting and instruction-following improved across all three models — answers looked cleaner and followed directions better. But the refusal gap did not close. GPT-4o and GPT-4o-mini kept refusing answerable questions at a much higher rate than GPT-4.
Once prompt quality was held roughly constant across the three runs and the refusal pattern persisted, I stopped treating it as something I could prompt my way out of. It was a model-behavior difference, and my decision had to respond to the behavior, not fight it.
The decision and its trade-off
I standardized on GPT-4 for both generation and judging. Let me name the caveat honestly: using the same model family for judging and generating is a real limitation of this evaluation — a judge that shares the generator's tendencies may grade its siblings more favorably. I didn't run a cross-model judge to check, so I'll state the comparison as what it was: an apples-to-apples comparison under a fixed methodology, with that confound on the table.
The trade-off I accepted was cost and speed. GPT-4 is the more expensive, slower option, and I gave up the cheaper per-question economics of the newer models. At my scale — a personal study tool where a wrong or missing answer costs me study time and erodes trust in the tool — accuracy dominated the economics. Latency stayed acceptable for interactive studying. I didn't run a formal cost or latency benchmark; that wasn't the deciding dimension, and I won't pretend I optimized for it.
What I took from this
- Newer ≠ better for every workload. General benchmark leadership doesn't automatically transfer to grounded, retrieval-constrained tasks. Test on your own distribution before assuming the upgrade is free.
- Measure refusal as its own metric. I almost missed this entirely because "score went down" hides the mechanism. Breaking failures into hallucination vs. over-refusal changed the decision — prompt tuning fixes the former more than the latter.
- Try the cheap fix before blaming the model — then trust the result. Prompt iteration was worth doing and I don't regret it. It just didn't work, and "it didn't work" is also a finding.
The system kept serving me through my study months, June to September 2026, with GPT-4 behind it. Not the cheapest choice. The right one for the job I gave it.
A question for other builders: have you seen newer models refuse more on grounded QA tasks, or is this specific to my setup? And if you track refusal rates separately from accuracy — what's your threshold for "too cautious to be useful"? I'd genuinely like to compare notes.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article