AWS Builder Center
The Paradox of Post-Training: How Fine-Tuning Teaches Language Models to Hallucinate with Confidence

The Paradox of Post-Training: How Fine-Tuning Teaches Language Models to Hallucinate with Confidence

Recent research shows that supervised fine-tuning and reinforcement learning - the standard methods for making AI useful - systematically teach models to fabricate information and present it with unwavering conviction.

The Student Who Forgot the Textbook

Imagine hiring a brilliant graduate. They studied for years, absorbing knowledge from every textbook in the library. You send them on an intensive course to specialise. They come back sounding polished, assertive, confident. But they have quietly forgotten some of what they originally knew, and they have no idea it happened. They do not fall silent on the topics they have forgotten. They keep talking, substituting fabrication for knowledge with the same poise they use for facts they genuinely remember.

This is what happens when we fine-tune large language models. Five papers published between February and August 2026 converge on a single conclusion: supervised fine-tuning (SFT) and reinforcement learning (RL) directly cause hallucinations and inflate model self-confidence. This is not a bug. It is a structural consequence of how these optimisation methods work.

What SFT Does to Knowledge

Before any fine-tuning, a pre-trained model is - perhaps surprisingly - better calibrated than its aligned descendants. It distributes probability across multiple plausible continuations, and that distribution reflects genuine uncertainty. A model claiming 90% confidence is wrong roughly 10% of the time. Imperfect, but real.
SFT breaks this in two ways at once.
SFT corrupts existing knowledge. When a model learns new facts through SFT, the new knowledge physically interferes with overlapping representations in parameter space. Gradient updates for new entities propagate through shared neural regions and garble nearby existing facts. The model does not notice. It continues generating at the garbled locations, producing fabrication with unchanged confidence. The most vulnerable facts - knowledge the model retrieves correctly perhaps 60 - 65% of the time - lost up to 18 -19 percentage points of accuracy under standard SFT. Edge-case knowledge, the kind that makes a model genuinely useful, is the first to go.
SFT teaches a confident voice that has nothing to do with accuracy. When researchers compared fine-tuned models against their base models across thousands of scientific prompts, the pattern was consistent and unsettling: fine-tuned models were internally less confident (flatter, noisier probability distributions) yet linguistically more assertive (more declarative, less hedged). They had absorbed the tone of expert writing without the expertise. The result held across every domain tested - life sciences, engineering, physical sciences, social sciences, arts and humanities. Not one field was spared.
The damage is not one-dimensional. Models fabricated unverifiable claims, overclaimed certainty, and attributed findings to sources that did not support them. These failure modes were largely independent - fixing one did not fix the others. Hallucination is not a single bug. It is a family.

What RL Does to Honesty

Then comes reinforcement learning - RLHF, DPO, GRPO - meant to make the model more helpful. It makes the overconfidence problem actively worse, through a mechanism that is almost comically self-defeating.
During preference training, human raters compare pairs of outputs and pick the better one. Humans prefer answers that sound clear and decisive. "The capital of Australia is Canberra" beats "I believe the capital might be Canberra, though I'm not entirely certain" - even though the second is more honest about the model's actual knowledge. The model learns this quickly. Sounding sure earns reward. Hedging earns penalty. Within a few training epochs, the model stops hedging entirely - not because it has become more knowledgeable, but because confidence is what gets reinforced.
This would be merely unfortunate if the RL objective could at least distinguish between correct confidence and misplaced confidence. Unfortunately it cannot. Standard RL rewards correct answers and penalises incorrect ones, but has no concept of appropriate confidence. A model that guesses boldly and hits 80% gets the same reward as a model that says "I'm 80% sure" and hits the same 80% - even though the second is far more useful, because you know when to trust it. There is no reward for honest uncertainty. The rational strategy is to always guess boldly.
So we arrive at a genuinely counterintuitive situation: correctness and honesty are competing objectives under standard RL. The harder you push a model to be correct, the less honest it becomes about what it does not know, because the pressure to produce a right answer overwhelms any incentive to express doubt. Experiments consistently show that combining correctness and calibration rewards in a single objective fails - the correctness signal drowns out the calibration signal, every time.

The problem is not limited to text. In vision-language models, RL fine-tuning raises benchmark scores while making reasoning unfaithful - models produce correct answers backed by incorrect reasoning, or the reverse. Simple misleading captions break RL-finetuned models that their un-finetuned base models handle without difficulty. Frotunately, in the vision-language domain specifically, closed-source frontier models show markedly greater resilience, suggesting the problem is not inherent to RL but reflects the current state of open-source training recipes.

What to Do About It

Fixing the training pipeline

One principled fix for SFT-induced hallucinations is self-distillation (Li & Hoiem, 2017; Kaplan et al., arXiv:2604.15574): add a KL divergence penalty to the fine-tuning loss to keep the model's output distribution anchored to the base model's, preventing the parameter updates that acquire new knowledge from interfering with existing factual representations:. Add a KL divergence penalty to the fine-tuning loss to keep the model's output distribution anchored to the base model's:
import torch.nn.functional as F

def sft_loss_with_self_distillation(logits_ft, logits_base, labels,
lam=1.0, tau=0.5):
"""SFT loss + KL divergence penalty against the base model.
Note: logits_base requires a frozen forward pass of the base model,
meaning both models must fit in GPU memory simultaneously.
For large models, use LoRA on the fine-tuned copy to reduce the
memory overhead, or compute base logits in a separate offline pass.
"""
loss_sft = F.cross_entropy(logits_ft.view(-1, logits_ft.size(-1)),
labels.view(-1), ignore_index=-100)
p_ft = F.log_softmax(logits_ft / tau, dim=-1)
p_base = F.softmax(logits_base / tau, dim=-1)
loss_kl = F.kl_div(p_ft, p_base, reduction="batchmean") * (tau ** 2)
return loss_sft + lam * loss_kl

This is not generic regularisation. It specifically prevents gradient updates from propagating through shared representational regions and displacing existing facts. In experiments, it recovered up to 90% of the knowledge lost to standard SFT. Generic ℓ₂ regularisation did not help - the fix works because it targets the interference mechanism directly. If you are using a managed fine-tuning API that does not expose the loss function, this technique is not directly applicable - but understanding why it works will help you evaluate which managed services are likely to produce better results.
Self-distillation protects what the model already knows, but it does not teach the model to say "I don't know." For that, you need to change the training data itself: include 20–25% of prompts where the correct response is an honest refusal. Without this, the model learns from thousands of examples that the expected behaviour is always to produce an answer, and it will invent one rather than stay silent.
For the RL stage, the intuitive fix - adding a calibration term directly to the GRPO reward - has been found ineffective. GRPO computes rewards at the rollout level, and models assign near-1.0 confidence to most claims during training. The calibration reward ends up tracking the factuality reward almost exactly, carrying no independent signal.
The alternative approach that has shown consistent results is CURE (Liu & Wang, "Think Through Uncertainty," arXiv:2604.12046). It improves long-form factuality by teaching models to reason about uncertainty at the claim level rather than optimizing solely for correctness. Its core mechanism is a claim-aware reasoning protocol that decomposes responses into atomic claims, each paired with an explicit confidence estimate. The training pipeline has three sequential stages. First, SFT followed by a warm-up GRPO pass establishes the structured reasoning format - without this, the model cannot reliably produce calibrated claim decompositions. Second, DPO aligns model-predicted confidence with empirical correctness: operating on output pairs, it directly targets the gap between stated confidence and actual accuracy in a way that correctness-only RL rewards do not address. Third, GRPO with token-masked rewards optimizes claim-level factuality, with the confidence tokens masked out so that factuality training cannot overwrite the calibration established in stage two:
def compute_masked_grpo_reward(claims, confidences, verifier):
"""
Pseudocode: GRPO factuality reward with confidence tokens masked.
In practice, masking is implemented via token-level loss masks
(zeroing advantage weights on confidence token positions)
during the GRPO policy gradient step.
"""
rewards = []
for claim, conf in zip(claims, confidences):
is_correct = verifier.check(claim)
claim_reward = 1.0 if is_correct else -1.0
rewards.append({
"claim_tokens": claim.token_ids, # receive gradient
"conf_tokens": conf.token_ids, # masked - no gradient
"reward": claim_reward,
})
return rewards

The difference in output is significant. When asked to write a biography of David Bowie, a CURE-trained model assigned 0.98 confidence to "born January 8, 1947, in London" (correct), 0.95 to "changed his name in 1966" (correct), and 0.30 to "passed away on March 3, 2016" — flagging its own uncertainty about a date that was indeed wrong (the actual date was January 10). Standard RL baselines, which optimise only for correctness, do not produce this signal: they allow overconfident errors to persist without distinguishing reliable claims from speculative ones. In benchmarks, CURE improved claim-level factual accuracy by up to 39.9% on Biography generation over competitive RL baselines, and improved calibration by 16.0% in AUROC on FactBench — making the confidence scores a reliable signal for downstream selective prediction.

When you cannot retrain

If you are deploying an off-the-shelf model without access to the weights, you still have options.
The most widely deployed defence is retrieval-augmented generation (RAG). Ground the model's output against an external knowledge base at query time. RAG does not fix the underlying calibration problem - the model still does not know what it does not know - but it provides a factual backstop that catches many fabrications before they reach the user. If you are running any production LLM application, RAG should be your baseline, not an afterthought.
Beyond RAG, prompt engineering for honesty is the simplest calibration-specific technique: modify the system prompt to explicitly penalise confident guessing. Something like "+10 for a verified fact, +5 for admitting uncertainty, −100 for a confident fabrication." This is crude, and no one would call it elegant, but it shifts the model's internal cost–benefit calculation enough to matter - and it costs nothing to implement.
If you are self-hosting (vLLM, TGI, or similar - not a managed API like Bedrock, which typically do not expose per-token logprobs), you can monitor the model's log-probabilities during generation. Every token has a probability the user never sees but the serving infrastructure can inspect. When names, dates, or quantities are generated with unusually low confidence, flag the output - either for self-correction (feed the low-confidence passage back with a correction prompt) or for RAG grounding (use the uncertain claim as a retrieval query). This catches many fabrications before they reach the user.
For self-hosted deployments where you have access to model internals, the most powerful approach is probing the model's hidden states. Even after alignment distorts the surface probabilities, the internal activations - in the middle layers of the network - often remain better calibrated. A lightweight probe trained on these activations can reduce calibration error by up to 96%, recovering the honest uncertainty that alignment buried. For high-stakes applications - clinical decision support, legal analysis, scientific research - this is the technique that makes confidence estimates trustworthy enough to act on.

A working example: Qwen3.8 inference on vLLM

The following implements the inference-time pipeline - prompt incentivisation, logprob monitoring, and automatic self-correction - against a Qwen3.8 model served by vLLM:
import httpx, math

VLLM_URL = "http://localhost:8000/v1/chat/completions"

SYSTEM_PROMPT = """
You are a precise, calibrated assistant. Accuracy takes priority over helpfulness.
- Correct factual statement: +10 points.
- Stating "I do not know" on an unverified claim: +5 points.
- False or unverified claim stated with confidence: -100 points.
If certainty is below 80%, abstain or express doubt.
"""

def generate_with_confidence(prompt: str, threshold: float = 0.75):
payload = {
"model": "Qwen/Qwen3.8-27B",
"messages": [
{"role": "system", "content": SYSTEM_PROMPT},
{"role": "user", "content": prompt},
],
"temperature": 0.1,
"logprobs": True,
"top_logprobs": 1,
}

resp = httpx.post(VLLM_URL, json=payload, timeout=60.0).json()
choice = resp["choices"][0]
text = choice["message"]["content"]
token_data = choice["logprobs"]["content"]

# Find substantive tokens with low confidence
low = [(t["token"], round(math.exp(t["logprob"]), 3))
for t in token_data
if len(t["token"].strip()) > 2
and t["token"].isalpha()
and math.exp(t["logprob"]) < threshold]

if not low:
return text

# Self-correction: flag uncertain tokens, ask for rewrite
correction = {
"model": "Qwen/Qwen3.8-27B",
"messages": [
{"role": "system", "content": SYSTEM_PROMPT},
{"role": "user", "content": prompt},
{"role": "assistant", "content": text},
{"role": "user", "content":
f"AUDIT: Unverified assertions on {[t[0] for t in low]}. "
f"Rewrite to fix errors or state 'I don't know'."},
],
"temperature": 0.0,
}
return httpx.post(VLLM_URL, json=correction, timeout=60.0
).json()["choices"][0]["message"]["content"]

Qwen3.8 is a particularly good fit for this two-pass pattern because its native  <thinking>  blocks preserve the reasoning trace through the correction loop. The model can trace back to where its generation deviated from grounded reasoning, rather than starting from scratch.

Closing Thought

There is an irony at the heart of all this. The methods we built to make language models helpful are the same methods that make them unreliable. Alignment, as currently practised, trades calibration for compliance.
The pre-trained model - raw, unhelpful, unable to follow an instruction - had one virtue that post-training systematically destroys: it was relatively honest about what it did not know. That honesty is recoverable. Self-distillation, calibrated reward design, refusal-aware training data, logprob monitoring, hidden-state probes = the techniques exist and they work. What they require is a decision to treat calibration as a first-class engineering objective, not an afterthought bolted on after deployment.
As AI moves into medicine, science, law, and policy, a model that says "I don't know" when it does not know is not a failure of helpfulness. It is the important feature we can build.



Sources

  • Kaplan, G., Gekhman, Z., Zhu, Z., Rozner, L., Reif, Y., Swayamdipta, S., Hoiem, D., & Schwartz, R. (2026). Why Fine-Tuning Encourages Hallucinations and How to Fix It. Preprint, under review. arXiv:2604.15574.
  • Abu Ahmad, R., Rauscher, N., Borisova, E., Barth, F., Rehm, G., & Möller, S. (2026). Finetuning with Scientific Data Increases Hallucinations: A Multi-domain Factuality Evaluation of LLMs. arXiv preprint. arXiv:2606.21359.
  • Liu, X. & Wang, L. (2026). Think Through Uncertainty: Improving Long-Form Generation Factuality via Reasoning Calibration. Preprint, under review. arXiv:2604.12046.
  • Berkowitz, J. S., Patock, J. R., Nawaz, A., Gonzalez-Hernandez, G., & Tatonetti, N. P. (2026). A Crisis of Overconfidence: Why Confidence, Not Accuracy, Is the Real Risk in Clinical AI. BioData Mining, 19, 10.
  • Zhao, R., Shah, A., Zhu, X., Deng, X., Jiang, Z., Yang, Y., Liebelt, J., & Mondal, A. (2026). On Robustness and Chain-of-Thought Consistency of RL-Finetuned VLMs. arXiv preprint. arXiv:2602.12506.
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article