When AI Should Say “I Don’t Know”: Building Uncertainty-Aware AI Agents on AWS
AI agents can produce confident answers even when evidence is incomplete or unreliable. Explore an uncertainty-aware architecture that helps agents evaluate evidence, verify uncertain results, and know when to answer, defer, or ask for human intervention.
When AI Should Say “I Don’t Know”: Building Uncertainty-Aware AI Agents on AWS
Introduction
AI agents are becoming capable of more than generating text. They can retrieve information, call APIs, use tools, make decisions, and perform actions on behalf of users.
But there is a fundamental problem:
What should an AI agent do when it does not have enough information to answer confidently?
Most AI applications are designed around a simple flow:
1
User → AI Model → Answer
This assumes that the model should always produce an answer.
In real-world applications, however, an incorrect answer can be worse than no answer at all. A financial assistant giving an unsupported recommendation, an enterprise agent using outdated documentation, or an operational agent executing an incorrect action can create significant consequences.
This leads to a different design principle:
A reliable AI agent should know when it has enough evidence to answer—and when it should verify, defer, or say “I don't know.”
This article presents an uncertainty-aware architecture for AI agents on AWS. Instead of treating every model response as equally trustworthy, the architecture introduces an additional decision layer that evaluates evidence, retrieval quality, answer support, and action risk before the agent responds or acts.
1. The Problem With Always-Answering AI
Large language models are optimized to generate useful responses. However, generating a fluent response does not guarantee that the response is correct.
Consider this question:
1
Who won a specific competition in a future year?
If the model has no reliable information about that event, a traditional application may still return a plausible-looking answer.
Now consider an enterprise scenario:
1
2
3
4
5
User:
"Can I delete the production database?"
Agent:
"Yes, you can delete it."
The problem is not only hallucination.
The deeper problem is that the system has no explicit mechanism asking:
1
Do I actually have enough evidence to make this decision?
A production AI system therefore needs another possible outcome besides "answer."
Instead of:
1
Answer
the system should support:
1
2
3
4
5
Answer
Verify
Ask for clarification
Request human approval
Say "I don't know"
This transforms uncertainty from an unexpected failure into an explicit part of the system design.
2. What Is Uncertainty-Aware AI?
Uncertainty-aware AI does not mean asking an LLM to produce an arbitrary confidence percentage and assuming that percentage is accurate.
For example:
1
"I am 92% confident."
does not necessarily mean the answer has a 92% probability of being correct.
Instead, an application can evaluate several observable signals.
For a retrieval-based agent, these signals can include:
- Quality of retrieved evidence
- Relevance of retrieved documents
- Agreement between multiple sources
- Whether the answer is supported by retrieved evidence
- Whether important information is missing
- Whether tool execution succeeded
- Whether the requested action is high risk
The system can then classify the result into a decision state.
1
2
3
4
5
6
7
8
Evidence Evaluation
│
┌──────────┼──────────┐
↓ ↓ ↓
Confident Uncertain Insufficient
│ │ │
↓ ↓ ↓
Answer Verify "I don't know"
The important idea is that uncertainty becomes an architectural decision rather than just a model output.
3. A Different Architecture for AI Agents
A conventional RAG or agent architecture might look like:
1
2
3
4
5
6
7
8
9
User
↓
Agent
↓
Retrieve / Use Tools
↓
LLM
↓
Answer
An uncertainty-aware architecture introduces a verification layer:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
┌───────────────────┐
│ User │
└─────────┬─────────┘
↓
┌───────────────────┐
│ AI Agent │
│ Reason + Plan │
└─────────┬─────────┘
↓
┌────────────────────────────┐
│ Evidence Collection │
│ │
│ Knowledge Base / Tools │
└─────────────┬──────────────┘
↓
┌────────────────────────────┐
│ Uncertainty Assessment │
│ │
│ Evidence │
│ Relevance │
│ Agreement │
│ Answer Support │
│ Action Risk │
└─────────────┬──────────────┘
↓
┌───────────────┼────────────────┐
↓ ↓ ↓
ANSWER VERIFY DON'T KNOW
│ │ │
↓ ↓ ↓
Respond Retrieve Again Escalate/
or Review Clarify
This architecture separates two responsibilities:
The model reasons about the problem.
The application decides whether the result is trustworthy enough to use.
That separation is important for production systems.
4. Building the Architecture With AWS
An implementation can combine several AWS services.
A possible architecture is:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
User
│
↓
Amazon API Gateway
│
↓
Agent Application
│
↓
Amazon Bedrock
│
┌──────────┴──────────┐
↓ ↓
Knowledge Retrieval Tools
│ │
↓ ↓
Amazon Bedrock AWS Lambda
Knowledge Bases │
│ │
└──────────┬──────────┘
↓
Uncertainty Layer
│
┌─────────────┼─────────────┐
↓ ↓ ↓
Answer Verify Escalate
│ │ │
↓ ↓ ↓
User More Evidence Human Review
│
↓
Amazon CloudWatch
The exact services can vary depending on the application, but the important architectural component is the uncertainty assessment layer.
5. Evidence Should Come Before Confidence
One of the most important principles in this architecture is:
Do not confuse model confidence with evidence quality.
Suppose a user asks:
1
What is our company's current leave policy?
The agent retrieves three documents.
1
2
3
Document A → Updated policy
Document B → Policy from two years ago
Document C → Employee handbook
A model might generate a confident answer even if Document B contradicts Document A.
A better system evaluates the evidence first.
For example:
1
2
3
4
5
6
7
8
9
10
11
Retrieved Evidence
↓
Is the source relevant?
↓
Is the source current?
↓
Do multiple sources agree?
↓
Does the proposed answer follow from the evidence?
↓
Decision
This creates a much stronger relationship between the answer and its supporting information.
6. An Uncertainty Decision Model
The uncertainty layer can combine multiple signals.
For example:
1
2
3
4
5
6
7
8
9
10
11
Evidence Score
+
Retrieval Relevance
+
Source Agreement
+
Answer-Evidence Alignment
+
Tool Success
-
Action Risk
The result does not need to be presented to the user as a mathematical probability.
Instead, the application can classify the state:
1
2
3
4
HIGH
MEDIUM
LOW
UNKNOWN
For example:
| State | Meaning | Agent behavior |
|---|---|---|
| HIGH | Strong evidence and low risk | Answer |
| MEDIUM | Some uncertainty exists | Verify |
| LOW | Weak or conflicting evidence | Ask or retrieve again |
| UNKNOWN | Evidence is insufficient | Say “I don't know” |
| HIGH RISK | Action could cause significant impact | Require approval |
The thresholds should be defined according to the application's requirements rather than copied from another system.
7. The “Verify” State
Uncertainty should not always immediately result in refusal.
There is a useful middle state:
Verify.
Suppose the agent retrieves one relevant document but the evidence is incomplete.
Instead of answering immediately:
1
Agent → Answer
the system can perform another retrieval:
1
2
3
4
5
6
7
8
9
First Retrieval
↓
Insufficient Evidence
↓
Second Retrieval
↓
Compare Evidence
↓
Final Decision
This allows the agent to recover from incomplete context.
For example:
1
2
if evidence_quality == "medium":
retrieve_additional_e
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article