Teach AI Agents to Say “I Don’t Know” Before They Cause Damage
Build evidence-aware AI agents with Amazon Bedrock, AWS Lambda, and Step Functions to reduce unsupported diagnoses and prevent unsafe production actions.
Building evidence-aware AI agents on AWS that know when to answer, investigate, or ask for human approval.
An AI agent can generate a convincing answer, call an API, analyze a log, and even trigger a workflow. But there is one question we should ask before giving an agent access to production systems:
What happens when the agent doesn't have enough information, but acts as if it does?
Imagine an AWS Glue job fails at 2 AM. An AI agent examines the CloudWatch logs and concludes that the job ran out of memory. It increases the worker count and retries the job.
But the actual problem was an S3 access-denied error.
The retry fails again. The additional compute increases the cost, and the underlying permission issue remains unresolved.
The agent wasn't necessarily incapable of analyzing logs. It made a decision without sufficient evidence.
This is one of the challenges of bringing AI agents into production. We often focus on making agents more capable, but we also need to design systems that prevent them from acting when their evidence is incomplete.
In this article, we'll build an evidence-aware AI agent workflow using AWS services. The goal is simple: when an agent cannot justify its conclusion, it should investigate further, abstain, or request human approval instead of guessing.
1. Why telling an AI agent to be honest isn't enough
A common approach to handling uncertainty is adding an instruction to the system prompt:
"If you don't know the answer, say you don't know."
This is useful, but it is not a complete safety mechanism.
A language model can still produce an incorrect answer with confident wording. It can misinterpret retrieved information, overlook a relevant log entry, or recommend an action based on an assumption.
Asking the model how confident it feels is not enough either. A self-reported confidence score is not automatically a reliable measure of correctness.
For a production system, we need something stronger than a prompt instruction.
We need an external decision layer that evaluates the available evidence and enforces rules before the agent can perform consequential actions.
For example:
- If the required information is available and verified, allow the agent to provide an answer.
- If evidence is missing, let the agent retrieve additional information.
- If the evidence remains inconclusive, return an explicit uncertainty response.
- If the proposed action could affect production, require the appropriate authorization or human approval.
This moves uncertainty handling from a conversational preference into application logic.
2. A practical AWS production scenario
Let's use a familiar data engineering problem: an AWS Glue ingestion pipeline fails during execution.
The pipeline uses Amazon S3 as its data source, AWS Glue for processing, and AWS Step Functions to orchestrate the workflow.
When a failure occurs, an AI agent receives the job identifier and investigates the available evidence.
The agent might inspect:
- CloudWatch logs and error messages.
- Glue job execution status and run metadata.
- Recent schema or configuration changes, if available.
- Relevant S3 and IAM error details.
- Previous incidents, if a trusted incident history exists.
The agent then produces a diagnosis and proposes a next step.
Consider three examples.
Case A: Evidence supports the diagnosis
The logs explicitly show an
AccessDenied error for the source S3 path. The agent can identify a likely permissions issue and recommend checking the relevant IAM policy and bucket policy.Case B: Evidence is incomplete
The logs end unexpectedly, and there is no reliable indication of the failure cause. The agent should report that the root cause cannot yet be determined and request additional diagnostic information.
Case C: The agent proposes a risky action
The agent recommends modifying an IAM policy or changing production compute settings. Even if the diagnosis appears plausible, the action should not execute merely because the model suggested it.
The application must enforce authorization and approval independently of the agent's answer.
3. The AWS architecture
We can build this workflow using existing AWS services rather than relying on the language model to enforce its own limits.
Architecture:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
AWS Glue Job Failure
|
v
Amazon CloudWatch Logs
|
v
AWS Step Functions
|
v
AWS Lambda - Evidence Collection
|
v
Amazon Bedrock - AI Diagnosis
|
v
Evidence Validation Layer
|
+----------------------+
| | |
v v v
Verified Insufficient Risky
Finding Evidence Action
| | |
v v v
Recommend Investigate Human Approval
Next Step or Abstain Before Execution
|
v
Record Decision and OutcomeThe responsibilities are deliberately separated.
Amazon Bedrock generates a diagnosis and recommendation.
AWS Lambda retrieves the required evidence and applies deterministic validation rules.
AWS Step Functions manages the investigation, retries, timeouts, and approval workflow.
Amazon CloudWatch provides operational logs and metrics.
Amazon S3 can store approved diagnostic snapshots and evaluation datasets, subject to the organization's data-retention and access policies.
The key design decision is that the model proposes a conclusion, but application code determines whether that conclusion is sufficiently supported to proceed.
4. Define what counts as sufficient evidence
Before writing code, we need to define our rules.
For a pipeline diagnosis, I would start with four checks:
- Evidence availability: Did the investigation retrieve the required logs and job metadata?
- Evidence relevance: Do the retrieved records actually relate to the failed job run?
- Evidence support: Does the proposed diagnosis match the observed error or other verified facts?
- Action authorization: Is the proposed next step permitted under the application's security policy?
These checks should not be collapsed into one model-generated confidence score.
For example, an agent might correctly identify a permissions issue while still lacking authorization to change a production IAM policy.
Correctness and permission to act are two different questions.
Likewise, having several log lines does not necessarily mean the evidence is sufficient. Five irrelevant log entries do not outweigh one missing critical record.
For the first implementation, we'll use a conservative policy: the agent can make a diagnosis only when the required evidence is available and a deterministic rule supports the conclusion.
5. Implement an evidence gate in Python
The following example illustrates the decision layer that runs after the model proposes a diagnosis.
It is intentionally independent of the model invocation. You can place it in an AWS Lambda function that receives structured output from Amazon Bedrock.
The example uses a narrow set of known error signatures. A real implementation should expand these rules carefully and include additional diagnostics rather than assuming every AWS failure matches a small list of patterns.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
from dataclasses import dataclass
from typing import Optional
class Diagnosis:
category: str
explanation: str
evidence_ids: list[str]
proposed_action: str
class Evidence:
evidence_id: str
text: str
job_run_id: str
verified: bool = True
def evaluate_diagnosis(
diagnosis: Diagnosis,
evidence: list[Evidence],
expected_job_run_id: str,
) -> dict:
# Only use verified evidence from the expected job run.
relevant = [
item for item in evidence
if item.verified
and item.job_run_id == expected_job_run_id
]
evidence_by_id = {
item.evidence_id: item for item in relevant
}
# Every cited evidence item must actually exist.
if not diagnosis.evidence_ids:
return {
"decision": "ABSTAIN",
"reason": "No supporting evidence was provided."
}
if any(
evidence_id not in evidence_by_id
for evidence_id in diagnosis.evidence_ids
):
return {
"decision": "ABSTAIN",
"reason": "Evidence reference is missing or unverified."
}
cited_text = " ".join(
evidence_by_id[evidence_id].text
for evidence_id in diagnosis.evidence_ids
).lower()
# Example rule for a narrowly defined failure signature.
if diagnosis.category == "S3_ACCESS_DENIED":
if "accessdenied" not in cited_text:
return {
"decision": "INVESTIGATE",
"reason": (
"The cited evidence does not contain "
"the expected access-denied signature."
)
}
# Recommendations are not permissions to execute.
return {
"decision": "REVIEW",
"reason": (
"Evidence passed basic validation. "
"Check action-specific policy before execution."
),
"category": diagnosis.category,
"proposed_action": diagnosis.proposed_action,
"evidence_ids": diagnosis.evidence_ids,
}This code illustrates a few important safeguards.
First, it rejects evidence references that do not exist in the verified evidence set.
Second, it prevents evidence from an unrelated job run from being used to justify a diagnosis.
Third, it checks a known error signature before accepting a particular classification.
Finally, even a diagnosis that passes the checks returns
REVIEW. It does not automatically execute the proposed action.That last point is intentional.
The code is a starting point, not a complete production diagnostic engine. It doesn't prove that the diagnosis is correct, validate every AWS error category, or establish that a recommendation is safe. Those require more comprehensive rules, tests, and operational controls.
6. Add an explicit abstention response
An agent should not return a vague message such as "Something may be wrong."
Its response should identify what it knows, what it cannot establish, and what evidence is needed next.
For example:
1
2
3
4
5
6
7
8
9
10
11
12
13
{
"status": "INSUFFICIENT_EVIDENCE",
"job_run_id": "jr_example_123",
"known_facts": [
"The Glue job failed.",
"The available logs do not establish the root cause."
],
"unknowns": [
"Whether the failure was caused by permissions, memory, or schema changes."
],
"next_step": "Retrieve the remaining logs and job run metadata.",
"production_action_allowed": false
}This is more useful than an invented diagnosis because it gives the next component in the workflow a clear decision.
Notice that the agent is not claiming the failure is mysterious or impossible to resolve. It is saying that the currently available evidence is insufficient.
That distinction matters. The system should remain open to gathering more information.
7. Let Step Functions handle the next decision
Once the evidence gate returns a decision, Step Functions can route the workflow accordingly.
For example:
SUPPORTED: return the verified finding and an allowed recommendation.INVESTIGATE: retrieve additional evidence within defined limits.ABSTAIN: stop the diagnostic process and report the unknowns.REVIEW: request approval before any consequential action.
A simplified workflow might look like this:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
Collect Evidence
|
v
Generate Diagnosis
|
v
Validate Evidence
|
+-- Supported -----> Return Recommendation
|
+-- Investigate ---> Gather More Evidence
|
+-- Abstain --------> Report Unknowns
|
+-- Review ---------> Human ApprovalIn a production implementation, the investigation loop must have a maximum number of attempts, a timeout, and a limit on the amount of data retrieved.
Otherwise, an agent could repeatedly search logs, call tools, and consume resources without reaching a useful conclusion.
The approval path should use a real authorization mechanism, such as a Step Functions callback task integrated with an approved internal approval system. A model-generated
"approved": true field must never be treated as actual authorization.8. What should happen before a production action?
Not every AI recommendation carries the same risk.
Suggesting a CloudWatch query is different from modifying an IAM policy or restarting a production ingestion workflow.
I would divide actions into three categories.
Low risk: Read-only diagnostic queries, inspecting metadata, or summarizing verified logs. These can often be automated using narrowly scoped IAM permissions.
Moderate risk: Retrying an idempotent job or changing a non-production configuration. These actions should have explicit preconditions, retry limits, and monitoring.
High risk: Modifying production permissions, deleting data, overwriting datasets, or changing critical infrastructure. These actions should require explicit authorization, suitable validation, and human approval where required by the operating policy.
Even an evidence-supported diagnosis should not bypass these controls.
For destructive operations, the safest default is to deny execution unless an explicit, independently enforced policy authorizes it.
9. How do we know whether this approach works?
Adding an abstention mechanism is not automatically an improvement. If an agent refuses every question, it will avoid many incorrect answers but won't be useful.
We need to measure both safety and usefulness.
I would build a test dataset containing known incidents such as:
- S3 access-denied errors.
- Spark executor or driver memory failures.
- Schema mismatches.
- Missing or incomplete logs.
- Conflicting evidence from different sources.
- Historical logs that belong to a different job run.
- Cases where the evidence supports no reliable diagnosis.
Then compare the agent with and without the evidence gate.
Track these metrics:
| Metric | What it tells us |
|---|---|
| Supported-diagnosis accuracy | How often accepted diagnoses are correct |
| Unsupported-claim rate | How often the system makes claims without adequate evidence |
| Abstention precision | How often abstentions are appropriate |
| Unnecessary abstention rate | How often the system refuses despite sufficient evidence |
| Unsafe-action block rate | How often the policy prevents disallowed actions |
| Investigation success rate | How often additional evidence resolves uncertainty |
| Cost per resolved incident | Whether the workflow is operationally worthwhile |
The most important safety metric is not how often the agent answers. It is how often the system allows an unsupported conclusion or unauthorized action to pass through.
Use a labeled evaluation dataset and test the system against previously unseen incidents. Measure results before claiming that the approach improves production reliability.
10. Production considerations that are easy to miss
Before deploying this pattern, there are a few details I would not skip.
Treat logs as untrusted input. A log message may contain user-controlled content or prompt-injection instructions. Retrieved text must not be allowed to override system policy or grant permissions.
Keep evidence traceable. Store identifiers, timestamps, job-run IDs, and references to the source records. Avoid logging credentials, sensitive payloads, or unnecessary personal information.
Validate model output. Use a defined schema and reject malformed responses. A valid JSON response is not necessarily a valid diagnosis.
Enforce permissions outside the model. IAM and application-level policy must control which tools and actions are available. A prompt saying "do not delete data" is not an access-control mechanism.
Make retries safe. Before retrying a failed ingestion, consider idempotency, duplicate writes, partial output, and whether the previous run changed downstream state.
Monitor the agent itself. Track invocation failures, latency, token usage, repeated investigations, abstention rates, and policy rejections. These are useful signals for improving the system over time.
Conclusion: Reliability sometimes means refusing to guess
When building AI applications, it's tempting to optimize for fast responses and successful tool calls. But in production, an answer that sounds correct can be more dangerous than an explicit admission of uncertainty.
An evidence-aware agent follows a different approach.
It gathers information, connects conclusions to verifiable evidence, recognizes when critical information is missing, and stops before taking actions it cannot justify.
AWS provides the building blocks for this design: Amazon Bedrock for model inference, Lambda for validation logic, Step Functions for orchestration, CloudWatch for observability, and IAM for access control.
The real engineering work is deciding which evidence is sufficient, which actions are permitted, and when the system must stop.
The goal isn't to build an AI agent that never says "I don't know." It's to build one that knows when saying it is the safest and most useful decision.
What would you add?
If you were deploying an AI agent to investigate production failures, would you trust evidence-based validation, a separate verification agent, deterministic rules, or a combination of all three?
I'd be interested in how other AWS builders are handling uncertainty before giving AI agents access to real systems.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article