From Logs to Stories: Building an AI Incident Investigator on AWS
What if your monitoring system could investigate an outage instead of simply telling you that one happened?
Introduction
When a production system goes down, the hardest part is often not detecting the problem — it's understanding what actually happened.
Modern applications generate logs, metrics, traces, deployment events, and infrastructure signals. Engineers often have to jump between multiple dashboards and manually connect these signals before they can identify a possible cause.
What if an AI agent could perform the first stage of this investigation automatically?
This article presents a simple architecture for an AI Incident Investigator on AWS.
The Architecture
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
Applications
│
▼
Logs / Metrics / Traces
│
▼
Amazon CloudWatch
│
▼
Amazon EventBridge
│
▼
AI Incident Investigator
│
├── Query Logs
├── Analyze Metrics
├── Inspect Traces
└── Check Deployments
│
▼
Incident Timeline
│
▼
Possible Causes + Evidence
│
▼
Human Engineer
The key idea is simple:
CloudWatch detects. EventBridge triggers. The AI investigates. The engineer decides.
How It Works
Suppose an application's error rate suddenly increases from 1% to 15%.
CloudWatch detects the anomaly and an EventBridge event triggers the AI investigator.
The agent then uses controlled tools to investigate:
1
2
3
4
get_metric_data()
get_log_events()
get_trace_data()
get_recent_deployments()
For example, it might discover:
1
2
3
4
02:06 — Payment service deployed
02:11 — Latency begins increasing
02:14 — Connection errors increase
02:17 — Checkout error rate crosses threshold
The agent can correlate these events and produce an investigation such as:
"The payment service experienced a significant latency increase shortly after deployment. Connection timeout errors also increased, while database CPU remained within its normal range. The recent deployment is therefore a potential contributor and should be investigated."
The important distinction is that the AI reports evidence and hypotheses, rather than presenting an unverified conclusion as fact.
Why Use an Agent?
A normal AI chatbot waits for a question:
1
Engineer → Question → AI → Answer
An incident investigator can work differently:
1
2
3
4
5
6
7
8
9
10
11
12
13
Incident
↓
Collect evidence
↓
Choose relevant tools
↓
Analyze results
↓
Form hypothesis
↓
Verify with more evidence
↓
Generate report
This makes the AI useful as an investigation assistant rather than simply a conversational interface.
Amazon Bedrock AgentCore can provide the infrastructure for running and observing production AI agents, while AWS services can be exposed to the agent through controlled tools.
Keeping Humans in the Loop
The AI should not automatically receive unrestricted production access.
It can:
- investigate incidents,
- summarize evidence,
- build timelines,
- identify possible causes,
- recommend next steps.
High-impact actions such as rolling back deployments or modifying production infrastructure should remain behind appropriate policies and human approval.
This creates a safer model:
1
2
3
4
5
6
7
AI Investigation
↓
Recommendation
↓
Human Approval
↓
Production Action
Conclusion
The future of cloud observability may not be about producing more dashboards.
It may be about making existing telemetry easier to understand.
By combining Amazon CloudWatch, Amazon EventBridge, Amazon Bedrock AgentCore, and controlled investigation tools, we can build systems that don't just say:
"Something is wrong."
but instead provide:
"Here is what changed, here is the evidence, here are the possible explanations, and here is what you should investigate next."
The goal isn't to replace the engineer.
It is to give the engineer a much faster starting point when production goes wrong.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article