Building IncidentLens: An Evidence-Based AI Agent for Production Incident Investigation with Strands and OpenSearch
IncidentLens is an AI-powered incident investigation system built with Strands Agents and OpenSearch. It simulates production failures, collects real-time telemetry, and uses an AI agent to investigate and correlate evidence before identifying the likely root cause. In this article, I explain the architecture, AWS technologies used, key engineering challenges, and lessons learned while building an evidence-based AI investigation workflow.
Building IncidentLens: An Evidence-Based AI Agent for Production Incident Investigation with Strands and OpenSearch
When a production service goes down, the difficult part is often not noticing that something is wrong.
The difficult part is answering:
What actually happened, when did it start, what caused it, and what evidence proves it?
That question is what led me to build IncidentLens, an AI-powered incident investigation system designed to investigate production-like incidents using actual telemetry rather than simply generating an answer from an LLM.
For the AWS Build It hackathon, I combined Strands Agents and OpenSearch with a simulated production environment to create an AI investigation workflow:
Simulated Production ↓Logs / Metrics / Events ↓OpenSearch ↓Evidence Collection ↓Strands Investigation Agent ↓Evidence Correlation ↓Incident Analysis
The central idea is simple:
Don't ask AI to guess what went wrong. Make it investigate the evidence first.
The Problem
Imagine that a payment service suddenly starts failing.
A monitoring system might tell you:
payment-api: ERROR RATE HIGH
But that isn't enough.
An engineer still needs to determine:
- When did the problem begin?
- Was latency increasing before the errors?
- Which service was affected?
- What events occurred immediately before the failure?
- Was the service healthy earlier?
- What evidence connects these signals?
- What is the most likely root cause?
In a real production environment, this information can be spread across logs, metrics, events, traces, and monitoring systems.
The traditional workflow often looks something like:
Alert ↓Open logs ↓Search timestamps ↓Check metrics ↓Check recent events ↓Compare services ↓Build a timeline ↓Form a hypothesis ↓Verify the hypothesis
This is time-consuming and requires an engineer to manually correlate multiple signals.
I wanted to explore whether an AI agent could perform this initial investigation while remaining grounded in actual operational evidence.
The Idea Behind IncidentLens
The first design decision was also the most important one:
IncidentLens would not be a chatbot.
A conventional AI application could work like this:
User:"Why is payment-api failing?"
↓
LLM
↓
"payment-api is probably experiencinga database connection issue."
That answer might sound reasonable, but where did the conclusion come from?
The model could simply be guessing.
Instead, IncidentLens follows an investigation loop:
User asks a question ↓Agent investigates ↓Agent calls evidence tools ↓Tools query telemetry ↓Relevant evidence is returned ↓Evidence is correlated ↓Agent reasons over the evidence ↓Investigation is produced
This distinction became the foundation of the entire architecture.
Building a Simulated Production Environment
I didn't want to use a static dataset containing a pre-written incident.
That would make the demonstration predictable and would not really test an investigation system.
So I created a small simulated production environment.
The primary simulated service is:
payment-api
The service can move through states such as:
HEALTHY ↓DEGRADED ↓FAILING
While running, the simulator generates production-like telemetry.
For example:
logsmetricseventslatencyfailure signalsstate transitions
The important part is that the telemetry is generated by the simulated environment.
The investigation agent does not receive the answer beforehand.
Why OpenSearch?
The next problem was deciding where the operational evidence should live.
I needed a system that could:
- store telemetry,
- search telemetry,
- filter records,
- retrieve relevant events,
- support time-based investigation.
This is where OpenSearch became an important part of the architecture.
The simulated telemetry is indexed into:
incidentlens-telemetry
The high-level flow is:
Production Simulator ↓Telemetry ↓OpenSearch ↓Searchable Evidence
OpenSearch therefore isn't simply included because it is an AWS technology.
It serves a real architectural purpose: it is the evidence layer for the investigation agent.
Introducing Strands Agents
Once the telemetry was available, I needed an AI component that could actually investigate it.
This is where Strands Agents became the core of the AI layer.
Instead of giving the model a large block of telemetry and asking it to summarize everything, I designed the agent around tools.
Conceptually:
┌───────────────┐ │ Strands Agent │ └───────┬───────┘ │ chooses a tool │ ▼ ┌───────────────┐ │ Evidence Tool │ └───────┬───────┘ │ ▼ OpenSearch │ ▼ Evidence │ ▼ Strands Agent
This gives the agent access to operational information without requiring the model to already know what happened.
The agent can investigate the incident using the available evidence.
Separating Evidence Collection From Reasoning
One architectural decision that became particularly important was separating evidence collection from reasoning.
The evidence collector is responsible for retrieving information.
The agent is responsible for interpreting it.
That gives us:
Agent ↓Evidence Collection ↓OpenSearch ↓Telemetry
rather than:
Agent ↓Raw database access ↓Everything
This separation makes the system easier to reason about and gives us more control over what information reaches the model.
From Raw Records to an Incident Timeline
Raw telemetry alone isn't necessarily useful to an AI agent.
Consider these records:
10:41:02 latency = 180ms10:41:05 service = degraded10:41:07 error = payment request failed10:41:09 latency = 620ms10:41:11 service = failing
Individually, these are just records.
Together, they tell a story:
Latency increased ↓Service degraded ↓Payment failures appeared ↓Latency increased further ↓Service entered failing state
IncidentLens therefore includes an evidence correlation layer.
Its purpose is to transform individual signals into a more useful incident-oriented representation.
This gives the agent temporal context when reasoning about the incident.
A Real Problem I Encountered: Historical Telemetry
One of the most interesting problems appeared after the basic system was working.
I could run the simulator.
Then I could run it again.
Then again.
All of those runs were writing telemetry into the same OpenSearch index.
That created a subtle problem.
Suppose the system had:
Run A └── payment-api failure
Run B └── payment-api failure
If the collector simply searched:
payment-api
it could retrieve evidence from both runs.
The AI might then produce a perfectly reasonable explanation based on evidence that actually belonged to an older incident.
This was a much more serious problem than a simple bug.
An incident investigation system must investigate the correct incident.
Introducing run_id
The solution was to give each simulator execution its own identifier.
For example:
run_id = 20260920-001
Telemetry then becomes associated with a particular execution:
Run 001 ├── log ├── metric ├── event └── failure
Run 002 ├── log ├── metric ├── event └── failure
The collector can then restrict its queries to the appropriate
run_id .This gives us:
Current Incident ↓Current run_id ↓Relevant telemetry ↓Correct evidence ↓Correct investigation
This was one of the most important lessons from building IncidentLens:
Reliable AI investigation depends on reliable evidence selection.
A sophisticated model cannot compensate for incorrect evidence.
Local AI Development With Ollama
For development, I also wanted to experiment with the system locally.
I used Ollama to run a local model and initially experimented with:
Qwen3 8B
The model worked, but it was slower and placed noticeably more load on the development machine.
Since this was a hackathon project where fast iteration was important, I tested:
Qwen3 4B
The smaller model provided a better balance for my local development environment.
This was an important practical lesson:
The largest model is not always the best model for the development workflow.
For an agentic application, the model is only one part of the latency.
Tool calls, database queries, evidence processing, and application logic all contribute to the overall experience.
The Complete Architecture
The final architecture can be simplified into five major layers.
┌───────────────────────────────────────┐│ Simulated Production ││ ││ payment-api │└───────────────────┬───────────────────┘ │ │ telemetry ▼┌───────────────────────────────────────┐│ OpenSearch ││ ││ incidentlens-telemetry │└───────────────────┬───────────────────┘ │ │ queries ▼┌───────────────────────────────────────┐│ Evidence Collection ││ ││ Collector + Correlator │└───────────────────┬───────────────────┘ │ │ evidence ▼┌───────────────────────────────────────┐│ Strands Agent ││ ││ Investigate → Retrieve → ││ Correlate → Reason │└───────────────────┬───────────────────┘ │ ▼┌───────────────────────────────────────┐│ Streamlit UI ││ ││ Evidence + Investigation │└───────────────────────────────────────┘
The complete flow is:
Simulated service ↓Dynamic telemetry ↓OpenSearch ↓Evidence collector ↓Evidence correlation ↓Strands investigation agent ↓Incident analysis ↓Streamlit
The User Experience
The user doesn't need to manually search through the telemetry.
They can ask IncidentLens to investigate an incident.
For example:
Investigate why payment-api is failing.
The system then starts the investigation.
The agent retrieves relevant evidence from OpenSearch.
The UI can show the evidence collected during the investigation and the resulting analysis.
The result can include:
- affected service,
- likely cause,
- supporting evidence,
- incident timeline,
- relevant operational signals.
The important distinction is that the evidence is shown alongside the conclusion.
Why This Is More Than an LLM Wrapper
This was probably the most important design goal of the project.
A typical LLM wrapper looks like:
Input ↓LLM ↓Output
IncidentLens looks like:
Production ↓Telemetry ↓Search ↓Evidence ↓Agent tools ↓Investigation ↓Reasoning ↓Explanation
The AI is one component in a larger system.
The system provides the AI with an environment to investigate.
That is what makes the project interesting from a DevOps perspective.
The Technology Stack
The core stack is:
Technology | Role |
|---|---|
Strands Agents | AI investigation agent and tool-based workflow |
OpenSearch | Telemetry and evidence store |
Ollama | Local model runtime during development |
Qwen3 4B | Local development model |
Streamlit | Investigation UI |
Python | Application and agent implementation |
Docker / Docker Compose | Local OpenSearch environment |
WSL2 Ubuntu | Development environment |
Git / GitHub | Version control and project repository |
For the AWS Build It track, the central AWS open-source technologies were Strands Agents and OpenSearch.
Challenges Beyond Writing Code
The most difficult parts of this project were not necessarily writing individual functions.
The bigger challenges were getting the entire system to work together.
The development environment involved:
Windows ↓WSL2 ↓Ubuntu ↓Docker ↓OpenSearch ↓Python ↓Strands ↓Local model ↓Streamlit
A problem at any layer could make the application appear broken.
For example, when OpenSearch was stopped, the application could no longer query:
localhost:9200
That sounds obvious in hindsight, but it reinforced an important debugging principle:
Always distinguish an application problem from a dependency problem.
Another Challenge: Learning the Stack While Building
This was also my first project combining several of these technologies into one system.
I had to learn and integrate:
- Python
- Docker
- OpenSearch
- Strands Agents
- local LLMs
- tool calling
- Streamlit
- Git/GitHub
- telemetry concepts
- agent architecture
Trying to build everything simultaneously would have made debugging extremely difficult.
So the system was built incrementally.
The progression was approximately:
Python ↓Ollama + local model ↓Strands ↓OpenSearch ↓Telemetry simulator ↓Evidence collector ↓Evidence correlation ↓Agent investigation ↓Streamlit
Each layer was tested before becoming part of the next layer.
What I Learned About Agentic Systems
The biggest lesson was that building an agent is not just about selecting a model.
A useful agent needs:
Model+Tools+Data+Context+Reliable tool results+Good system design
In IncidentLens, the model alone doesn't know why
payment-api failed.The environment contains the information.
OpenSearch contains the evidence.
The tools retrieve the evidence.
The correlation layer gives the evidence context.
The agent reasons over it.
This makes the entire system more important than the model itself.
What I Learned About Observability
Building the simulated production environment also changed how I thought about observability.
Previously, it was easy to think of logs, metrics, and events as separate things.
During this project, I started thinking about them as pieces of a single incident story.
For example:
Metric:latency increased
+
Event:service entered degraded state
+
Log:payment requests started failing
+
State:service entered failing state
Together, these signals provide much more information than any individual record.
This is exactly the kind of context an AI investigation agent can use.
Why OpenSearch and Strands Work Well Together
The combination is particularly interesting because the two technologies solve different parts of the problem.
OpenSearch provides searchable operational knowledge.
Strands Agents provides the agentic reasoning and tool-use layer.
Together:
OpenSearch"What happened?"
+
Strands"What should I investigate next,and what does the evidence mean?"
↓
Incident investigation
This creates a natural architecture for AI-assisted operations.
Where IncidentLens Could Go Next
The current project is a hackathon-scale prototype, but the architecture can be extended.
Some possible next steps are:
Multiple services
Instead of only:
payment-api
the environment could contain:
frontend ↓payment-api ↓database ↓external-payment-service
The agent could then investigate service dependencies.
More telemetry
The system could incorporate:
- traces,
- container events,
- deployment events,
- infrastructure metrics,
- Kubernetes events.
Automated remediation
The agent could eventually move from:
Investigate
to:
Investigate ↓Recommend remediation ↓Request human approval ↓Execute remediation
The human approval step would be important before allowing an AI system to modify production infrastructure.
Better incident visualization
A richer UI could show:
TimelineService dependency graphTelemetry anomaliesEvidenceRoot causeRecommended actions
Final Thoughts
IncidentLens started with a simple question:
Can an AI agent investigate a production incident using actual operational evidence instead of guessing?
The project became an exploration of how to combine:
- simulated production,
- telemetry,
- OpenSearch,
- evidence collection,
- evidence correlation,
- Strands Agents,
- local AI,
- and a developer-facing interface
into one workflow.
The most important lesson wasn't about any individual technology.
It was this:
AI reliability starts before the model generates an answer. It starts with giving the model the right evidence.
With IncidentLens, the goal is therefore not to make AI sound confident.
The goal is to make AI investigate first and explain second.
Project Architecture at a Glance
INCIDENTLENS
┌──────────────────────┐ │ Simulated Production │ │ │ │ payment-api │ └──────────┬───────────┘ │ ▼ ┌──────────────────────┐ │ Telemetry │ │ │ │ Logs / Metrics / │ │ Events / Failures │ └──────────┬───────────┘ │ ▼ ┌──────────────────────┐ │ OpenSearch │ │ │ │ incidentlens- │ │ telemetry │ └──────────┬───────────┘ │ ▼ ┌──────────────────────┐ │ Evidence Collection │ │ + │ │ Evidence Correlation │ └──────────┬───────────┘ │ ▼ ┌──────────────────────┐ │ Strands Agent │ │ │ │ Investigate │ │ Retrieve │ │ Correlate │ │ Reason │ └──────────┬───────────┘ │ ▼ ┌──────────────────────┐ │ Streamlit UI │ │ │ │ Evidence + Analysis │ └──────────────────────┘
IncidentLens: investigate the evidence first.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article