AWS Builder Center
IncidentGraph: Making Distributed System Incident Investigation More Understandable with AI

IncidentGraph: Making Distributed System Incident Investigation More Understandable with AI

For the Bharat Builds Tour hackathon, I built IncidentGraph, an AI-assisted incident investigation platform designed to help engineers connect telemetry, service dependencies, and operational evidence when something goes wrong. This blog explores the problem, the architecture, how AWS powers the platform, and how AI assists the investigation process.

The problem

When a distributed application experiences a failure, figuring out what actually happened can be surprisingly difficult. An engineer might need to inspect logs, check metrics, investigate service dependencies, review deployment changes, and search through old incident reports.
The information exists, but it is scattered across different systems. An alert tells you that something may be wrong, but it doesn't necessarily tell you where to look next.
That was the problem I wanted to address with IncidentGraph.

What is IncidentGraph?

IncidentGraph is an AI-assisted incident investigation platform that brings together operational evidence to help engineers investigate failures.
Rather than acting as another monitoring dashboard, it organizes information such as:
  • Application metrics and logs.
  • Service dependencies and relationships.
  • Incident timelines and deployment information.
  • Historical incidents and runbooks.
  • Evidence, uncertainties, and suggested investigation steps.
The goal is to make it easier for an engineer to move from an alert to a structured investigation, without pretending that every correlation automatically proves a root cause.

The demonstration workload

To demonstrate IncidentGraph, I built a serverless, real-time chat application using Amazon API Gateway WebSocket APIs, AWS Lambda, and Amazon DynamoDB.
This application provides a real workload that I can exercise and deliberately break during demonstrations. I also developed IncidentGraph Lab, where I can manually inject controlled failures and observe how the system responds.
The chat application is a demonstration workload, not a requirement of the platform. IncidentGraph is designed to be project-agnostic: it can be adapted to existing applications through workload configuration and telemetry integration, with relatively small changes rather than rebuilding the investigation system from scratch.

How it works under the hood

The platform follows a pipeline that connects failure detection with evidence gathering and investigation.

1. Detecting an incident

Amazon CloudWatch collects application telemetry and evaluates configured alarms. When an alarm changes state, Amazon EventBridge routes the event to IncidentGraph's detection pipeline.
This allows IncidentGraph to respond to alarm state changes instead of relying on continuous polling to discover every incident.

2. Gathering evidence

Once an incident is detected, the backend gathers relevant operational information through AWS APIs. This can include CloudWatch metrics and logs, along with service dependencies, deployment details, historical incidents, and runbooks.
The collected information is organized into a structured evidence package that can be inspected by the engineer and used by the investigation process.

3. AI-assisted investigation

I integrated Gemini through its API to support investigation and retrieval-augmented generation (RAG).
The LLM works with evidence gathered by the backend. It can summarize findings, explain relevant correlations, identify what remains uncertain, and suggest useful next checks. Runbooks and historical incident knowledge can also provide additional context.
The AI provider is designed to be replaceable. Gemini is the provider used in my integration, but the architecture could also be adapted to Amazon Bedrock or another compatible LLM provider.
Importantly, the model is not given unrestricted access to AWS. The backend controls what information and investigation tools are available to it.

4. Investigation and recovery

Engineers can inspect the incident timeline, supporting evidence, and suggested investigation paths. They can then use the Lab to disable a controlled failure and observe the recovery process.
Incident history preserves previous events so engineers can review them and use relevant operational knowledge during future investigations.

Why IncidentGraph is project-agnostic

I wanted the core investigation platform to remain separate from the application used to demonstrate it.
The chat application supplies the workload and telemetry. The IncidentGraph backend handles detection, evidence collection, correlation, and investigation. Adapting the platform to another application would involve configuring its workload-specific details and connecting the relevant telemetry sources.
This separation makes the chat application a practical demonstration environment without limiting IncidentGraph to messaging systems.

What I learned

Building IncidentGraph brought together several areas I wanted to explore: serverless architecture, event-driven systems, observability, distributed applications, and LLM integration.
One important lesson is that an AI-generated explanation is only useful when it is grounded in reliable evidence. A system should be clear about what it observed, what it inferred, and what still needs to be checked. Correlation is useful for guiding an investigation, but it should not be presented as proof of causation.
I also learned how important it is to validate the entire pipeline. A dashboard displaying an incident is not enough by itself; detection, evidence gathering, investigation, and recovery need to behave consistently with the underlying application and telemetry.
IncidentGraph is my attempt to make distributed-system incident investigation more structured and understandable by bringing operational evidence and AI assistance into one workflow.
Built for the Bharat Builds Tour hackathon, the project combines AWS-based detection and telemetry with a configurable investigation system and a real serverless workload for demonstration.
The broader idea is simple: help engineers spend less time searching across disconnected systems and more time understanding the evidence in front of them.
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article