AWS Builder Center

From Observability to Understanding: Building TattvaAI for Evidence-Grounded Incident Investigation

Production incidents rarely come from a single signal. TattvaAI brings logs, metrics, traces, evidence correlation, reasoning, human review, and reporting into one AWS-powered investigation workflow.

Introduction

When a production system fails, the hardest part is often not finding telemetry. Modern applications can generate enormous amounts of logs, metrics, and distributed traces.
The difficult part is understanding what those signals mean together.
An engineer investigating an incident may move between dashboards, log viewers, tracing systems, databases, deployment information, and application code. Each tool provides a piece of information, but the investigation itself still depends heavily on manually connecting those pieces.
I wanted to explore a different approach.
Instead of treating observability as simply a collection of dashboards, I built TattvaAI, an AI-assisted incident investigation platform designed to help engineers move from raw observability evidence toward a structured investigation.
The central idea is simple:
Observability tells us what happened. Investigation helps us understand what the evidence means.
TattvaAI is designed around that distinction.



The Problem

A production incident can produce several signals at the same time.
For example:
  • request latency may increase,
  • a service may generate errors,
  • traces may show an unusual dependency,
  • logs may contain related failures,
  • and infrastructure metrics may change around the same time.
Looking at each signal independently can make the investigation fragmented.
The engineer has to determine:
  1. Which signals are actually related?
  1. Which evidence is relevant to the incident?
  1. What happened first?
  1. Which symptoms are consequences rather than causes?
  1. What hypothesis is supported by the evidence?
  1. What should be investigated next?
This led to the design goal behind TattvaAI:
Build an investigation workflow that organizes evidence before producing conclusions.



Designing TattvaAI Around Evidence

One of the most important principles I wanted to apply was disciplined action based on understanding rather than assumptions.
That translated directly into the architecture.
TattvaAI does not treat an initial hypothesis as a fact.
Instead, the investigation progresses through multiple stages:
Observability Evidence
↓
Evidence Collection
↓
Evidence Normalization
↓
Evidence Correlation
↓
Reasoning & Hypothesis Generation
↓
Human Review
↓
Report Generation
This separation is important.
The system first gathers evidence, then organizes it, then reasons about it.
That makes the investigation process easier to inspect and reduces the temptation to jump directly from a symptom to a conclusion.



Architecture

TattvaAI is deployed using AWS services across the application, compute, data, security, and observability layers.
The frontend is a React application hosted using AWS Amplify.
Requests are routed through Amazon API Gateway to the TattvaAI backend running on AWS Lambda.
The backend coordinates the investigation workflow and interacts with the application's persistence and reporting layers.
The main architecture is:
Engineer
↓
React Frontend
AWS Amplify
↓
Amazon API Gateway
↓
AWS Lambda
↓
TattvaAI Investigation Engine
├── Evidence Collection
├── Evidence Normalization
├── Evidence Correlation
├── Reasoning
├── Human Review
└── Report Generation
↓
DynamoDB / S3
AWS observability services provide the evidence used by the investigation workflow.
Amazon CloudWatch provides logs and metrics, while AWS X-Ray provides distributed traces and service dependency information.
Amazon DynamoDB stores investigation state and findings.
Amazon S3 stores generated investigation reports.
AWS Secrets Manager is used to protect application secrets and configuration.



1. Evidence Collection

The first stage is evidence collection.
Rather than immediately asking a reasoning system to explain an incident, TattvaAI first obtains the available telemetry evidence.
In the deployed AWS environment, this includes evidence from CloudWatch and X-Ray.
The goal is not simply to collect as much information as possible.
The goal is to collect relevant evidence that can contribute to the investigation.
This distinction becomes important as systems grow larger. More telemetry does not automatically mean better understanding.



2. Evidence Normalization

Different observability systems represent information differently.
Logs, metrics, and traces have different structures and different kinds of context.
TattvaAI therefore introduces a normalization stage.
The purpose is to convert evidence from different sources into a consistent internal representation that the investigation pipeline can work with.
This provider-based approach also makes the architecture less dependent on a single telemetry backend.
The investigation engine can work with different telemetry providers through defined interfaces rather than embedding provider-specific assumptions throughout the application.



3. Evidence Correlation

After normalization, TattvaAI correlates related signals.
This is one of the most important parts of the investigation workflow.
A single log entry might not mean much by itself.
A latency change might also be ambiguous.
A trace might show a dependency but not explain why that dependency became problematic.
When these signals are examined together, they can provide stronger investigative context.
The correlation stage therefore attempts to move the investigation from:
individual signals
to:
related evidence.
This creates a foundation for the next stage.



4. Reasoning and Hypothesis Generation

Only after the evidence has been collected, normalized, and correlated does the reasoning stage produce an investigation hypothesis.
The output is not intended to be treated as unquestionable truth.
Instead, TattvaAI produces:
  • a root-cause hypothesis,
  • supporting evidence,
  • a confidence level,
  • and recommended next steps.
This is an important design decision.
An AI-assisted investigation system should help an engineer reason about an incident, not replace the engineer's responsibility to evaluate the evidence.
The current deployed environment uses the available fallback reasoning path, while the architecture includes a provider-based design for integrating reasoning providers.
This separation allowed the rest of the investigation workflow to remain functional even when a particular external reasoning capability was unavailable.



Human Review

Human review is a core part of TattvaAI rather than an afterthought.
Once the system produces its findings, the engineer can review the investigation before accepting it.
This creates an important boundary:
The system assists with investigation; the engineer remains responsible for the final decision.
This approach is especially important for production systems because an incorrect hypothesis can lead to an incorrect operational action.
A useful investigation system therefore needs not only intelligence, but also transparency and accountability.



Investigation Reports

After review, TattvaAI generates a structured investigation report.
The report brings together the investigation context, evidence, findings, hypothesis, confidence, recommendations, and review outcome.
This provides a persistent record of how the investigation progressed.
Instead of ending with a temporary conversation or a collection of screenshots, the investigation becomes a structured artifact that can be revisited later.



Why AWS?

AWS became more than a deployment target for this project.
It became part of the investigation architecture itself.
The major components are:
AWS Service
Role in TattvaAI
AWS Amplify
React frontend hosting
Amazon API Gateway
API routing
AWS Lambda
Serverless backend
Amazon DynamoDB
Investigation state and findings
Amazon S3
Investigation reports
Amazon CloudWatch
Logs and metrics
AWS X-Ray
Distributed tracing
AWS Secrets Manager
Secure configuration and secrets
This architecture also gave me practical experience with the operational side of software development.
Building the application locally was only one part of the problem.
Deploying it required understanding API routing, Lambda execution, IAM permissions, authentication, CORS, persistence, observability, secrets, and debugging across multiple AWS services.



What I Learned

One of the biggest lessons from building TattvaAI was that building software is not only about producing a working feature. It is also about understanding the consequences of how that feature is deployed and operated.
I learned how to move a full-stack application from a local development environment into a serverless AWS architecture.
I gained hands-on experience with Amplify, API Gateway, Lambda, DynamoDB, S3, CloudWatch, X-Ray, and Secrets Manager.
I also learned that observability becomes much more valuable when it is treated as evidence rather than simply displayed as dashboards.
Another important lesson was the value of separating evidence from interpretation.
A telemetry signal is an observation.
A hypothesis is an interpretation of those observations.
Keeping those concepts separate makes the investigation easier to inspect and review.
I also learned the importance of designing for failure.
During development, the intended Amazon Bedrock integration could not be activated because of an AWS account authorization restriction. Instead of making the entire application dependent on that unavailable capability, the architecture allowed the investigation workflow to continue using its fallback reasoning path.
That experience reinforced an important engineering lesson:
A production system should degrade gracefully when one dependency is unavailable.



Building With Human Responsibility in Mind

There is another lesson that influenced the architecture.
Automation should not remove responsibility.
An investigation system can collect evidence faster, identify relationships between signals, and suggest possible explanations.
But the final operational decision belongs to the engineer.
That is why human review is explicitly represented in the TattvaAI workflow.
The system's role is to make the engineer's investigation more structured and informed.
The engineer's role is to evaluate the evidence, question the hypothesis when necessary, and decide what action should actually be taken.
This principle also influenced how I approached the investigation pipeline itself.
The system should not rush toward an answer.
It should first establish what is known, identify what is connected, distinguish evidence from interpretation, and then present a hypothesis that can be reviewed.



Challenges

The project involved several challenges beyond application development.
Deploying the backend required coordinating API Gateway, Lambda, IAM, authentication, persistence, secrets, and observability.
Another challenge was making the telemetry provider flexible enough that the core investigation workflow would not depend on a single observability platform.
AWS account-level restrictions also affected the availability of the intended Bedrock reasoning integration.
Rather than hiding these limitations, I treated them as part of the engineering process.
The result was a more resilient architecture where telemetry and reasoning capabilities can be treated as replaceable providers.



What Comes Next

There are several directions in which TattvaAI can be extended.
Future versions could integrate additional telemetry providers, improve evidence correlation, introduce more sophisticated reasoning providers, and expand investigation memory across incidents.
Another direction is improving the relationship between historical incidents and current investigations.
If similar evidence appeared in previous incidents, the system could surface those investigations as additional context for the engineer.
The long-term goal is not simply to generate more automated answers.
It is to build a system that helps engineers develop a clearer understanding of complex production behavior.



Conclusion

TattvaAI started with a simple observation: production observability systems can provide enormous amounts of information, but information alone does not complete an investigation.
The engineering challenge is connecting those signals into a coherent investigation process.
TattvaAI approaches this through a structured workflow:
Collect
↓
Normalize
↓
Correlate
↓
Reason
↓
Review
↓
Report
AWS provides the infrastructure and observability foundation, while TattvaAI adds the investigation layer on top.
The result is an AI-assisted workflow designed to help engineers move from scattered telemetry evidence toward a structured, reviewable investigation—while keeping human judgment at the center of the final decision.
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article