Sentinel: Building an AI SRE Agent That Doesn't Just Diagnose Incidents—It Demands Proof
Production incidents demand more than error messages and plausible AI explanations. They demand evidence, actionable remediation, and confidence that recovery has actually occurred. Sentinel is an AI-powered Site Reliability Engineering (SRE) project designed to help developers investigate application failures and move toward reliable incident resolution. In this article, I share the engineering behind Sentinel, how I approached realistic failure scenarios.
1. The Problem: From Error Messages to Incident Resolution
A production service rarely announces exactly what went wrong. An API begins returning HTTP 503 errors, database connections become exhausted, and developers must identify the underlying problem while minimizing downtime.
Traditional monitoring tools can identify symptoms. AI assistants can explain error messages. But turning that information into a controlled remediation workflow is a different challenge.
I built Sentinel, an AI-powered SRE agent, to explore how AI can move beyond conversational assistance and become a practical part of the incident-response process.
The goal is to help developers understand incidents, investigate root causes, identify appropriate corrective actions, and establish whether an application has actually recovered.
Sentinel is intended for developers, platform engineers, and small engineering teams who need a more structured way to investigate application failures.
2. What I Built
Sentinel is designed around an incident lifecycle rather than a simple question-and-answer interaction.
Its workflow focuses on four responsibilities:
- Incident intake: Receive incident reports from an application experiencing a failure.
- Root-cause investigation: Analyze available logs and operational evidence to identify likely causes.
- Remediation guidance: Recommend corrective actions grounded in the observed failure.
- Recovery verification: Check the application's behavior before treating the incident as resolved.
The distinction between these responsibilities is important. An agent can identify a likely cause without having enough evidence to prove it. Likewise, a proposed fix can be technically correct without restoring the application.
Sentinel is built to make these stages explicit.
3. Engineering a Realistic Failure Scenario
To test the incident-response workflow against a meaningful engineering problem, I worked with a dedicated Demo 1 Order API that reproduces database connection-pool exhaustion.
In this scenario, database connections become unavailable, preventing the application from serving requests reliably. The order endpoint starts returning HTTP 503 responses.
The application can report incidents to Sentinel through its configured
SENTINEL_URL. This integration connects the failing service to the incident-management workflow.During testing, the connection pool showed ten connections in use and zero available, while the order endpoint returned HTTP 503. These observations supplied concrete diagnostic evidence rather than relying entirely on an AI-generated interpretation.
This exercise also highlighted an important operational distinction: successfully delivering an incident report proves that the reporting integration works. It does not prove that the underlying service has recovered.
That distinction shaped how I think about the entire system.
4. How I Built Sentinel
I approached Sentinel as an engineering system with distinct responsibilities, rather than simply connecting a language model to a chat interface.
The project includes backend configuration, incident-reporting integration, and a demonstration application capable of reproducing an operational failure.
I also investigated and corrected a logging issue involving
structlog's BoundLogger.warning() method, which received conflicting values for the event argument. The correction involved six files, and 27 tests passed following that fix.This reinforced a practical lesson: reliable AI systems depend on the quality of their surrounding software, including logging, error handling, configuration, integration boundaries, and tests.
AWS architecture and services
For the AWS implementation, the relevant architecture separates the user interface, API orchestration, AI reasoning, and incident verification.
The intended service mapping is:
- Amazon Bedrock: Model inference for interpreting incident evidence and generating diagnostic guidance.
- AWS Lambda: Serverless backend execution and workflow orchestration.
- Amazon API Gateway: API access for the frontend or incident-reporting integration.
- Amazon CloudWatch: Application logs and operational observability, where integrated.
These services provide a practical foundation for an AWS-hosted implementation. The exact services used in the published version should match the deployed application; services that have not been integrated should not be represented as completed components.
The key architectural principle is separation of responsibilities: the application reports what it observes, the AI reasons over the available evidence, and verification evaluates the operational outcome.
5. The Delightful Detail: Never Confuse Confidence With Recovery
The most important design principle behind Sentinel is simple:
An AI agent should never declare an incident resolved merely because it has generated a convincing explanation.
Consider a database connection-pool exhaustion incident. Sentinel identifies the likely cause and recommends a correction. The recommendation sounds reasonable, but the application continues returning HTTP 503.
A trustworthy agent must recognize that the incident remains unresolved.
The experience I want Sentinel to deliver separates the workflow into three understandable states:
Diagnosed: Evidence supports a likely explanation.
Fix proposed: A corrective action has been recommended, but recovery has not been confirmed.
Verified: An actual test confirms the expected application behavior.
These states help developers understand what the system knows and what remains uncertain. They also prevent a common failure mode in AI-assisted engineering: presenting a plausible answer as if it were a verified operational result.
The feature's success must be evaluated through repeatable tests. For example, a connection-pool exhaustion test should reproduce the failure, allow the incident to be investigated, and then check whether the endpoint returns the expected healthy response after remediation.
A recovery state should only be awarded when the corresponding verification test passes. Where the full workflow has not yet been tested, the result should remain unverified.
6. Testing and Evidence
I used a realistic failure scenario to examine the incident-reporting integration and the application's behavior under database connection exhaustion.
The available test observations were:
- Ten database connections in use.
- Zero database connections available.
- HTTP 503 responses from the order endpoint.
- Twenty-seven tests passing after a six-file logging-related correction.
These results demonstrate concrete debugging and testing activity. They do not, by themselves, establish that the complete autonomous remediation and recovery-verification workflow has passed end-to-end testing.
That is why the next validation milestone is to reproduce the incident, confirm that Sentinel receives it, evaluate the diagnosis, apply or guide the remediation, and run a real recovery test against the application.
Proof that Sentinel works
Live demo: [Insert your working demo URL]
Source code: [Insert your GitHub repository URL]
Screenshot or short video: [Insert an image or video showing an incident being investigated]
The demonstration should show the original failure, Sentinel's response, the proposed corrective action, and the verification result. If any stage is not yet implemented, the evidence should clearly identify the stages that are working.
7. What I Learned
Building Sentinel reinforced my view that the hardest part of an AI SRE agent is not producing a plausible root-cause analysis. It is connecting that reasoning to observable system behavior.
Incident ingestion, diagnosis, remediation, and recovery verification are different engineering problems, and each needs appropriate tests.
Realistic failure scenarios are particularly valuable because they expose integration problems, logging defects, and incorrect assumptions that a successful demonstration can easily hide.
The broader lesson is that AI agents become more useful when their reasoning is supported by operational evidence, explicit state management, and measurable outcomes.
8. What's Next?
The next milestone is completing and validating the full incident-response loop: reproduce a failure, deliver the incident to Sentinel, establish a diagnosis, execute or guide an appropriate remediation, and verify that the Order API has returned to a healthy state.
From there, I want to expand the system toward more incident types, stronger observability integrations, and safer remediation workflows with appropriate human oversight.
My goal is not simply to build an agent that sounds intelligent. It is to build one that behaves predictably, communicates uncertainty, and earns trust through measurable results.
Sentinel is built around one principle: don't just tell me the problem is solved. Show me the evidence.
#agents
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article