
Building AutoOps: An AI-Powered Incident Intelligence Platform with AWS DevOps Agent
How I Built a Fully Automated On-Call System That Investigates, Diagnoses, and Recommends Fixes
Introduction
On-call incident response is one of the most mentally taxing parts of running a cloud-hosted service. When an alarm fires at 2 AM, an engineer has to wake up, log into multiple AWS consoles, correlate logs and metrics across services, piece together a root cause, and then decide on a fix — all under pressure. I wanted to build a system that handles that entire investigation automatically, so the engineer only needs to approve the fix.
In this article, I'll walk through how I built AutoOps — an end-to-end incident intelligence platform using the AWS DevOps Agent, Lambda, EventBridge, CloudWatch, DynamoDB, API Gateway, AWS Chatbot, and Terraform. The system automatically detects infrastructure incidents, runs a multi-phase AI investigation, posts a root cause analysis to Slack, and presents remediation options for human approval before taking any action.
The Problem I Set Out to Solve
Traditional on-call workflows look like this:
- CloudWatch alarm fires
- Engineer gets paged
- Engineer opens CloudWatch, Lambda console, DynamoDB metrics, X-Ray — one by one
- Engineer correlates timestamps across services to find the root cause
- Engineer decides on a fix and applies it manually
This process can take 20–40 minutes for a straightforward throttling issue. The goal of AutoOps was to reduce that to zero manual investigation time — the system does the analysis automatically and presents the engineer with a clear summary and a one-click approval button.
Architecture Overview
The system has two independent planes:
Plane 1 — The API (the thing that can break) A REST API backed by Lambda and DynamoDB, intentionally provisioned with 1 Write Capacity Unit to make throttling easy to trigger. This creates a realistic incident scenario.
Plane 2 — The Intelligence Pipeline (the thing that fixes it) An event-driven chain starting from CloudWatch alarms, routing through EventBridge, triggering a webhook Lambda that notifies the AWS DevOps Agent, which then runs a full multi-phase investigation and posts findings to Slack.
CLIENT REQUESTS │ ▼API Gateway REST API (autoops-api-dev) GET /health GET /items POST /items GET /items/{id} DELETE /items/{id} POST /simulate-error │ ▼Lambda Function (Python 3.12 · 256 MB · X-Ray Active Tracing) │ ▼DynamoDB Table (1 WCU — intentional for demo throttling)
━━━━━━━━━━━━━━ MONITORING & ALERTING PLANE ━━━━━━━━━━━━━━
CloudWatch Alarms (6 alarms) │ ├──► SNS Topic │ ├──► Email notification │ └──► AWS Chatbot → Slack #aws-incidents │ └──► EventBridge Rule (ALARM state only) │ ▼ Lambda: Webhook Forwarder ├── Retrieve HMAC credentials (Secrets Manager) ├── Sign payload (HMAC SHA-256) └── POST to DevOps Agent Webhook │ ▼ DevOps Agent Space (autoops-incident-intelligence) ├── Topology Discovery ├── CloudWatch Log Analysis ├── X-Ray Trace Analysis ├── DynamoDB Metrics Analysis └── Root Cause Report → Slack #aws-incidents (with HITL approval buttons)
Infrastructure — Built Entirely with Terraform
Every resource in this system is defined as code. I structured the Terraform files by service concern so each component can be reasoned about independently.
terraform/├── apigateway.tf # REST API + CloudWatch access logging├── cloudwatch.tf # 6 alarms + dashboard├── devops_agent.tf # SSM params, EventBridge rule, webhook Lambda├── dynamodb.tf # Table with intentionally low WCU├── iam.tf # All roles and policies├── lambda.tf # Function + X-Ray tracing├── outputs.tf # Endpoint URLs and resource names├── providers.tf # AWS provider├── s3.tf # Lambda artifact bucket├── sns.tf # Topic + email + Chatbot subscription└── variables.tf # Input definitions
Deploying the Infrastructure
cd terraformterraform initterraform planterraform apply
After apply, Terraform outputs the API endpoint URL, CloudWatch dashboard URL, DynamoDB table name, Lambda function name, S3 artifact bucket, and SNS topic ARN. It also prints next-step instructions for post-deployment configuration.
The API Application
The Lambda function handles five API routes, all backed by DynamoDB:
# Simplified handler structuredef lambda_handler(event, context): method = event['httpMethod'] path = event['path']
if path == '/health': return health_check() elif path == '/items' and method == 'GET': return list_items() elif path == '/items' and method == 'POST': return create_item(event) elif path.startswith('/items/') and method == 'GET': return get_item(item_id) elif path.startswith('/items/') and method == 'DELETE': return delete_item(item_id) elif path == '/simulate-error': return simulate_error()
X-Ray tracing is enabled at the function level with
TracingConfig: Active , which means every DynamoDB call, every exception, and every subsegment is captured and available for the DevOps Agent to query during investigation.CloudWatch Alarms — 6 Alarms Covering Every Layer
I created alarms at every layer of the stack so no class of failure goes undetected:
Alarm | Condition | What It Catches |
|---|---|---|
autoops-lambda-errors-dev | Lambda errors ≥ 1 | Unhandled exceptions in the Lambda function |
autoops-lambda-high-duration-dev | Lambda duration > 3000ms | Latency regressions or downstream slowness |
autoops-lambda-throttles-dev | Lambda throttles ≥ 1 | Concurrency limit hits |
autoops-dynamodb-write-throttles-dev | DynamoDB write throttles ≥ 1 | Write capacity exceeded (the main demo scenario) |
autoops-dynamodb-read-throttles-dev | DynamoDB read throttles ≥ 1 | Read capacity exceeded |
autoops-api-5xx-errors-dev | API Gateway 5XX ≥ 5 | Error rate at the API layer |
All alarms publish to a single SNS topic (
autoops-alarms-dev ). That topic has two subscriptions: an email endpoint and AWS Chatbot. This is a critical design choice — I want engineers notified immediately through familiar channels, independently of the AI investigation pipeline.
The Intelligence Pipeline — How Incidents Trigger Investigations
Step 1: EventBridge Captures the Alarm State Change
When any CloudWatch alarm transitions to
ALARM state, EventBridge captures the event:{ "source": "aws.cloudwatch", "detail-type": "CloudWatch Alarm State Change", "detail": { "state": { "value": "ALARM" } }}
The EventBridge rule filters specifically for
state.value = "ALARM" — this prevents the pipeline from triggering on OK or INSUFFICIENT_DATA transitions, which are not actionable.Key design principle: EventBridge is additive. It adds the DevOps Agent notification channel without modifying any existing SNS subscriptions, Chatbot configuration, or alarm thresholds. Email and Slack via Chatbot continue working exactly as before.
Step 2: Webhook Lambda Formats and Signs the Payload
The EventBridge rule targets a Lambda function (
autoops-devops-agent-webhook-dev ) that:- Retrieves the webhook URL and signing secret from AWS Secrets Manager at runtime — credentials are never hardcoded or stored in environment variables
- Formats the alarm data into a DevOps Agent-compatible incident payload
- Computes an HMAC SHA-256 signature using the format
{timestamp}:{payload}
- POSTs to the DevOps Agent webhook with authentication headers
import hmac, hashlib, base64, jsonfrom datetime import datetime, timezone
def sign_payload(payload: str, secret: str, timestamp: str) -> str: message = f"{timestamp}:{payload}".encode('utf-8') signature = hmac.new(secret.encode('utf-8'), message, hashlib.sha256).digest() return base64.b64encode(signature).decode('utf-8')
def send_webhook(url: str, payload: dict, key: str, secret: str): timestamp = datetime.now(timezone.utc).isoformat() body = json.dumps(payload) signature = sign_payload(body, secret, timestamp)
headers = { 'Content-Type': 'application/json', 'x-amzn-event-timestamp': timestamp, 'x-amzn-event-signature': signature, 'User-Agent': 'AWS-Lambda-DevOps-Agent-Integration/1.0' } # POST with exponential backoff retry (max 3 attempts, no retry on 4xx)
The payload format that the DevOps Agent expects:
{ "eventType": "incident", "incidentId": "{alarmName}-{timestamp}", "action": "created", "priority": "HIGH", "title": "CloudWatch Alarm: {alarmName}", "description": "{stateReason}\n\nAlarm: {alarmName}\nState: {previousState} → ALARM\nRegion: us-east-1", "service": "autoops", "timestamp": "2026-04-13T01:04:00.000Z", "data": { "metadata": { "environment": "dev", "region": "us-east-1", "alarmArn": "arn:aws:cloudwatch:..." } }}
The DevOps Agent Investigation — Phase by Phase
Once the webhook is received, the DevOps Agent runs a structured investigation across the connected resources. The Agent Space (
autoops-incident-intelligence ) is connected to:- Lambda function + its CloudWatch log group
- DynamoDB table metrics
- API Gateway logs
- All 6 CloudWatch alarms
- X-Ray default group
- Slack workspace and
#aws-incidentschannel
Phase 1: Triage (0–20 seconds)
The Agent first checks whether an investigation for the same alarm already exists within a 20-minute window. If a duplicate is found, it links the new investigation to the existing one and posts a "linked" message to Slack — no redundant analysis runs. This prevents alert flooding from repeated alarm state flaps.
Phase 2: Topology Discovery (10–30 seconds)
The Agent maps the resource relationships: Lambda function → DynamoDB table → API Gateway. This topology map guides which metrics and logs to query next.
Phase 3: Log Analysis (30–60 seconds)
The Agent queries
/aws/lambda/autoops-api-dev CloudWatch Logs, scanning for error patterns including ProvisionedThroughputExceededException . It correlates log timestamps against the alarm trigger time to confirm the relationship.Phase 4: Trace Analysis (1–2 minutes)
X-Ray traces are queried to identify which specific API route caused the DynamoDB write failures (e.g.,
POST /items ). The trace data shows the exact call chain: API Gateway → Lambda → DynamoDB → ProvisionedThroughputExceededException .Phase 5: Metrics Analysis (2–3 minutes)
The Agent queries DynamoDB CloudWatch metrics to confirm the capacity mismatch — 1 WCU provisioned versus the actual write demand that hit the table. This quantifies the gap and forms the basis for the remediation recommendation.
Phase 6: Root Cause Report (3–4 minutes)
The Agent generates a natural-language root cause analysis and posts it to
#aws-incidents in Slack. For real incidents, this includes HITL action buttons — the engineer can approve a remediation (such as increasing WCU or switching to On-Demand billing mode) directly from Slack.What the Investigation Looks Like in Practice
Test Alarm Investigation
When I manually trigger an alarm using
SetAlarmState , the Agent correctly identifies that it is not a real incident. Here is what the evidence chain looks like:Key Evidence collected by the Agent:
- Lambda
Errorsmetric: 0 datapoints in the last 30 minutes — no actual errors occurring
-
StateReason: Contains the custom string I set ("Testing DevOps Agent Slack integration") rather than an auto-generated threshold breach message
- CloudTrail: Confirms
SetAlarmStateAPI was called manually — not triggered by a real metric breach
Agent Summary: "This is not a real incident. The alarm was manually set to ALARM state via the
SetAlarmState API. There are zero Lambda errors — the function is operating normally. No further investigation or remediation is needed."This behavior demonstrates an important capability: the Agent is intelligent enough to distinguish between real infrastructure failures and manually triggered test alarms.
Real Throttling Investigation
When
trigger_throttle.py fires 50 concurrent POST requests at the 1 WCU table:- ~45 requests are throttled, ~5 succeed
- Within 1 minute, the
autoops-dynamodb-write-throttles-devalarm fires
- The full investigation completes in 3–4 minutes
- The Agent posts findings with HITL buttons: "Increase Write Capacity" or "Switch to On-Demand"
- No action is taken until a human approves
Notification Channels — Parallel and Independent
All three notification channels run in parallel. They do not depend on each other:
Channel 1 — Email (SNS) CloudWatch alarm → SNS topic → email subscription. This fires within seconds of the alarm state change and includes full alarm details.
Channel 2 — Slack via AWS Chatbot CloudWatch alarm → SNS topic → AWS Chatbot →
#aws-incidents . Chatbot posts an inline CloudWatch graph showing the metric that breached, alarm metadata, and quick-access buttons ("List dashboards", "Query logs").Channel 3 — DevOps Agent (EventBridge) CloudWatch alarm → EventBridge → webhook Lambda → DevOps Agent. This is the AI investigation channel. It posts investigation status, linked investigation notifications, and completed root cause analysis to the same
#aws-incidents channel.IAM Design — Least Privilege Throughout
Each component has its own IAM role scoped to exactly what it needs:
API Lambda Role (
autoops-lambda-role-dev )DynamoDB: GetItem, PutItem, DeleteItem, Query, Scan (scoped to autoops-items-dev ARN only)CW Logs: CreateLogGroup, CreateLogStream, PutLogEventsX-Ray: PutTraceSegments, PutTelemetryRecords
Webhook Lambda Role (
autoops-devops-agent-lambda-role-dev )CW Logs: Basic Lambda execution loggingSecrets Manager: GetSecretValue (scoped to webhook secret ARN only)
DevOps Agent Role (
DevOpsAgentRole-AgentSpace-* )CloudWatch: Read metrics, logs, alarmsLambda: DescribeFunction, GetFunctionConfigurationDynamoDB: DescribeTable, ListTagsOfResourceX-Ray: GetTraceSummaries, BatchGetTracesAWS Support: CreateCase, DescribeCases (for raising support tickets)
No role has
* on any action or resource. Every permission is scoped to the minimum required for that component's function.Triggering the Full Pipeline — Step by Step
Trigger a Real Throttling Incident
python tests/trigger_throttle.py --url https://<your-api-id>.execute-api.us-east-1.amazonaws.com/dev
Timeline of what happens:
Time | Event |
|---|---|
0s | 50 concurrent POST requests sent to the API |
~10s | DynamoDB starts returning ProvisionedThroughputExceededException |
~1 min | CloudWatch alarm fires: autoops-dynamodb-write-throttles-dev |
~1 min | Email notification sent via SNS |
~1 min | Slack notification posted by AWS Chatbot (with CloudWatch graph) |
~1 min | EventBridge triggers webhook Lambda |
~1 min | DevOps Agent receives webhook (HTTP 200 OK) |
~2 min | Agent begins topology discovery and log analysis |
~3–4 min | Root cause analysis posted to #aws-incidents |
~4 min | HITL action buttons appear — engineer approves remediation |
Trigger a Test Alarm (Integration Verification Only)
python tests/trigger_alarm.py
The Agent detects this as a test alarm, posts investigation status to Slack, and correctly concludes that no remediation is needed.
What I Learned Building This
1. The EventBridge additive pattern is powerful. By routing alarm events through EventBridge separately from SNS, I could add the entire DevOps Agent investigation pipeline without touching any existing notification infrastructure. This is a pattern worth applying anywhere you want to augment existing alerting with AI capabilities.
2. HMAC authentication for webhooks is non-negotiable. The DevOps Agent webhook requires HMAC SHA-256 authentication. Implementing this correctly — with a timestamp-prefixed message format and credentials stored in Secrets Manager — took careful attention but is the right approach. Never pass webhook secrets through environment variables.
3. Triage deduplication prevents noise at scale. In a real environment, a DynamoDB throttle might trigger multiple alarms (write throttles alarm, Lambda errors alarm, API 5XX alarm) within seconds of each other. Without deduplication, the Agent would run three simultaneous investigations of the same incident. The 20-minute deduplication window is a thoughtful design by the service team.
4. The Agent correctly distinguishes real incidents from test alarms. I was impressed that the Agent checks CloudTrail for
SetAlarmState API calls to determine whether an alarm was manually triggered. This is genuinely useful — it means test alarms during development do not generate false remediation recommendations.5. Intentional failures make better demos and better tests. Using a 1 WCU DynamoDB table means I can trigger a real, observable incident with a single script. Synthetic alarms are useful, but real infrastructure failures produce more authentic investigation data — the X-Ray traces, the CloudWatch log entries with actual exception messages, and the DynamoDB throttle metrics all paint a cleaner picture for the Agent to analyze.
Current System Status
Component | Status |
|---|---|
API Gateway + Lambda + DynamoDB | ✅ Deployed and operational |
CloudWatch Alarms (6) | ✅ Configured and firing correctly |
Email notifications (SNS) | ✅ Working |
Slack notifications (AWS Chatbot) | ✅ Working |
EventBridge rule | ✅ Capturing alarm state changes |
Webhook Lambda | ✅ Deployed, receiving HTTP 200 OK from Agent |
DevOps Agent investigations | ✅ Completing with root cause analysis |
Investigation results in Slack | ✅ Posting correctly |
HITL action buttons | ✅ Appearing for real incidents; correctly suppressed for test alarms |
Next Steps
The next phase of this project is to run the full HITL workflow end-to-end with a real throttling incident and document the remediation approval flow. The Agent recommends either increasing the WCU or switching to On-Demand billing mode — I plan to test both approval paths and verify that the Agent applies the change correctly after human approval.
I also plan to extend the monitoring coverage with:
- Lambda cold start tracking
- API Gateway latency percentiles (P50, P95, P99)
- DynamoDB consumed capacity trending
- Cost anomaly detection via AWS Cost Anomaly Detection, with alerts routed through the same pipeline
Resources
Built and written by Naveen G. All infrastructure is provisioned as code using Terraform. No resources were created manually.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article