AWS Builder Center
Building AutoOps: An AI-Powered Incident Intelligence Platform with AWS DevOps Agent

Building AutoOps: An AI-Powered Incident Intelligence Platform with AWS DevOps Agent

How I Built a Fully Automated On-Call System That Investigates, Diagnoses, and Recommends Fixes

Introduction

On-call incident response is one of the most mentally taxing parts of running a cloud-hosted service. When an alarm fires at 2 AM, an engineer has to wake up, log into multiple AWS consoles, correlate logs and metrics across services, piece together a root cause, and then decide on a fix — all under pressure. I wanted to build a system that handles that entire investigation automatically, so the engineer only needs to approve the fix.
In this article, I'll walk through how I built AutoOps — an end-to-end incident intelligence platform using the AWS DevOps Agent, Lambda, EventBridge, CloudWatch, DynamoDB, API Gateway, AWS Chatbot, and Terraform. The system automatically detects infrastructure incidents, runs a multi-phase AI investigation, posts a root cause analysis to Slack, and presents remediation options for human approval before taking any action.

The Problem I Set Out to Solve

Traditional on-call workflows look like this:
  1. CloudWatch alarm fires
  1. Engineer gets paged
  1. Engineer opens CloudWatch, Lambda console, DynamoDB metrics, X-Ray — one by one
  1. Engineer correlates timestamps across services to find the root cause
  1. Engineer decides on a fix and applies it manually
This process can take 20–40 minutes for a straightforward throttling issue. The goal of AutoOps was to reduce that to zero manual investigation time — the system does the analysis automatically and presents the engineer with a clear summary and a one-click approval button.

Architecture Overview

The system has two independent planes:
Plane 1 — The API (the thing that can break) A REST API backed by Lambda and DynamoDB, intentionally provisioned with 1 Write Capacity Unit to make throttling easy to trigger. This creates a realistic incident scenario.
Plane 2 — The Intelligence Pipeline (the thing that fixes it) An event-driven chain starting from CloudWatch alarms, routing through EventBridge, triggering a webhook Lambda that notifies the AWS DevOps Agent, which then runs a full multi-phase investigation and posts findings to Slack.
CLIENT REQUESTS
│
▼
API Gateway REST API (autoops-api-dev)
GET /health
GET /items
POST /items
GET /items/{id}
DELETE /items/{id}
POST /simulate-error
│
▼
Lambda Function (Python 3.12 · 256 MB · X-Ray Active Tracing)
│
▼
DynamoDB Table (1 WCU — intentional for demo throttling)

━━━━━━━━━━━━━━ MONITORING & ALERTING PLANE ━━━━━━━━━━━━━━

CloudWatch Alarms (6 alarms)
│
├──► SNS Topic
│ ├──► Email notification
│ └──► AWS Chatbot → Slack #aws-incidents
│
└──► EventBridge Rule (ALARM state only)
│
▼
Lambda: Webhook Forwarder
├── Retrieve HMAC credentials (Secrets Manager)
├── Sign payload (HMAC SHA-256)
└── POST to DevOps Agent Webhook
│
▼
DevOps Agent Space (autoops-incident-intelligence)
├── Topology Discovery
├── CloudWatch Log Analysis
├── X-Ray Trace Analysis
├── DynamoDB Metrics Analysis
└── Root Cause Report → Slack #aws-incidents
(with HITL approval buttons)


Infrastructure — Built Entirely with Terraform

Every resource in this system is defined as code. I structured the Terraform files by service concern so each component can be reasoned about independently.
terraform/
├── apigateway.tf # REST API + CloudWatch access logging
├── cloudwatch.tf # 6 alarms + dashboard
├── devops_agent.tf # SSM params, EventBridge rule, webhook Lambda
├── dynamodb.tf # Table with intentionally low WCU
├── iam.tf # All roles and policies
├── lambda.tf # Function + X-Ray tracing
├── outputs.tf # Endpoint URLs and resource names
├── providers.tf # AWS provider
├── s3.tf # Lambda artifact bucket
├── sns.tf # Topic + email + Chatbot subscription
└── variables.tf # Input definitions


Deploying the Infrastructure

cd terraform
terraform init
terraform plan
terraform apply

After apply, Terraform outputs the API endpoint URL, CloudWatch dashboard URL, DynamoDB table name, Lambda function name, S3 artifact bucket, and SNS topic ARN. It also prints next-step instructions for post-deployment configuration.



The API Application

The Lambda function handles five API routes, all backed by DynamoDB:
# Simplified handler structure
def lambda_handler(event, context):
method = event['httpMethod']
path = event['path']

if path == '/health':
return health_check()
elif path == '/items' and method == 'GET':
return list_items()
elif path == '/items' and method == 'POST':
return create_item(event)
elif path.startswith('/items/') and method == 'GET':
return get_item(item_id)
elif path.startswith('/items/') and method == 'DELETE':
return delete_item(item_id)
elif path == '/simulate-error':
return simulate_error()

X-Ray tracing is enabled at the function level with  TracingConfig: Active , which means every DynamoDB call, every exception, and every subsegment is captured and available for the DevOps Agent to query during investigation.

CloudWatch Alarms — 6 Alarms Covering Every Layer

I created alarms at every layer of the stack so no class of failure goes undetected:
Alarm
Condition
What It Catches
autoops-lambda-errors-dev
Lambda errors ≥ 1
Unhandled exceptions in the Lambda function
autoops-lambda-high-duration-dev
Lambda duration > 3000ms
Latency regressions or downstream slowness
autoops-lambda-throttles-dev
Lambda throttles ≥ 1
Concurrency limit hits
autoops-dynamodb-write-throttles-dev
DynamoDB write throttles ≥ 1
Write capacity exceeded (the main demo scenario)
autoops-dynamodb-read-throttles-dev
DynamoDB read throttles ≥ 1
Read capacity exceeded
autoops-api-5xx-errors-dev
API Gateway 5XX ≥ 5
Error rate at the API layer
All alarms publish to a single SNS topic ( autoops-alarms-dev ). That topic has two subscriptions: an email endpoint and AWS Chatbot. This is a critical design choice — I want engineers notified immediately through familiar channels, independently of the AI investigation pipeline.



The Intelligence Pipeline — How Incidents Trigger Investigations

Step 1: EventBridge Captures the Alarm State Change

When any CloudWatch alarm transitions to  ALARM  state, EventBridge captures the event:
{
"source": "aws.cloudwatch",
"detail-type": "CloudWatch Alarm State Change",
"detail": {
"state": {
"value": "ALARM"
}
}
}

The EventBridge rule filters specifically for  state.value = "ALARM"  — this prevents the pipeline from triggering on  OK  or  INSUFFICIENT_DATA  transitions, which are not actionable.
Key design principle: EventBridge is additive. It adds the DevOps Agent notification channel without modifying any existing SNS subscriptions, Chatbot configuration, or alarm thresholds. Email and Slack via Chatbot continue working exactly as before.

Step 2: Webhook Lambda Formats and Signs the Payload

The EventBridge rule targets a Lambda function ( autoops-devops-agent-webhook-dev ) that:
  1. Retrieves the webhook URL and signing secret from AWS Secrets Manager at runtime — credentials are never hardcoded or stored in environment variables
  1. Formats the alarm data into a DevOps Agent-compatible incident payload
  1. Computes an HMAC SHA-256 signature using the format  {timestamp}:{payload} 
  1. POSTs to the DevOps Agent webhook with authentication headers
import hmac, hashlib, base64, json
from datetime import datetime, timezone

def sign_payload(payload: str, secret: str, timestamp: str) -> str:
message = f"{timestamp}:{payload}".encode('utf-8')
signature = hmac.new(secret.encode('utf-8'), message, hashlib.sha256).digest()
return base64.b64encode(signature).decode('utf-8')

def send_webhook(url: str, payload: dict, key: str, secret: str):
timestamp = datetime.now(timezone.utc).isoformat()
body = json.dumps(payload)
signature = sign_payload(body, secret, timestamp)

headers = {
'Content-Type': 'application/json',
'x-amzn-event-timestamp': timestamp,
'x-amzn-event-signature': signature,
'User-Agent': 'AWS-Lambda-DevOps-Agent-Integration/1.0'
}
# POST with exponential backoff retry (max 3 attempts, no retry on 4xx)

The payload format that the DevOps Agent expects:
{
"eventType": "incident",
"incidentId": "{alarmName}-{timestamp}",
"action": "created",
"priority": "HIGH",
"title": "CloudWatch Alarm: {alarmName}",
"description": "{stateReason}\n\nAlarm: {alarmName}\nState: {previousState} → ALARM\nRegion: us-east-1",
"service": "autoops",
"timestamp": "2026-04-13T01:04:00.000Z",
"data": {
"metadata": {
"environment": "dev",
"region": "us-east-1",
"alarmArn": "arn:aws:cloudwatch:..."
}
}
}


The DevOps Agent Investigation — Phase by Phase

Once the webhook is received, the DevOps Agent runs a structured investigation across the connected resources. The Agent Space ( autoops-incident-intelligence ) is connected to:
  • Lambda function + its CloudWatch log group
  • DynamoDB table metrics
  • API Gateway logs
  • All 6 CloudWatch alarms
  • X-Ray default group
  • Slack workspace and  #aws-incidents  channel

Phase 1: Triage (0–20 seconds)

The Agent first checks whether an investigation for the same alarm already exists within a 20-minute window. If a duplicate is found, it links the new investigation to the existing one and posts a "linked" message to Slack — no redundant analysis runs. This prevents alert flooding from repeated alarm state flaps.

Phase 2: Topology Discovery (10–30 seconds)

The Agent maps the resource relationships: Lambda function → DynamoDB table → API Gateway. This topology map guides which metrics and logs to query next.

Phase 3: Log Analysis (30–60 seconds)

The Agent queries  /aws/lambda/autoops-api-dev  CloudWatch Logs, scanning for error patterns including  ProvisionedThroughputExceededException . It correlates log timestamps against the alarm trigger time to confirm the relationship.

Phase 4: Trace Analysis (1–2 minutes)

X-Ray traces are queried to identify which specific API route caused the DynamoDB write failures (e.g.,  POST /items ). The trace data shows the exact call chain: API Gateway → Lambda → DynamoDB →  ProvisionedThroughputExceededException .

Phase 5: Metrics Analysis (2–3 minutes)

The Agent queries DynamoDB CloudWatch metrics to confirm the capacity mismatch — 1 WCU provisioned versus the actual write demand that hit the table. This quantifies the gap and forms the basis for the remediation recommendation.

Phase 6: Root Cause Report (3–4 minutes)

The Agent generates a natural-language root cause analysis and posts it to  #aws-incidents  in Slack. For real incidents, this includes HITL action buttons — the engineer can approve a remediation (such as increasing WCU or switching to On-Demand billing mode) directly from Slack.

What the Investigation Looks Like in Practice

Test Alarm Investigation

When I manually trigger an alarm using  SetAlarmState , the Agent correctly identifies that it is not a real incident. Here is what the evidence chain looks like:
Key Evidence collected by the Agent:
  • Lambda  Errors  metric: 0 datapoints in the last 30 minutes — no actual errors occurring
  •  StateReason : Contains the custom string I set ("Testing DevOps Agent Slack integration") rather than an auto-generated threshold breach message
  • CloudTrail: Confirms  SetAlarmState  API was called manually — not triggered by a real metric breach
Agent Summary: "This is not a real incident. The alarm was manually set to ALARM state via the  SetAlarmState  API. There are zero Lambda errors — the function is operating normally. No further investigation or remediation is needed."
This behavior demonstrates an important capability: the Agent is intelligent enough to distinguish between real infrastructure failures and manually triggered test alarms.

Real Throttling Investigation

When  trigger_throttle.py  fires 50 concurrent POST requests at the 1 WCU table:
  • ~45 requests are throttled, ~5 succeed
  • Within 1 minute, the  autoops-dynamodb-write-throttles-dev  alarm fires
  • The full investigation completes in 3–4 minutes
  • The Agent posts findings with HITL buttons: "Increase Write Capacity" or "Switch to On-Demand"
  • No action is taken until a human approves

Notification Channels — Parallel and Independent

All three notification channels run in parallel. They do not depend on each other:
Channel 1 — Email (SNS) CloudWatch alarm → SNS topic → email subscription. This fires within seconds of the alarm state change and includes full alarm details.
Channel 2 — Slack via AWS Chatbot CloudWatch alarm → SNS topic → AWS Chatbot →  #aws-incidents . Chatbot posts an inline CloudWatch graph showing the metric that breached, alarm metadata, and quick-access buttons ("List dashboards", "Query logs").
Channel 3 — DevOps Agent (EventBridge) CloudWatch alarm → EventBridge → webhook Lambda → DevOps Agent. This is the AI investigation channel. It posts investigation status, linked investigation notifications, and completed root cause analysis to the same  #aws-incidents  channel.

IAM Design — Least Privilege Throughout

Each component has its own IAM role scoped to exactly what it needs:
API Lambda Role ( autoops-lambda-role-dev )
DynamoDB: GetItem, PutItem, DeleteItem, Query, Scan (scoped to autoops-items-dev ARN only)
CW Logs: CreateLogGroup, CreateLogStream, PutLogEvents
X-Ray: PutTraceSegments, PutTelemetryRecords

Webhook Lambda Role ( autoops-devops-agent-lambda-role-dev )
CW Logs: Basic Lambda execution logging
Secrets Manager: GetSecretValue (scoped to webhook secret ARN only)

DevOps Agent Role ( DevOpsAgentRole-AgentSpace-* )
CloudWatch: Read metrics, logs, alarms
Lambda: DescribeFunction, GetFunctionConfiguration
DynamoDB: DescribeTable, ListTagsOfResource
X-Ray: GetTraceSummaries, BatchGetTraces
AWS Support: CreateCase, DescribeCases (for raising support tickets)

No role has  *  on any action or resource. Every permission is scoped to the minimum required for that component's function.

Triggering the Full Pipeline — Step by Step

Trigger a Real Throttling Incident

python tests/trigger_throttle.py --url https://<your-api-id>.execute-api.us-east-1.amazonaws.com/dev

Timeline of what happens:
Time
Event
0s
50 concurrent POST requests sent to the API
~10s
DynamoDB starts returning ProvisionedThroughputExceededException
~1 min
CloudWatch alarm fires: autoops-dynamodb-write-throttles-dev
~1 min
Email notification sent via SNS
~1 min
Slack notification posted by AWS Chatbot (with CloudWatch graph)
~1 min
EventBridge triggers webhook Lambda
~1 min
DevOps Agent receives webhook (HTTP 200 OK)
~2 min
Agent begins topology discovery and log analysis
~3–4 min
Root cause analysis posted to #aws-incidents
~4 min
HITL action buttons appear — engineer approves remediation

Trigger a Test Alarm (Integration Verification Only)

python tests/trigger_alarm.py

The Agent detects this as a test alarm, posts investigation status to Slack, and correctly concludes that no remediation is needed.

What I Learned Building This

1. The EventBridge additive pattern is powerful. By routing alarm events through EventBridge separately from SNS, I could add the entire DevOps Agent investigation pipeline without touching any existing notification infrastructure. This is a pattern worth applying anywhere you want to augment existing alerting with AI capabilities.
2. HMAC authentication for webhooks is non-negotiable. The DevOps Agent webhook requires HMAC SHA-256 authentication. Implementing this correctly — with a timestamp-prefixed message format and credentials stored in Secrets Manager — took careful attention but is the right approach. Never pass webhook secrets through environment variables.
3. Triage deduplication prevents noise at scale. In a real environment, a DynamoDB throttle might trigger multiple alarms (write throttles alarm, Lambda errors alarm, API 5XX alarm) within seconds of each other. Without deduplication, the Agent would run three simultaneous investigations of the same incident. The 20-minute deduplication window is a thoughtful design by the service team.
4. The Agent correctly distinguishes real incidents from test alarms. I was impressed that the Agent checks CloudTrail for  SetAlarmState  API calls to determine whether an alarm was manually triggered. This is genuinely useful — it means test alarms during development do not generate false remediation recommendations.
5. Intentional failures make better demos and better tests. Using a 1 WCU DynamoDB table means I can trigger a real, observable incident with a single script. Synthetic alarms are useful, but real infrastructure failures produce more authentic investigation data — the X-Ray traces, the CloudWatch log entries with actual exception messages, and the DynamoDB throttle metrics all paint a cleaner picture for the Agent to analyze.

Current System Status

Component
Status
API Gateway + Lambda + DynamoDB
✅ Deployed and operational
CloudWatch Alarms (6)
✅ Configured and firing correctly
Email notifications (SNS)
✅ Working
Slack notifications (AWS Chatbot)
✅ Working
EventBridge rule
✅ Capturing alarm state changes
Webhook Lambda
✅ Deployed, receiving HTTP 200 OK from Agent
DevOps Agent investigations
✅ Completing with root cause analysis
Investigation results in Slack
✅ Posting correctly
HITL action buttons
✅ Appearing for real incidents; correctly suppressed for test alarms



Next Steps

The next phase of this project is to run the full HITL workflow end-to-end with a real throttling incident and document the remediation approval flow. The Agent recommends either increasing the WCU or switching to On-Demand billing mode — I plan to test both approval paths and verify that the Agent applies the change correctly after human approval.
I also plan to extend the monitoring coverage with:
  • Lambda cold start tracking
  • API Gateway latency percentiles (P50, P95, P99)
  • DynamoDB consumed capacity trending
  • Cost anomaly detection via AWS Cost Anomaly Detection, with alerts routed through the same pipeline

Resources



Built and written by Naveen G. All infrastructure is provisioned as code using Terraform. No resources were created manually.
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article