Building a Multi-Agent AI DevOps Incident Response Agent on AWS with Strands Agents SDK and Amazon Bedrock

When a production incident fires at 2 AM, every minute of downtime translates to lost revenue, eroded customer trust, and exhausted engineers. The typical incident response cycle — alert fires, on-call is paged, engineer logs in, diagnoses the issue, applies a fix, and writes a post-mortem — can stretch from minutes to hours depending on how well your systems, runbooks, and escalation paths are automated.

sweta jha

Published Jun 27, 2026

Last modified Jun 27, 2026

Building a Multi-Agent AI DevOps Incident Response Agent on AWS with Strands Agents SDK and Amazon Bedrock

GitHub: https://github.com/its-me-sweta/Devops_Inncident_Response_Agent.git

Introduction

In this post I'll walk through a production-ready, eight-agent AI pipeline that automates the full DevOps incident response lifecycle — from CloudWatch alarm to blameless post-mortem — using:

Strands Agents SDK with @tool decorators
Amazon Bedrock Nova Pro for RCA generation, runbook RAG, and post-mortem drafting
Amazon Titan Embed v2 for runbook semantic search (FAISS + Titan)
Amazon CloudWatch Logs Insights for real-time log forensics
AWS X-Ray for distributed trace analysis
AWS Systems Manager for automated EC2 and ECS remediation
AWS CodeDeploy for deployment rollback
Amazon DynamoDB for incident records, change management, and ITIL audit trail
Amazon SES + SNS for stakeholder notifications and war room coordination
Amazon EventBridge Scheduler for 30-minute recurring updates
AgentCore Runtime for serverless agent deployment

Architecture Overview

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
                   ┌──────────────────────────────────────────────┐
                   │       Alert Sources                           │
                   │ CloudWatch · PagerDuty · Datadog · Synthetic  │
                   └───────────────────┬──────────────────────────┘
                                       │
                   ┌───────────────────▼──────────────────────────┐
                   │           Orchestrator Agent                   │
                   │  SEV-1→4 · ITIL Routing · Error Budget · SRE  │
                   └──────────┬────────────────────────────┬──────┘
                               │                            │
              ┌────────────────▼──────┐     ┌──────────────▼────────────┐
              │   Triage Agent         │     │   Diagnosis Agent          │
              │   Service Graph        │     │   CloudWatch Metrics       │
              │   SLO Impact           │     │   Logs Insights            │
              │   Affected Users       │     │   X-Ray Traces             │
              │   Change Correlation   │     │   Runbook RAG (FAISS)      │
              │   Blast Radius         │     │   RCA (confidence score)   │
              └────────────────┬──────┘     └──────────────┬────────────┘
                               └──────────────┬─────────────┘
                                              │
                   ┌──────────────────────────▼───────────────────┐
                   │         Auto-Remediation Agent                │
                   │  Deployment Rollback · ECS Restart            │
                   │  ASG Scale-Out · Cache Flush · SSM Runbook    │
                   │  Validate → Rollback-if-Failed                │
                   └──────────────────────────┬───────────────────┘
                                              │
               ┌──────────────────────────────┼─────────────────────────┐
               │                              │                         │
  ┌────────────▼───────────┐  ┌───────────────▼──────────┐  ┌──────────▼──────────┐
  │  Escalation Agent       │  │  Communication Agent      │  │  Change Mgmt Agent  │
  │  PagerDuty Page         │  │  Slack 30-min Updates     │  │  Emergency RFC       │
  │  War Room               │  │  Status Page              │  │  CAB Approval        │
  │  Jira Ticket            │  │  Customer Emails          │  │  Pipeline Freeze     │
  │  SLA Breach Alarm       │  │  Leadership Brief         │  │  Change Calendar     │
  └────────────────────────┘  └───────────────────────────┘  └─────────────────────┘
                                              │
                   ┌──────────────────────────▼───────────────────┐
                   │           Post-Mortem Agent                   │
                   │  Blameless Timeline · Five Whys               │
                   │  Action Items (Jira) · Confluence Doc         │
                   │  Runbook Update · SLO/Error Budget Tracking   │
                   └──────────────────────────────────────────────┘

ITIL/SRE Severity Framework

1
2
3
4
5
6
SEV_SLA = {
    "SEV-1": 30,   # minutes to resolve — war room + all-hands + pipeline freeze
    "SEV-2": 120,  # minutes — war room + status page
    "SEV-3": 240,  # minutes — on-call only
    "SEV-4": 480,  # minutes — next business day acceptable
}

Stage 1 — Triage Agent

The Triage Agent is the first responder — it contextualises the alert before any diagnosis or remediation.

Service Graph lookup from CMDB determines upstream/downstream impact:

1
2
3
4
5
SERVICE_GRAPH = {
    "api-gateway":    {"downstream": ["auth-service", "user-service", "order-service"]},
    "order-service":  {"downstream": ["payment-service", "inventory-service", "notification-service"]},
    "payment-service":{"downstream": ["payment-gateway", "fraud-detection"]},
}

SLO/Error Budget assessment:

1
2
3
4
5
@tool
def assess_slo_impact(service_name, severity) -> dict:
    # Queries CloudWatch for 5xx error count (30-min window)
    # Calculates error budget consumed this month
    # Returns: slo_breached, monthly_budget_used_pct, budget_critical

Change correlation is the most powerful triage signal — if a deployment happened within 2 hours of the incident, the correlation probability is 0.80 (rollback is the right call):

1
2
correlation = 0.80 if recent_deployments else 0.10
recommended_action = "ROLLBACK" if correlation > 0.70 else "CONTINUE_DIAGNOSIS"

Stage 2 — Diagnosis Agent

Three parallel evidence streams feed the RCA:

CloudWatch Logs Insights — scans for error patterns in real-time:

1
2
3
4
query = """fields @timestamp, @message
           | filter @message like /ERROR|CRITICAL|FATAL|Exception|OutOfMemory/
           | sort @timestamp desc | limit 25"""
logs_cl.start_query(logGroupName=log_group, startTime=start, endTime=end, queryString=query)

AWS X-Ray trace analysis — finds error traces and slow segments:

1
2
3
xray.get_trace_summaries(
    FilterExpression=f'service("{service_name}") AND error',
    StartTime=start, EndTime=end)

Runbook RAG (FAISS + Titan Embed v2) — retrieves the most relevant troubleshooting procedure:

1
2
3
embedding = get_embedding(f"{service_name} {error_pattern}")
# Semantic search over runbook knowledge base
# Returns: troubleshooting steps, estimated resolution time, known fixes

RCA confidence scoring:

1
2
3
4
if deployment_correlation:   confidence = max(confidence, 0.88)  # HIGH → auto-remediate
if log_pattern_found:        confidence = max(confidence, 0.70)  # MEDIUM → remediate + monitor
if xray_errors > 5:          confidence = max(confidence, 0.72)
# < 0.60 → escalate to L2/L3 for manual diagnosis

Stage 3 — Auto-Remediation Agent

Selects the right fix based on root cause, with built-in safety gates:

Deployment rollback (most common — used for regression incidents):

1
2
3
@tool
def trigger_deployment_rollback(service_name, deployment_id, incident_id) -> dict:
    codedeploy.stop_deployment(deploymentId=deployment_id, autoRollbackEnabled=True)

ECS service restart (for OOM, deadlock, hung processes):

1
ecs.update_service(cluster=cluster_name, service=service_name, forceNewDeployment=True)

Infrastructure scale-out (for capacity incidents):

1
2
asg.set_desired_capacity(AutoScalingGroupName=resource_name,
                          DesiredCapacity=new_capacity, HonorCooldown=False)

Remediation validation — waits 3 minutes, then checks 3 consecutive clean data points:

1
2
3
4
@tool
def validate_remediation(service_name, incident_id) -> dict:
    recent_errors = query_cloudwatch_5xx_count(last_5_minutes)
    return {"validation_passed": recent_errors < 10}

Rollback-if-Failed — if validation fails, immediately undoes the remediation and escalates to L2:

1
2
3
if not validation_passed:
    rollback_if_failed(incident_id, action_taken, rollback_procedure)
    # Escalate — maximum 2 auto-remediation attempts per incident

SAFETY RULES enforced in code:

NEVER restart Tier-1 stateful services (RDS primary) without human confirmation
ALWAYS take a snapshot before destructive operations
Maximum 2 auto-remediation attempts per incident

Stage 4 — Escalation Agent

War room creation for SEV-1/SEV-2 — auto-creates a Slack channel with full incident context:

1
2
3
lambda_cl.invoke(FunctionName="devops-slack-create-channel",
                 Payload=json.dumps({"channel_name": f"incident-{incident_id.lower()}",
                                      "initial_message": context_summary}))

SLA breach alarm — CloudWatch alarm fires before SLA deadline with 15-minute warning:

1
2
3
4
cw.put_metric_alarm(
    AlarmName=f"SLA-BREACH-{incident_id}",
    AlarmActions=[ONCALL_TOPIC],  # Pages VP Engineering on SLA risk
)

Pipeline freeze for SEV-1 — no production deployments until resolved:

1
2
sns.publish(TopicArn=FREEZE_TOPIC,
            Message=json.dumps({"action": "FREEZE", "message": "FREEZE: No production deployments"}))

Stage 5 — Communication Agent

Structured 30-minute updates for SEV-1 via EventBridge Scheduler:

1
2
3
4
5
scheduler.create_schedule(
    Name=f"ir-update-{incident_id}-{update_number}",
    ScheduleExpression=f"at({next_update})",
    Target={"Arn": "arn:aws:lambda:...:function:devops-incident-update", ...}
)

Status page update format:

1
2
3
4
Investigating: "We are aware of issues affecting {service}. Our engineers are investigating."
Identified:    "We have identified the root cause as {cause} and are implementing a fix."
Monitoring:    "A fix has been deployed. We are monitoring to confirm resolution."
Resolved:      "This incident has been resolved. See our post-mortem for full details."

Stage 6 — Change Management Agent

Every incident remediation creates an Emergency Change Record (RFC) — ITIL-compliant:

1
2
3
4
5
6
@tool
def create_emergency_change(incident_id, description, rollback_plan, implementer) -> dict:
    # RFC number: RFC-2025-XXXXXX
    # Change type: EMERGENCY (bypasses normal CAB)
    # SEV-1/2: VP Engineering pre-delegates approval authority
    # Post-hoc CAB review required at next weekly meeting

Pipeline freeze is a critical SEV-1 safety measure — blocks all production deployments and notifies all engineering teams via SNS.

Stage 7 — Post-Mortem Agent

Blameless post-mortem with Four Key Metrics:

1
2
3
4
MTTD (Mean Time to Detect):      5 minutes  — first alert to incident declared
MTTA (Mean Time to Acknowledge):  8 minutes  — incident declared to engineer engaged
MTTC (Mean Time to Contain):     22 minutes  — engaged to fix deployed
MTTR (Mean Time to Resolve):     45 minutes  — fix deployed to service fully restored

Five Whys generated automatically from the RCA:

1
2
3
4
5
6
7
whys = [
    {"why": "Why did the service fail?",          "answer": root_cause},
    {"why": "Why was that possible?",             "answer": "Missing input validation"},
    {"why": "Why wasn't it caught in testing?",   "answer": "Integration tests missed this path"},
    {"why": "Why wasn't it detected sooner?",     "answer": "Alert threshold too high (5%, should be 0.5%)"},
    {"why": "Why did it take so long to resolve?","answer": "Runbook was outdated"},
]

Runbook self-update — the agent writes the fix that worked back into the runbook:

1
2
runbooks_table.update_item(
    UpdateExpression="SET steps=list_append(steps, :ns), known_fixes=list_append(..., :fix)")

Error budget tracking:

1
2
3
error_budget_consumed = (duration_minutes / (30 * 24 * 60)) * 100  # % of monthly SLO budget
if error_budget_consumed > 50:
    trigger_reliability_review()   # Mandatory architecture review

AWS Infrastructure Summary

Resource	Purpose
Amazon Bedrock Nova Pro	RCA narrative, post-mortem drafting
Amazon Titan Embed v2	Runbook semantic search (FAISS)
Amazon CloudWatch Logs Insights	Real-time log forensics
AWS X-Ray	Distributed trace analysis
AWS CodeDeploy	Deployment rollback
AWS Systems Manager	EC2/ECS automated remediation
Amazon DynamoDB (6 tables)	Incidents, runbooks, changes, post-mortems, comms, SLO
Amazon S3	Post-mortem documents, runbook storage
Amazon SES	Stakeholder emails, customer notifications
Amazon SNS	War room, on-call pages, pipeline freeze, status page
Amazon EventBridge Scheduler	30-minute incident updates
AgentCore Runtime	Serverless agent deployment

Try It Yourself

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
git clone https://github.com/its-me-sweta/Devops_Inncident_Response_Agent.git
cd devops-incident-response-agent
pip install -r requirements.txt
uvicorn main:app --reload

# Simulate a SEV-2 production incident
curl -X POST http://localhost:8000/incident/create \
  -H "Content-Type: application/json" \
  -d '{
    "alert_source": "cloudwatch",
    "title": "order-service: 5xx error rate > 5% for 5 minutes",
    "description": "HTTPCode_Target_5XX_Count exceeded threshold. Error rate: 8.3%.",
    "affected_service": "order-service",
    "severity": "SEV-2",
    "region": "us-east-1",
    "log_group": "/aws/ecs/order-service",
    "cluster_name": "production-cluster"
  }'

Built with Strands Agents SDK · Amazon Bedrock Nova Pro · AgentCore Runtime

# strands-agents # amazon-bedrock # agentic-ai # aws-community-builders

Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.

Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article

Builder Center

Building a Multi-Agent AI DevOps Incident Response Agent on AWS with Strands Agents SDK and Amazon Bedrock

sweta jha

Building a Multi-Agent AI DevOps Incident Response Agent on AWS with Strands Agents SDK and Amazon Bedrock

GitHub: https://github.com/its-me-sweta/Devops_Inncident_Response_Agent.git

Introduction

Architecture Overview

ITIL/SRE Severity Framework

Stage 1 — Triage Agent

Stage 2 — Diagnosis Agent

Stage 3 — Auto-Remediation Agent

Stage 4 — Escalation Agent

Stage 5 — Communication Agent

Stage 6 — Change Management Agent

Stage 7 — Post-Mortem Agent

AWS Infrastructure Summary

Try It Yourself