Autonomous Incident Response with AWS DevOps Agent

Learn how to wire AWS DevOps Agent into your CloudWatch alarm pipeline so it automatically investigates incidents, follows your team's runbooks, and delivers a diagnosis and ready-to-approve remediation plan to Slack — before a human even opens a laptop.

Bhavya Chugh

AI Acceleration Architect at AWS

Published Jun 29, 2026

Last modified Jun 29, 2026

This article was co-authored with Adarsha Nagappasetty , Senior Solution Architect, AWS

The Problem
Solution Overview
What is AWS DevOps Agent?
How the Workflow Operates in Practice
Architecture
Technical Deep Dive
Deployment Steps
Testing
Conclusion

The Problem

When a production incident occurs, the typical response follows a familiar and frustrating pattern. A CloudWatch alarm fires and pages an on-call engineer. The engineer wakes up — often at 2 AM — opens a laptop, connects to VPN, and begins the manual triage process. They check service health, read logs, inspect resource states, and search internal wikis or runbooks for the right remediation steps. After applying a fix and verifying it works, they write up a post-mortem.

This process takes 30–60 minutes on average, and significantly longer during off-hours when the on-call engineer may be less familiar with the failing service. The diagnostic steps are repetitive and well-understood, yet a human must perform them every time. The quality of the response depends entirely on who happens to be on-call and how quickly they can context-switch between CloudWatch, service tooling, runbooks, and Slack.

The core issues are clear: slow mean time to resolution due to manual triage, constant context switching between tools, inconsistent response quality across team members, and significant toil from running the same diagnostic commands for every incident regardless of root cause.

Solution Overview

This project demonstrates AWS DevOps Agent as an autonomous first responder for infrastructure incidents. EKS CrashLoopBackOff is used as a concrete example, but the pattern applies broadly to any service where CloudWatch alarms, runbooks, and a defined remediation workflow exist.

Instead of paging a human to perform the initial investigation, the agent takes over the entire diagnostic workflow the moment a CloudWatch alarm fires.

The agent automatically investigates by running diagnostic commands to identify the failing component and gather signals. It then diagnoses the issue by correlating symptoms to specific root causes using structured runbooks stored in a GitHub repository and accessed via a custom MCP server. Based on the matching runbook, it proposes a concrete mitigation plan with specific commands, not generic advice. Throughout this process, it posts findings, root cause analysis, and the proposed mitigation plan to a designated Slack channel where the operations team works.

Critically, the agent does not blindly execute fixes. Any remediation action that mutates infrastructure state requires explicit human approval before execution. The on-call engineer receives a Slack message with a complete diagnosis and a ready-to-approve fix, rather than a raw alarm that demands 30+ minutes of manual investigation.

Please clone the code from github repo .

What is AWS DevOps Agent?

AWS DevOps Agent is a frontier AI agent built for operations teams. It functions as an always-available operations teammate that investigates incidents autonomously, identifies root causes, and recommends targeted mitigations — working the way an experienced DevOps engineer would, but available 24/7 without fatigue or context-switching overhead.

At its core, the agent learns your infrastructure by building an application topology — a graph of your resources and their relationships. When an incident occurs, it doesn't start from scratch. It already understands how your services connect, which resources depend on each other, and where to look for signals. It correlates telemetry data, code changes, and deployment history to narrow down root causes quickly, rather than relying on generic troubleshooting heuristics.

The agent is organized around Agent Spaces, which are logical containers that define what the agent can access and investigate. Each Agent Space holds your AWS account configurations, third-party tool integrations, access permissions, and custom MCP servers. This means you can scope the agent's access precisely — giving it visibility into the accounts and tools it needs for a particular application or team, without over-provisioning access.

One of the agent's key strengths is its integration ecosystem. It works natively with popular observability platforms including Amazon CloudWatch, Datadog, Dynatrace, New Relic, and Splunk. It connects to code repositories and CI/CD pipelines through GitHub and GitLab integrations, allowing it to correlate deployment events with operational issues. For communication, it integrates with Slack and ServiceNow, routing findings and mitigation plans directly into the channels and ticketing systems teams already use. And through custom MCP (Model Context Protocol) servers, you can extend the agent's capabilities to interact with any internal tool or data source — which is exactly how this project connects the agent to operational runbooks.

The agent operates through a dual-console architecture. Administrators use the AWS Management Console to create and manage Agent Spaces, configure integrations, and set up access controls. Operations teams use the DevOps Agent web app for day-to-day incident response — interacting with investigations, browsing cross-account application topology, and reviewing preventative recommendations for observability, code, pipelines, and infrastructure.

Beyond reactive incident response, the agent continuously analyzes patterns across historical incidents to generate targeted recommendations. These aren't generic best practices — they're specific, actionable improvements tailored to your environment, covering observability gaps (missing alarms, insufficient logging), infrastructure optimization (autoscaling thresholds, capacity tuning), and deployment pipeline enhancements (testing coverage, validation steps). The goal is to shift teams from reactive firefighting to proactive operational improvement.

How the Workflow Operates in Practice

The autonomous incident response pipeline handles any production issue that can be detected by a CloudWatch alarm. The core principle is straightforward: when something breaks, the system follows the exact same diagnostic steps your best engineers would follow, but it does so in seconds rather than minutes, and it does so consistently every single time.

Consider what happens when a production incident occurs at 3 AM. The traditional response requires a human to context-switch from sleep to a complex debugging session. They must remember which cluster is affected, what namespace to look at, which logs to check, and what remediation steps are safe to apply. This cognitive overhead is where mistakes happen. The autonomous pipeline eliminates this gap by encoding your team's operational knowledge into machine-executable runbooks and letting an AI agent execute the diagnostic portion immediately.

The workflow follows a three-phase execution model that applies regardless of the underlying service. In the first phase, autonomous diagnosis, the agent executes read-only commands to gather state. For EKS this means running kubectl get pods, kubectl describe, and kubectl logs. For RDS it might be aws rds describe-db-instances and aws cloudwatch get-metric-data. For Lambda it could be aws logs filter-log-events and aws lambda get-function-configuration. The agent runs any read-only command without human approval because these operations cannot change system state.

In the second phase, proposed remediation, the agent identifies the root cause by cross-referencing diagnostic output against your runbooks, then proposes specific commands to fix the issue. These proposals include the exact commands, the runbook they came from, the identified root cause, and a severity classification. No mutating command is ever executed without explicit human approval — this is enforced at the MCP server level, not merely by prompt engineering.

In the third phase, autonomous verification, the agent runs verification commands to confirm the fix worked after a human approves and the remediation is applied. It checks that the service returned to a healthy state and reports the outcome back to your incident channel.

This model means the agent handles the tedious, time-sensitive diagnostic work autonomously while preserving human judgment for the high-stakes remediation decisions. The human reviewer sees a complete picture: what's broken, why it's broken (with evidence), what the runbook recommends, and the exact commands the agent proposes to run. They approve with one click or reject with a comment explaining why.

The pipeline is deliberately service-agnostic. Four components change between use cases: the alarm (which CloudWatch metric triggers the pipeline), the runbook (what diagnostic and remediation steps to follow), the investigation prompt (what the trigger Lambda tells the agent to do), and the operator role permissions (what AWS APIs the agent can call). Everything else — the MCP server, the approval workflow, the Slack integration, the EventBridge routing — remains identical regardless of whether you're troubleshooting a database failover, a Lambda timeout spike, or a load balancer health check failure.

Architecture

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
┌──────────────────────────────────────────────────────────────────────┐
│                      Your Infrastructure                             │
│         (e.g. EKS cluster, ECS services, EC2 fleets, etc.)          │
│                                                                      │
│   Emits metrics, logs, and health signals continuously               │
└──────────────────────┬───────────────────────────────────────────────┘
                       │ Metrics / telemetry
                       ▼
              ┌─────────────────┐
              │  CloudWatch     │
              │  Alarms         │
              │  (any threshold)│
              └────────┬────────┘
                       │ State → ALARM
                       ▼
              ┌─────────────────┐
              │  EventBridge    │
              │  Rule           │
              │  (alarm match)  │
              └────────┬────────┘
                       │
                       ▼
              ┌─────────────────┐
              │  Lambda         │
              │  (trigger fn)   │──── Creates INVESTIGATION task
              │                 │──── Starts execution via SigV4
              └────────┬────────┘
                       │
                       ▼
              ┌─────────────────────────────────────────────┐
              │         AWS DevOps Agent                     │
              │         Agent Space                          │
              │  ┌───────────┐  ┌────────────┐              │
              │  │ Service   │  │ MCP Server │              │
              │  │ Access    │  │ (Runbooks) │              │
              │  │ (IAM)     │  │            │              │
              │  └───────────┘  └────────────┘              │
              │  ┌───────────┐  ┌────────────┐              │
              │  │ Slack     │  │ GitHub     │              │
              │  │ Notifs    │  │ Integration│              │
              │  └─────┬─────┘  └────────────┘              │
              └────────┼────────────────────────────────────┘
                       │
                       ▼
              ┌─────────────────┐
              │  Slack Channel  │
              │  - Findings     │
              │  - Root Cause   │
              │  - Mitigation   │
              │    Plan         │
              └─────────────────┘

Core Concepts

Concept	Description
Agent Space	An isolated environment where the agent operates. Contains associations, capability providers, and investigations.
Investigation	A task the agent executes — triggered by an alarm, a human request, or an API call.
Capability Provider	An external system the agent can interact with (Slack, MCP servers, EKS clusters).
MCP Server	A JSON-RPC endpoint that exposes tools the agent can call (e.g., `search_runbooks`, `get_runbook`).
Operator Role	An IAM role assumed by the agent to interact with your AWS resources (EKS, CloudWatch, etc.).

Technical Deep Dive

Authentication Chain

Each hop in the pipeline uses a distinct authentication mechanism. The trigger Lambda authenticates to the DevOps Agent API using SigV4 via its IAM role. The DevOps Agent accesses the EKS cluster through an IAM role and EKS Access Entry. It reaches the MCP server using an API key passed in the X-Api-Key header. The MCP server Lambda authenticates to GitHub using a Personal Access Token stored in SSM. Finally, the DevOps Agent communicates with Slack through a one-time OAuth registration.

Connection	Auth Method	Credential Storage
Lambda → DevOps Agent API	SigV4 (IAM role)	IAM role on Lambda
DevOps Agent → EKS	IAM role + EKS Access Entry	EKS access policy
DevOps Agent → MCP Server	API Key (`X-Api-Key` header)	SSM SecureString
MCP Server → GitHub API	Personal Access Token	SSM SecureString
DevOps Agent → Slack	OAuth	Console registration

Infrastructure Components

The pipeline deploys as two CloudFormation stacks. The first stack (devops-agent-mcp-server) creates the API Gateway HTTP endpoint that the DevOps Agent calls, the Lambda function implementing the MCP protocol and bridging to GitHub, and the IAM role scoped to reading specific SSM parameters for the GitHub token and API key.

The second stack (devops-agent-incident-response) creates the EKS cluster (v1.32) with a managed node group, CloudWatch alarms monitoring Container Insights metrics for CrashLoopBackOff and Pod-NotReady conditions, an EventBridge rule matching alarm state changes, the trigger Lambda that parses alarm events and creates investigations via SigV4, a KMS key for EKS secrets encryption, dead-letter queues for all Lambda functions, and an SNS topic for alarm notifications.

Technical Deep Dive

MCP Server: The Runbook Bridge

The Model Context Protocol (MCP) is the mechanism through which the DevOps Agent accesses external knowledge at investigation time. Rather than baking operational knowledge into the agent's prompt or relying on the agent's general training data, the MCP server provides a live, versioned connection to your team's actual runbooks stored in GitHub. When an engineer updates a runbook after learning from an incident, the next investigation automatically uses the updated procedure.

The MCP server acts as a JSON-RPC bridge between the DevOps Agent and your GitHub repository. When the agent encounters a production issue, it calls tools like search_runbooks("OOMKilled") to find relevant procedures, then get_runbook("eks-crashloopbackoff-runbook.md") to retrieve the full diagnostic and remediation steps. You can add support for entirely new failure modes by committing a new markdown file to your runbooks repository — no infrastructure changes, no redeployment, no agent reconfiguration required.

The server implements MCP protocol version 2024-11-05 and exposes seven tools. Three are read-only runbook access tools: list_runbooks enumerates all markdown files in the repository, get_runbook fetches the full content of a specific runbook, and search_runbooks performs full-text keyword search across all runbooks. Four support the human-in-the-loop approval workflow: propose_remediation submits a remediation plan with commands for approval, get_pending_approvals lists actions awaiting human review, get_action_status checks whether a specific proposed action has been approved or rejected, and review_action allows a human operator to approve or reject a proposed action.

The MCP server separates proposed commands into safe (auto-approved) and unsafe (requires human approval) using a server-side allowlist:

1
2
3
4
5
6
7
8
9
10
SAFE_COMMAND_PREFIXES = [
    "kubectl get ",
    "kubectl describe ",
    "kubectl logs ",
    "kubectl cluster-info",
    "kubectl top ",
    "aws eks describe-",
    "aws eks list-",
    "aws ecr describe-",
]

Any command not matching these prefixes — such as kubectl patch, kubectl delete, or kubectl rollout restart — is classified as needing human approval. This enforcement happens server-side in the MCP Lambda, meaning the agent cannot bypass it regardless of what its reasoning produces. To extend the system for new services, you add read-only command prefixes for those services (e.g., aws rds describe- for RDS, aws logs filter-log-events for CloudWatch Logs).

The MCP server also validates input to prevent path traversal attacks. The get_runbook tool sanitizes filenames before passing them to the GitHub API, rejecting any input containing .. or absolute paths.

Trigger Lambda: Alarm-to-Investigation Bridge

The trigger Lambda translates CloudWatch alarm events into DevOps Agent investigations. It is intentionally lightweight, containing no diagnostic logic. All intelligence lives in the agent and its runbooks. The Lambda simply receives an alarm event, extracts the alarm metadata, and creates an investigation task with a structured prompt that guides the agent toward the relevant diagnostic steps and runbooks.

The Lambda supports three event formats: EventBridge (the primary path, matching source: aws.cloudwatch with detail-type: CloudWatch Alarm State Change), SNS (for alarms routed through an SNS topic), and direct invocation (for manual testing during development).

The investigation task is created via the DevOps Agent API using SigV4 authentication. The Lambda calls two endpoints: one to create the task in the agent's backlog, and another to start execution immediately. The structured prompt it constructs tells the agent to list failing pods, check their events and logs, search runbooks for matching remediation procedures, cite the specific runbook section that applies, and propose remediation commands using the propose_remediation MCP tool.

The prompt is the most important customization point when adapting this pipeline to new services. For each failure domain, you change the prompt to reference the appropriate namespace, service name, diagnostic commands, and runbook search terms. The underlying infrastructure remains the same.

Human-in-the-Loop Execution Model

The approval workflow is enforced server-side in the MCP Lambda, not in the agent's prompt. This is a critical architectural decision: even if the agent's reasoning were to produce a destructive command, the MCP server would classify it as unsafe and return a "pending approval" response rather than executing it. The agent cannot bypass this check because the MCP server is the only path to remediation execution. This provides defense-in-depth that does not rely on prompt engineering alone.

In practice, the agent posts a structured remediation proposal to Slack that includes the identified root cause, the source runbook section, and the exact commands it wants to run. The on-call engineer reviews this in context — they see what the agent found, why it recommends this fix, and what will happen if they approve. This transforms the human's role from "debug everything from scratch" to "review a pre-built diagnosis and approve or reject a specific action."

Phase	Agent Autonomy	Example Commands
Diagnosis	Fully autonomous	`kubectl get pods`, `kubectl logs`, `kubectl describe`
Remediation	Proposal only — human approves	`kubectl patch`, `kubectl rollout restart`, `kubectl set image`
Verification	Fully autonomous	`kubectl get pod -w`, check restart count stability

IAM Permissions

The trigger Lambda role follows least-privilege principles with four explicit DevOps Agent actions scoped to the specific agent space:

1
2
3
4
5
6
7
8
9
10
{
  "Effect": "Allow",
  "Action": [
    "aidevops:InvokeAgent",
    "aidevops:CreateBacklogTask",
    "aidevops:GetBacklogTask",
    "aidevops:ListExecutions"
  ],
  "Resource": "arn:aws:aidevops:<region>:<account>:agentspace/*"
}

The DevOps Agent operator role (assumed by aidevops.amazonaws.com) has EKS read and API access (eks:DescribeCluster, eks:ListClusters, eks:AccessKubernetesApi), DevOps Agent task management scoped to the agent space, and an EKS Access Entry with AmazonEKSClusterAdminPolicy for kubectl operations within the cluster. The MCP server role has access only to the two specific SSM parameters containing the GitHub token and API key, plus KMS decrypt permission scoped to keys used via the SSM service.

Deployment Steps

Prerequisites

You will need the AWS CLI configured with admin access, kubectl installed locally, an AWS account in a supported region, a GitHub repository (or organization) containing your operational runbooks, and a Slack workspace where the agent will post incident findings. The GitHub repository should have a runbooks/ directory containing markdown files that follow the structured format described in this guide.

Step 1: Create an AWS DevOps Agent Space

Agent Space creation is done through the console — there is no CLI or CloudFormation support for this step. Navigate to the AWS DevOps Agent console and click Create Agent Space. Give it a name, select your AWS account, and note the Agent Space ID (a UUID). You will need this ID in subsequent steps when deploying the CloudFormation stacks and configuring integrations.

Step 2: Connect Your GitHub Organization

The DevOps Agent can connect directly to your GitHub organization, giving it the ability to correlate code changes and deployments with operational issues. This connection is separate from the MCP server (which provides runbook access) — it allows the agent to understand your application topology by scanning repositories for infrastructure definitions, deployment configurations, and service relationships.

To connect GitHub, go to your Agent Space in the console and navigate to Settings > Source code connections. Click Connect repository and select GitHub as the provider. You will be redirected to GitHub to authorize the AWS DevOps Agent application. Select the organization and repositories you want the agent to have access to. For this use case, ensure the repository containing your runbooks is included.

Once connected, the agent can discover your application topology by scanning infrastructure-as-code files (CloudFormation, Terraform, CDK), Kubernetes manifests, and CI/CD pipeline definitions. This gives it contextual understanding of how services relate to each other, which is valuable during investigations when a failure in one service may be caused by a change in an upstream dependency.

The GitHub connection uses a GitHub App installation (not a Personal Access Token). This means access is managed through GitHub's organization settings, follows your existing branch protection rules, and can be revoked centrally by an organization administrator without rotating individual tokens.

Step 3: Store Secrets in SSM Parameter Store

The MCP server needs two secrets stored in SSM Parameter Store: a GitHub Personal Access Token for reading runbook content, and an API key that secures the MCP server endpoint.

For the GitHub token, navigate to GitHub Settings > Developer settings > Personal access tokens > Fine-grained tokens. Generate a token named devops-agent-runbooks with an expiration appropriate to your team's rotation policy (90 days is typical). Grant it repository access only to the runbooks repository with Contents read-only permission. Store the token in SSM:

1
2
3
4
5
aws ssm put-parameter \
  --name /devops-agent/github-token \
  --value "ghp_YOUR_TOKEN_HERE" \
  --type SecureString \
  --region us-east-1

For the MCP API key, generate a random 32-byte hex string and store it. This key authenticates requests from the DevOps Agent to your MCP server endpoint, providing application-level security on top of the network-level controls:

1
2
3
4
5
6
7
MCP_KEY=$(openssl rand -hex 32)

aws ssm put-parameter \
  --name /github-mcp-server/MCP_API_KEY \
  --value "$MCP_KEY" \
  --type SecureString \
  --region us-east-1

Step 4: Register Slack

In the DevOps Agent console, navigate to Capability Providers > Slack > Register and complete the OAuth flow to authorize the DevOps Agent app in your Slack workspace. After authorization, open Slack and add the AWS DevOps Agent app to your incident response channel. The agent will post investigation findings, root cause analyses, and remediation proposals to this channel.

To find your Channel ID, right-click the channel name in Slack and select Copy Link. The URL format is https://app.slack.com/client/T01ABCDEF/C01ABCDEF — the second identifier (starting with C) is the Channel ID you will need if configuring Slack programmatically through the CloudFormation template.

Step 5: Deploy the MCP Server Stack

This stack creates the Lambda function implementing the MCP protocol, an API Gateway HTTP endpoint the DevOps Agent calls, and the IAM role with read access to your SSM parameters. The API Gateway stage includes access logging to CloudWatch, and the Lambda function is configured with a dead-letter queue and KMS encryption for environment variables.

1
2
3
4
5
6
7
8
9
10
aws cloudformation create-stack \
  --stack-name devops-agent-mcp-server \
  --template-body file://MyAgentSpace/lambda/github-mcp-server-cfn.yaml \
  --parameters \
    ParameterKey=GitHubTokenSSMParam,ParameterValue=/devops-agent/github-token \
    ParameterKey=MCPApiKeySSMParam,ParameterValue=/github-mcp-server/MCP_API_KEY \
    ParameterKey=GitHubRepo,ParameterValue=<YOUR_ORG>/<YOUR_REPO> \
    ParameterKey=GitHubBranch,ParameterValue=main \
  --capabilities CAPABILITY_NAMED_IAM \
  --region us-east-1

Wait for the stack to complete and retrieve the endpoint URL from the stack outputs:

1
2
3
4
5
6
7
8
aws cloudformation wait stack-create-complete \
  --stack-name devops-agent-mcp-server --region us-east-1

aws cloudformation describe-stacks \
  --stack-name devops-agent-mcp-server \
  --query 'Stacks[0].Outputs' \
  --output table \
  --region us-east-1

Step 6: Register the MCP Server in the DevOps Agent Console

Navigate to Capability Providers > MCP Servers > Add in the DevOps Agent console. Enter devops-agent-mcp-server as the name, paste the MCPServerEndpoint URL from the previous step's output, select API Key as the auth type, set the header name to X-Api-Key, and provide the API key value (retrieve it with

aws ssm get-parameter --name /github-mcp-server/MCP_API_KEY --with-decryption --query 'Parameter.Value' --output text --region us-east-1

Leave Dynamic Client Registration unchecked and leave the private connection option unchecked (the MCP server uses API Gateway, not a Lambda Function URL, which avoids account guardrails that may block public function URLs). Click Register, then associate the MCP server with your Agent Space.

Step 7: Deploy the Infrastructure Stack

This stack creates the EKS cluster with secrets encryption enabled, a managed node group, CloudWatch alarms monitoring Container Insights metrics, an EventBridge rule routing alarm state changes, and the trigger Lambda with a dead-letter queue. The EKS cluster's public endpoint is restricted to the CIDR block you specify — use your VPN or office IP range to prevent open access.

1
2
3
4
5
6
7
8
aws cloudformation create-stack \
  --stack-name devops-agent-incident-response \
  --template-body file://AWS-DevOpsAgent-eks-crashloop-test.yaml \
  --parameters \
    ParameterKey=AgentSpaceId,ParameterValue=<YOUR_AGENT_SPACE_ID> \
    ParameterKey=AllowedPublicAccessCidr,ParameterValue=<YOUR_VPN_CIDR>/32 \
  --capabilities CAPABILITY_NAMED_IAM \
  --region us-east-1

This takes 10–15 minutes due to EKS cluster creation. Wait for completion:

1
2
aws cloudformation wait stack-create-complete \
  --stack-name devops-agent-incident-response --region us-east-1

Step 8: Configure kubectl Access

Update your local kubeconfig to point at the new cluster and verify connectivity:

1
2
aws eks update-kubeconfig --name devops-agent-test-cluster --region us-east-1
kubectl cluster-info

If the cluster's public endpoint is CIDR-restricted and you are connecting from an IP outside that range, you will need to either add your current IP to the allowed CIDRs or connect through your VPN.

Step 9: Grant the DevOps Agent EKS Access

The agent's operator IAM role needs kubectl access to the cluster. First, enable the API authentication mode so the cluster accepts both ConfigMap and API-based access entries:

1
2
3
4
5
6
aws eks update-cluster-config \
  --name devops-agent-test-cluster \
  --access-config authenticationMode=API_AND_CONFIG_MAP \
  --region us-east-1

aws eks wait cluster-active --name devops-agent-test-cluster --region us-east-1

Then create the access entry for the DevOps Agent operator role and associate the cluster admin policy. Replace <ACCOUNT_ID> with your AWS account number:

1
2
3
4
5
6
7
8
9
10
11
aws eks create-access-entry \
  --cluster-name devops-agent-test-cluster \
  --principal-arn arn:aws:iam::<ACCOUNT_ID>:role/DevOpsAgentRole-WebappAdmin \
  --region us-east-1

aws eks associate-access-policy \
  --cluster-name devops-agent-test-cluster \
  --principal-arn arn:aws:iam::<ACCOUNT_ID>:role/DevOpsAgentRole-WebappAdmin \
  --policy-arn arn:aws:eks::aws:cluster-access-policy/AmazonEKSClusterAdminPolicy \
  --access-scope type=cluster \
  --region us-east-1

Step 10: Deploy Test Workloads

Deploy five pods that each fail in a different way, giving the agent distinct failure modes to diagnose:

1
kubectl apply -f MyAgentSpace/k8s-manifests/crashloop-test-deployment.yaml

Verify the pods are crashing as expected:

1
kubectl get pods -n devops-agent-test

You should see crashloop-app-error in CrashLoopBackOff (application exit code 1), crashloop-oomkilled in CrashLoopBackOff with OOMKilled reason (memory limit exceeded), crashloop-missing-config in CreateContainerConfigError (referencing a nonexistent ConfigMap), crashloop-probe-failure in CrashLoopBackOff after about 45 seconds (liveness probe failure), and crashloop-image-error in ImagePullBackOff (invalid image reference).

Testing

Triggering an Investigation

You can manually trigger the alarm to test the full pipeline without waiting for a real metric breach:

1
2
3
4
5
aws cloudwatch set-alarm-state \
  --alarm-name "devops-agent-test-cluster-CrashLoopBackOff" \
  --state-value ALARM \
  --state-reason "Manual test" \
  --region us-east-1

Within seconds, the trigger Lambda should fire. Within one to two minutes, the investigation appears in the DevOps Agent console. Within three to five minutes, the agent posts its findings and remediation plan to Slack, citing the specific runbook sections that match each failure mode.

Verifying the Pipeline

Check the trigger Lambda logs to confirm it received the alarm event and created an investigation:

1
2
aws logs tail "/aws/lambda/devops-agent-test-cluster-trigger-devops-agent" \
  --since 5m --region us-east-1

Look for Created task: and Started execution for task: in the output. Then navigate to your Agent Space in the console and check the Investigations section for a new entry with the alarm name as its title.

What the Agent Does

The agent executes this workflow autonomously: it discovers failing pods by listing all pods in the devops-agent-test namespace, diagnoses each one by running kubectl describe and kubectl logs --previous to check exit codes and error messages, searches your runbooks via MCP for matching remediation procedures, retrieves the full runbook content and identifies the specific cause section that applies, then proposes remediation commands from the runbook using the propose_remediation MCP tool. It reports its findings to Slack in a summary format showing each pod, its root cause, the matching runbook section, and the proposed commands.

Troubleshooting

If the trigger Lambda is not firing, verify the EventBridge rule matches your alarm name prefix, confirm the Lambda has the correct IAM permissions (aidevops:InvokeAgent, aidevops:CreateBacklogTask), and check CloudWatch Logs for errors.

If the agent is not posting to Slack, confirm the DevOps Agent app has been added to the correct Slack channel, verify Slack is associated with the correct Agent Space (not a previous one), and check the investigation status in the console — if it shows as running but nothing appears in Slack, the association may be misconfigured.

If the MCP server is returning errors, test the endpoint directly with curl:

1
2
3
4
5
curl -s -X POST "<MCP_ENDPOINT_URL>" \
  -H "Content-Type: application/json" \
  -H "X-Api-Key: <YOUR_API_KEY>" \
  -d '{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"list_runbooks","arguments":{}}}' \
  | python3 -m json.tool

If you see an authentication error, verify the API key in SSM matches what you registered in the console. If you see a GitHub error, verify the Personal Access Token has read access to the repository and has not expired.

Conclusion

What We Built

This guide walked through building a fully automated incident response pipeline that detects pod failures via CloudWatch alarms and Container Insights metrics, routes alarm events through EventBridge to a trigger Lambda, investigates autonomously by running kubectl diagnostics and correlating signals, retrieves team-specific remediation procedures from GitHub via MCP, proposes concrete fixes with the exact commands from the matching runbook section, enforces human approval for any mutating operation, and reports findings and remediation plans directly in Slack.

Extending the Pattern

This architecture handles any production issue where you can define an alarm, write a runbook, and describe the investigation steps. The infrastructure components remain identical regardless of the target service. To add a new failure domain, you create a CloudWatch alarm for the relevant metric, write a runbook in your GitHub repository following the structured format (YAML frontmatter with trigger conditions, diagnostic steps the agent can run autonomously, and remediation commands that require approval), and update the trigger Lambda's investigation prompt to reference the new service and runbook search terms.

For example, adding RDS failover investigation means creating an alarm on DatabaseConnections or CPUUtilization, writing an rds-failover-runbook.md with diagnostic steps like checking Multi-AZ status and recent events, and updating the prompt to instruct the agent to run aws rds describe-db-instances and search for the RDS runbook. Adding Lambda timeout investigation means alarming on Duration exceeding a threshold, writing a lambda-timeout-runbook.md covering cold start analysis and memory tuning, and updating the prompt accordingly. The pattern scales horizontally: each new service domain adds a runbook and an alarm, not new infrastructure.

Cleanup

To tear down all resources created by this guide:

1
2
3
4
5
6
7
8
kubectl delete namespace devops-agent-test

aws cloudformation delete-stack --stack-name devops-agent-incident-response --region us-east-1

aws cloudformation delete-stack --stack-name devops-agent-mcp-server --region us-east-1

aws ssm delete-parameter --name /devops-agent/github-token --region us-east-1
aws ssm delete-parameter --name /github-mcp-server/MCP_API_KEY --region us-east-1

Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.

Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article

Builder Center

Autonomous Incident Response with AWS DevOps Agent

Bhavya Chugh

Table of Contents

The Problem

Solution Overview

What is AWS DevOps Agent?

How the Workflow Operates in Practice

Architecture

Core Concepts

Technical Deep Dive

Authentication Chain

Infrastructure Components

Technical Deep Dive

MCP Server: The Runbook Bridge

Trigger Lambda: Alarm-to-Investigation Bridge

Human-in-the-Loop Execution Model

IAM Permissions

Deployment Steps

Prerequisites

Step 1: Create an AWS DevOps Agent Space

Step 2: Connect Your GitHub Organization

Step 3: Store Secrets in SSM Parameter Store

Step 4: Register Slack

Step 5: Deploy the MCP Server Stack

Step 6: Register the MCP Server in the DevOps Agent Console

Step 7: Deploy the Infrastructure Stack

Step 8: Configure kubectl Access

Step 9: Grant the DevOps Agent EKS Access

Step 10: Deploy Test Workloads

Testing

Triggering an Investigation

Verifying the Pipeline

What the Agent Does

Troubleshooting

Conclusion

What We Built

Extending the Pattern

Cleanup