AWS Builder Center
Autonomous Incident Response with AWS DevOps Agent, PagerDuty, and Slack

Autonomous Incident Response with AWS DevOps Agent, PagerDuty, and Slack

Build an automated pipeline that detects errors, creates PagerDuty incidents, alerts Slack, and runs autonomous root cause analysis with AWS DevOps Agent.

Introduction

When a production application fails at 3 AM, the typical response looks like this: PagerDuty wakes an on-call engineer, they log into the AWS console, dig through Amazon CloudWatch logs, correlate metrics, form a hypothesis, and eventually find the root cause. This process takes 30–60 minutes on a good night.
What if the investigation started automatically the moment an incident was created — and the on-call engineer woke up to a root cause analysis already posted to the incident?
That's exactly what AWS DevOps Agent does.
In this post, you will learn how to:
  • Build a fully automated incident response pipeline using AWS services
  • Configure PagerDuty and Slack integrations for real-time alerting
  • Deploy a realistic demo that simulates a production deployment regression
  • Leverage AWS DevOps Agent to autonomously investigate and identify root causes
The entire infrastructure is defined in AWS Cloud Development Kit (AWS CDK) using Python, deploys in under 5 minutes, and includes a realistic order processing API with a feature-flag-controlled bug that triggers autonomous investigation.



Solution overview

The solution implements a fully automated incident lifecycle across four phases:
  1. Error Detection — Amazon CloudWatch catches structured error logs and fires an alarm
  1. Alert & Incident Creation — An AWS Lambda notifier function creates a PagerDuty incident and posts to Slack
  1. Autonomous Investigation — AWS DevOps Agent reads logs and metrics, performs root cause analysis (RCA), and posts findings
  1. Auto-Resolution — When errors stop, the alarm clears, the incident resolves automatically, and Slack is updated
The following diagram shows the architecture of the solution:


Credentials - Zero hardcoding

All secrets are stored in AWS Secrets Manager and read at runtime:
Secret Path
Consumer
Purpose
serverless-demo/pagerduty/routing-key
Notifier Lambda
Trigger/resolve PagerDuty incidents
serverless-demo/pagerduty/api-token
AWS DevOps Agent
Read incidents, post RCA notes
serverless-demo/slack/config
Notifier Lambda
Post alerts to Slack channel
Credentials are injected at deploy time via AWS CDK context (stored in a gitignored  cdk.json ) and written to AWS Secrets Manager by the CDK stack. Nothing is committed to source control.



Prerequisites

Before you begin, ensure you have the following:
  • An AWS account
  • Python 3.11 or later
  • Node.js 18+ (for CDK CLI)
  • A  Slack  workspace with admin permissions to create apps



Setting up PagerDuty

Complete the following steps to configure PagerDuty for this solution:

Step 1: Create a PagerDuty service

  1. Log in to your PagerDuty account.
  1. Navigate to Services → Service Directory → + New Service.
  1. Configure the service:
  • Name:  AWS Serverless Demo 
  • Description:  Automated incident response demo with AWS DevOps Agent 
  • Escalation Policy: Select your default policy or create a new one
  • Integration: Select Events API V2
  1. Choose Create Service.
  1. On the service details page, copy the Integration Key (also called the routing key). Save this value — you'll need it during deployment.

Step 2: Generate a PagerDuty API token

  1. Navigate to Integrations → API Access Keys.
  1. Choose + Create New API Key.
  1. Enter a description:  AWS DevOps Agent - RCA automation 
  1. Choose Create Key.
  1. Copy the generated API key immediately (it won't be shown again). Save this value.

Step 3: Configure PagerDuty V3 webhook (for DevOps Agent)

  1. Navigate to Integrations → Generic Webhooks (v3).
  1. Choose + New Webhook.
  1. Configure:
  • Webhook URL: The endpoint URL for your AWS DevOps Agent (provided after CDK deployment)
  • Description:  AWS DevOps Agent - Autonomous RCA 
  • Scope Type: Service
  • Scope: Select  AWS Serverless Demo  (the service you created in Step 1)
  • Event Subscription: Select  incident.triggered 
  1. Choose Add Webhook.
Note: You'll complete this step after deploying the CDK stack, since the webhook URL is an output of the deployment.



Setting up Slack

Complete the following steps to configure Slack notifications:

Step 1: Create a Slack app with incoming webhook

  1. Go to the Slack API Apps page and choose Create New App.
  1. Select From scratch.
  1. Configure:
  • App Name:  AWS Incident Alerts 
  • Workspace: Select your target workspace
  1. Choose Create App.

Step 2: Enable incoming webhooks

  1. In your app settings, navigate to Features → Incoming Webhooks.
  1. Toggle Activate Incoming Webhooks to On.
  1. Choose Add New Webhook to Workspace at the bottom of the page.
  1. Select the channel where you want alerts posted (e.g.,  #incidents  or  #ops-alerts ).
  1. Choose Allow.
  1. Copy the Webhook URL. Save this value — you will need it during CDK deployment.

Step 3: (Optional) Create a dedicated alerts channel

For production use, we recommend creating a dedicated channel:
  1. In Slack, choose + next to Channels → Create a channel.
  1. Name it  #aws-incident-alerts  (or your preference).
  1. Set the purpose:  Automated AWS incident alerts and resolution notifications .
  1. When adding the webhook (Step 2), select this channel.

Step 4: Verify the webhook

Test the webhook with a cURL command:
curl -X POST -H 'Content-type: application/json' \
--data '{"text":"✅ Slack webhook configured successfully for AWS Incident Alerts!"}' \
YOUR_SLACK_WEBHOOK_URL

You should see the test message appear in your selected Slack channel.



Demo scenario: Order Processor (realistic deployment regression)

This demo simulates a real-world deployment regression — the kind of incident that wakes engineers at 3 AM:
  • An Amazon API Gateway endpoint accepts  POST /orders 
  • An Amazon DynamoDB table holds a feature flag ( payment_validation_v2 )
  • When the flag is enabled (simulating a bug introduced by a deployment), orders over $999.99 fail with a  ValidationError 
  • When the flag is disabled (bug fixed), all orders succeed
This scenario is powerful because AWS DevOps Agent can correlate the error pattern (only high-value orders fail), identify the  $999.99  threshold from the logs, and trace it back to the  payment_validation_v2  flag — exactly the kind of root cause analysis a human engineer would perform.



Walkthrough

Step 1: Clone and configure the project

git clone <YOUR_REPOSITORY_URL>
cd aws-devops-agent-demo
python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt


Step 2: Configure CDK context

Create a  cdk.context.json  file (this file is gitignored for security):
{
"pagerduty_routing_key": "YOUR_PAGERDUTY_ROUTING_KEY",
"pagerduty_api_token": "YOUR_PAGERDUTY_API_TOKEN",
"slack_webhook_url": "YOUR_SLACK_WEBHOOK_URL",
"slack_channel": "#aws-incident-alerts"
}


Step 3: Deploy the stack

cdk bootstrap # Only needed once per account/region
cdk deploy

The deployment creates the following resources:
AWS Service
Resource
Purpose
Amazon API Gateway
REST API (/orders)
HTTP endpoint for order submissions
AWS Lambda
order-processor
Processes orders, reads feature flag
Amazon DynamoDB
order-processor-config
Feature flag table
Amazon CloudWatch Logs
Log group + metric filter
Captures and counts errors
Amazon CloudWatch
Alarm
Fires when error count ≥ 1 in 1 minute
Amazon SNS
Topic
Fan-out alarm notifications
AWS Lambda
cloudwatch-notifier
Creates PagerDuty incidents + Slack alerts
AWS Secrets Manager
3 secrets
Stores PagerDuty and Slack credentials

Step 4: Note deployment outputs

After deployment, CDK outputs the following values:
Outputs:
ApiEndpoint = <your-api-endpoint-url>
DevOpsAgentWebhookUrl = <your-devops-agent-webhook-url>

Use the  DevOpsAgentWebhookUrl  to complete the PagerDuty V3 webhook configuration (see  Setting up PagerDuty, Step 3 ).

Step 5: Run the demo

The demo script sends orders that trigger the bug:
# Start sending orders (runs until you stop it)
./demo.sh

The script sends  $1,500  orders every 30 seconds. Since the feature flag starts enabled (simulating a bad deployment), all orders return HTTP 500.
What happens automatically:
  1. T+0s — First failed order. Lambda logs a structured error.
  1. T+60s — CloudWatch alarm transitions to  ALARM  state.
  1. T+62s — SNS triggers the notifier Lambda.
  1. T+63s — PagerDuty incident created + Slack alert posted.
  1. T+65s — PagerDuty webhook fires → AWS DevOps Agent begins investigation.
  1. T+3–8 min — DevOps Agent posts RCA findings to PagerDuty incident notes.

Step 6: Fix the bug (live demo)

During a live demo, "fix" the bug by disabling the feature flag:
aws dynamodb update-item \
--table-name order-processor-config \
--key '{"flag_name": {"S": "payment_validation_v2"}}' \
--update-expression "SET enabled = :val" \
--expression-attribute-values '{":val": {"BOOL": false}}' \
--region us-east-1

What happens after the fix:
  1. Next order succeeds (HTTP 200).
  1. No new errors → CloudWatch alarm transitions to  OK .
  1. Notifier Lambda resolves the PagerDuty incident.
  1. Slack channel receives a resolution message.



Infrastructure as code (AWS CDK)

The entire stack is defined in Python CDK with modular constructs:

Stack structure

cdk/
├── app.py # CDK app entry point
├── stacks/
│ ├── incident_pipeline_stack.py # Main stack
│ └── constructs/
│ ├── order_processor.py # API Gateway + Lambda + DynamoDB
│ ├── monitoring.py # CloudWatch Logs, Metric Filter, Alarm
│ ├── notification.py # SNS + Notifier Lambda + Secrets
│ └── devops_agent.py # DevOps Agent webhook integration
├── lambda/
│ ├── order_processor/
│ │ └── handler.py
│ └── cloudwatch_notifier/
│ └── handler.py
└── requirements.txt


Key CDK patterns used

  • L2 constructs for type-safe resource definitions
  • Environment-agnostic stacks that deploy to any account/region
  • CDK context for secret injection (gitignored)
  • Lambda bundling with automatic dependency packaging



Cleaning up

To avoid ongoing charges, delete the resources created by this solution:
cdk destroy

This removes all deployed resources including Lambda functions, API Gateway, DynamoDB tables, CloudWatch alarms, SNS topics, and Secrets Manager secrets.
Additionally, manually clean up:



Conclusion

In this post, you learned how to build a fully autonomous incident response pipeline using AWS services. By combining Amazon CloudWatch, Amazon SNS, AWS Lambda, and AWS DevOps Agent, you created a system that detects production errors, creates incidents, performs root cause analysis, and resolves incidents — all without human intervention.
The key takeaways:
  • Autonomous RCA reduces MTTR — Engineers wake up to root cause analysis, not raw alerts
  • Feature-flag bugs are realistic — The demo mirrors actual deployment regressions seen in production
  • Infrastructure as code enables reproducibility — The entire pipeline deploys in under 5 minutes with AWS CDK
  • Zero-trust credentials — All secrets live in AWS Secrets Manager, never in source control
To get started, clone the repository and deploy the CDK stack. For more information about AWS DevOps Agent, see the AWS DevOps Agent documentation in the Amazon Q Developer User Guide.



About the author

Godwin Sahayaraj Vincent is an Enterprise Solutions Architect at AWS specializing in AI/ML workloads. He works with customers to design and implement scalable, cost-effective solutions on AWS.
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article