
Autonomous Incident Response with AWS DevOps Agent, PagerDuty, and Slack
Build an automated pipeline that detects errors, creates PagerDuty incidents, alerts Slack, and runs autonomous root cause analysis with AWS DevOps Agent.
Introduction
When a production application fails at 3 AM, the typical response looks like this: PagerDuty wakes an on-call engineer, they log into the AWS console, dig through Amazon CloudWatch logs, correlate metrics, form a hypothesis, and eventually find the root cause. This process takes 30–60 minutes on a good night.
What if the investigation started automatically the moment an incident was created — and the on-call engineer woke up to a root cause analysis already posted to the incident?
That's exactly what AWS DevOps Agent does.
In this post, you will learn how to:
- Build a fully automated incident response pipeline using AWS services
- Configure PagerDuty and Slack integrations for real-time alerting
- Deploy a realistic demo that simulates a production deployment regression
- Leverage AWS DevOps Agent to autonomously investigate and identify root causes
The entire infrastructure is defined in AWS Cloud Development Kit (AWS CDK) using Python, deploys in under 5 minutes, and includes a realistic order processing API with a feature-flag-controlled bug that triggers autonomous investigation.
Solution overview
The solution implements a fully automated incident lifecycle across four phases:
- Error Detection — Amazon CloudWatch catches structured error logs and fires an alarm
- Alert & Incident Creation — An AWS Lambda notifier function creates a PagerDuty incident and posts to Slack
- Autonomous Investigation — AWS DevOps Agent reads logs and metrics, performs root cause analysis (RCA), and posts findings
- Auto-Resolution — When errors stop, the alarm clears, the incident resolves automatically, and Slack is updated
The following diagram shows the architecture of the solution:

Credentials - Zero hardcoding
All secrets are stored in AWS Secrets Manager and read at runtime:
Secret Path | Consumer | Purpose |
|---|---|---|
serverless-demo/pagerduty/routing-key | Notifier Lambda | Trigger/resolve PagerDuty incidents |
serverless-demo/pagerduty/api-token | AWS DevOps Agent | Read incidents, post RCA notes |
serverless-demo/slack/config | Notifier Lambda | Post alerts to Slack channel |
Credentials are injected at deploy time via AWS CDK context (stored in a gitignored
cdk.json ) and written to AWS Secrets Manager by the CDK stack. Nothing is committed to source control.
Prerequisites
Before you begin, ensure you have the following:
- An AWS account
- AWS CDK v2 installed and bootstrapped
- Python 3.11 or later
- Node.js 18+ (for CDK CLI)
- A PagerDuty account (free tier works)
- A Slack workspace with admin permissions to create apps
- AWS CLI v2 configured with appropriate credentials
Setting up PagerDuty
Complete the following steps to configure PagerDuty for this solution:
Step 1: Create a PagerDuty service
- Log in to your PagerDuty account.
- Navigate to Services → Service Directory → + New Service.
- Configure the service:
- Name:
AWS Serverless Demo
- Description:
Automated incident response demo with AWS DevOps Agent
- Escalation Policy: Select your default policy or create a new one
- Integration: Select Events API V2
- Choose Create Service.
- On the service details page, copy the Integration Key (also called the routing key). Save this value — you'll need it during deployment.
Step 2: Generate a PagerDuty API token
- Navigate to Integrations → API Access Keys.
- Choose + Create New API Key.
- Enter a description:
AWS DevOps Agent - RCA automation
- Choose Create Key.
- Copy the generated API key immediately (it won't be shown again). Save this value.
Step 3: Configure PagerDuty V3 webhook (for DevOps Agent)
- Navigate to Integrations → Generic Webhooks (v3).
- Choose + New Webhook.
- Configure:
- Webhook URL: The endpoint URL for your AWS DevOps Agent (provided after CDK deployment)
- Description:
AWS DevOps Agent - Autonomous RCA
- Scope Type: Service
- Scope: Select
AWS Serverless Demo(the service you created in Step 1)
- Event Subscription: Select
incident.triggered
- Choose Add Webhook.
Note: You'll complete this step after deploying the CDK stack, since the webhook URL is an output of the deployment.
Setting up Slack
Complete the following steps to configure Slack notifications:
Step 1: Create a Slack app with incoming webhook
- Go to the Slack API Apps page and choose Create New App.
- Select From scratch.
- Configure:
- App Name:
AWS Incident Alerts
- Workspace: Select your target workspace
- Choose Create App.
Step 2: Enable incoming webhooks
- In your app settings, navigate to Features → Incoming Webhooks.
- Toggle Activate Incoming Webhooks to On.
- Choose Add New Webhook to Workspace at the bottom of the page.
- Select the channel where you want alerts posted (e.g.,
#incidentsor#ops-alerts).
- Choose Allow.
- Copy the Webhook URL. Save this value — you will need it during CDK deployment.
Step 3: (Optional) Create a dedicated alerts channel
For production use, we recommend creating a dedicated channel:
- In Slack, choose + next to Channels → Create a channel.
- Name it
#aws-incident-alerts(or your preference).
- Set the purpose:
Automated AWS incident alerts and resolution notifications.
- When adding the webhook (Step 2), select this channel.
Step 4: Verify the webhook
Test the webhook with a cURL command:
curl -X POST -H 'Content-type: application/json' \ --data '{"text":"✅ Slack webhook configured successfully for AWS Incident Alerts!"}' \ YOUR_SLACK_WEBHOOK_URL
You should see the test message appear in your selected Slack channel.
Demo scenario: Order Processor (realistic deployment regression)
This demo simulates a real-world deployment regression — the kind of incident that wakes engineers at 3 AM:
- An Amazon API Gateway endpoint accepts
POST /orders
- An Amazon DynamoDB table holds a feature flag (
payment_validation_v2)
- When the flag is enabled (simulating a bug introduced by a deployment), orders over $999.99 fail with a
ValidationError
- When the flag is disabled (bug fixed), all orders succeed
This scenario is powerful because AWS DevOps Agent can correlate the error pattern (only high-value orders fail), identify the
$999.99 threshold from the logs, and trace it back to the payment_validation_v2 flag — exactly the kind of root cause analysis a human engineer would perform.
Walkthrough
Step 1: Clone and configure the project
git clone <YOUR_REPOSITORY_URL>cd aws-devops-agent-demopython -m venv .venvsource .venv/bin/activatepip install -r requirements.txt
Step 2: Configure CDK context
Create a
cdk.context.json file (this file is gitignored for security):{ "pagerduty_routing_key": "YOUR_PAGERDUTY_ROUTING_KEY", "pagerduty_api_token": "YOUR_PAGERDUTY_API_TOKEN", "slack_webhook_url": "YOUR_SLACK_WEBHOOK_URL", "slack_channel": "#aws-incident-alerts"}
Step 3: Deploy the stack
cdk bootstrap # Only needed once per account/regioncdk deploy
The deployment creates the following resources:
AWS Service | Resource | Purpose |
|---|---|---|
Amazon API Gateway | REST API (/orders) | HTTP endpoint for order submissions |
AWS Lambda | order-processor | Processes orders, reads feature flag |
Amazon DynamoDB | order-processor-config | Feature flag table |
Amazon CloudWatch Logs | Log group + metric filter | Captures and counts errors |
Amazon CloudWatch | Alarm | Fires when error count ≥ 1 in 1 minute |
Amazon SNS | Topic | Fan-out alarm notifications |
AWS Lambda | cloudwatch-notifier | Creates PagerDuty incidents + Slack alerts |
AWS Secrets Manager | 3 secrets | Stores PagerDuty and Slack credentials |
Step 4: Note deployment outputs
After deployment, CDK outputs the following values:
Outputs: ApiEndpoint = <your-api-endpoint-url> DevOpsAgentWebhookUrl = <your-devops-agent-webhook-url>
Use the
DevOpsAgentWebhookUrl to complete the PagerDuty V3 webhook configuration (see Setting up PagerDuty, Step 3 ).Step 5: Run the demo
The demo script sends orders that trigger the bug:
# Start sending orders (runs until you stop it)./demo.sh
The script sends
$1,500 orders every 30 seconds. Since the feature flag starts enabled (simulating a bad deployment), all orders return HTTP 500.What happens automatically:
- T+0s — First failed order. Lambda logs a structured error.
- T+60s — CloudWatch alarm transitions to
ALARMstate.
- T+62s — SNS triggers the notifier Lambda.
- T+63s — PagerDuty incident created + Slack alert posted.
- T+65s — PagerDuty webhook fires → AWS DevOps Agent begins investigation.
- T+3–8 min — DevOps Agent posts RCA findings to PagerDuty incident notes.
Step 6: Fix the bug (live demo)
During a live demo, "fix" the bug by disabling the feature flag:
aws dynamodb update-item \ --table-name order-processor-config \ --key '{"flag_name": {"S": "payment_validation_v2"}}' \ --update-expression "SET enabled = :val" \ --expression-attribute-values '{":val": {"BOOL": false}}' \ --region us-east-1
What happens after the fix:
- Next order succeeds (HTTP 200).
- No new errors → CloudWatch alarm transitions to
OK.
- Notifier Lambda resolves the PagerDuty incident.
- Slack channel receives a resolution message.
Infrastructure as code (AWS CDK)
The entire stack is defined in Python CDK with modular constructs:
Stack structure
cdk/├── app.py # CDK app entry point├── stacks/│ ├── incident_pipeline_stack.py # Main stack│ └── constructs/│ ├── order_processor.py # API Gateway + Lambda + DynamoDB│ ├── monitoring.py # CloudWatch Logs, Metric Filter, Alarm│ ├── notification.py # SNS + Notifier Lambda + Secrets│ └── devops_agent.py # DevOps Agent webhook integration├── lambda/│ ├── order_processor/│ │ └── handler.py│ └── cloudwatch_notifier/│ └── handler.py└── requirements.txt
Key CDK patterns used
- L2 constructs for type-safe resource definitions
- Environment-agnostic stacks that deploy to any account/region
- CDK context for secret injection (gitignored)
- Lambda bundling with automatic dependency packaging
Cleaning up
To avoid ongoing charges, delete the resources created by this solution:
cdk destroy
This removes all deployed resources including Lambda functions, API Gateway, DynamoDB tables, CloudWatch alarms, SNS topics, and Secrets Manager secrets.
Additionally, manually clean up:
- The PagerDuty service and webhook created in Setting up PagerDuty
- The Slack app created in Setting up Slack
Conclusion
In this post, you learned how to build a fully autonomous incident response pipeline using AWS services. By combining Amazon CloudWatch, Amazon SNS, AWS Lambda, and AWS DevOps Agent, you created a system that detects production errors, creates incidents, performs root cause analysis, and resolves incidents — all without human intervention.
The key takeaways:
- Autonomous RCA reduces MTTR — Engineers wake up to root cause analysis, not raw alerts
- Feature-flag bugs are realistic — The demo mirrors actual deployment regressions seen in production
- Infrastructure as code enables reproducibility — The entire pipeline deploys in under 5 minutes with AWS CDK
- Zero-trust credentials — All secrets live in AWS Secrets Manager, never in source control
To get started, clone the repository and deploy the CDK stack. For more information about AWS DevOps Agent, see the AWS DevOps Agent documentation in the Amazon Q Developer User Guide.
About the author
Godwin Sahayaraj Vincent is an Enterprise Solutions Architect at AWS specializing in AI/ML workloads. He works with customers to design and implement scalable, cost-effective solutions on AWS.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article