Automating Incident Response for AWS ECS with AI
Tired of slow, manual investigations when your ECS services hit 5XX errors? Discover the open-source aws-ecs-incident-response-framework—a Terraform module that automates incident detection, context gathering, and AI-driven diagnosis for AWS ECS clusters behind ALBs. Let's dig in!
Overview
Modern cloud-native applications demand fast, reliable incident response. If you’re running workloads on Amazon ECS behind an Application Load Balancer (ALB), you know how critical it is to detect, diagnose, and resolve issues quickly, especially when 5XX errors spike. Manual investigation is slow and error-prone. That’s why I built the
aws-ecs-incident-response-framework : an open-source Terraform module that automates incident detection, context gathering, AI-powered diagnosis, and notification, turning raw alarms into actionable insights.In this article, I’ll walk you through the motivation, architecture, and how to get started, so you can supercharge your ECS incident response with automation and generative AI.
Why Automate Incident Response on ECS?
Enabling observability features like CloudWatch Container Insights and X-Ray on your ECS cluster is a great start. But when an incident occurs, you’re still left sifting through logs and metrics, piecing together the root cause. This process can take precious time, especially in complex environments.
With this framework, you can:
- Automate alarm-driven investigations: When the ALB’s 5XX error count crosses a threshold, the system triggers a Lambda function to collect logs, metrics, and traces.
- Leverage generative AI for diagnosis: The Lambda invokes a Bedrock foundation model (e.g., Anthropic Claude) to analyze the data and generate a concise, actionable incident report.
- Accelerate response and reduce downtime: Teams are notified instantly with a summary and recommendations, so they can focus on remediation, not manual triage.
Architecture Overview
Here’s how it works:
- CloudWatch Alarm monitors the ALB’s 5XX error count.
- When the threshold is breached, the alarm triggers an SNS Topic.
- An AWS Lambda Function is invoked, which:
- Gathers relevant ECS logs, metrics, and traces.
- Calls the configured Bedrock AI model for analysis.
- Publishes a summary report to another SNS topic (for email, Slack, or other integrations).
- S3 Bucket stores analysis results for auditing and further investigation.
Getting Started
Prerequisites
- AWS account with permissions to create IAM roles, policies, CloudWatch, SNS, Lambda, and Bedrock resources.
- ECS services with logging and X-Ray enabled.
- AWS Bedrock enabled in your region, with access to your preferred model (e.g., Anthropic Claude).
- Network connectivity from Lambda/CloudWatch/X-Ray to your ECS cluster. For air-gapped environments, configure VPC endpoints as needed.
Example Terraform Usage
module "ecs_incident_response" {
source = "github.com/mrphuongbn/aws-ecs-incident-response-framework"
name_prefix = "prodapp"
alb_name = "app/prodapp-alb/50abc123def"
error_threshold = 50
evaluation_periods = 5
period = 60
bedrock_model_id = "anthropic.claude-3-sonnet-20240229-v1:0"
ecs_cluster_name = "prodapp-ecs-cluster"
ecs_service_names = ["service1", "service2", "service3"] # or empty for all services
# Optional: email_receiver, existing SNS ARNs, etc.
}
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article