AWS Builder Center

Postmortem: How a Misconfigured AWS Auto Scaling Group Warm-Up Procedure Took Down Our Entire Web Fleet

Misconfigured AWS Auto Scaling Group Warm-Up

Cloud Engineer
Incident Summary
================ On 2023-02-15, between 14:45 UTC and 15:08 UTC, our web fleet experienced a critical outage due to a misconfigured AWS Auto Scaling Group warm-up procedure. The severity of the incident was classified as SEV-1, with 847 API requests failing and payment processing being unavailable for 23 minutes. The affected services included our primary web application, user authentication, and payment processing. The user impact was significant, with approximately 12,000 users experiencing errors during the outage.
Timeline
14:45 UTC — The first alert was triggered by our Datadog monitoring system, indicating a spike in 5xx errors from our ELB (Elastic Load Balancer). 14:46 UTC — The on-call engineer began investigating and initially suspected a database connection issue, as the error messages mentioned "connection refused" errors. 14:50 UTC — The engineer checked the database connection pool and found no issues, then moved on to check the application logs for any clues. 14:55 UTC — The engineer noticed a pattern of "503 Service Unavailable" errors from the ELB, which led to a review of the ELB configuration and the associated Auto Scaling Group (ASG). 15:00 UTC — The engineer discovered that the ASG had been scaled up to 50 instances, which was significantly higher than the usual 10-15 instances. 15:02 UTC — The engineer realized that the ASG warm-up procedure had been misconfigured, causing the ASG to scale up too aggressively. 15:04 UTC — The engineer applied a temporary fix by reducing the ASG's maximum size to 20 instances and adjusting the warm-up procedure. 15:06 UTC — The error rate began to decrease, and the web fleet started to recover. 15:08 UTC — The incident was marked as resolved, with all services returning to normal.
Root Cause Analysis The root cause of the incident was a misconfigured AWS Auto Scaling Group warm-up procedure. Specifically, the aws_autoscaling_schedule resource had been configured with an incorrect max_size value, causing the ASG to scale up too aggressively:
-----------------------------------------------------------------
resource "aws_autoscaling_schedule" "example" { scheduled_action_name = "example" min_size = 0 max_size = 50 // incorrect value, should be 20 desired_capacity = 10 recurrence = "0 12 * * *" }
-----------------------------------------------------------------
Additionally, the aws_autoscaling_group resource had been configured with an incorrect warm_up value, which caused the ASG to scale up too quickly:
-----------------------------------------------------------------
resource "aws_autoscaling_group" "example" { name = "example" max_size = 50 // incorrect value, should be 20 min_size = 0 desired_capacity = 10 health_check_grace_period = 300 warm_up = 300 // incorrect value, should be 600 }
-----------------------------------------------------------------
The combination of these two misconfigurations caused the ASG to scale up too aggressively, leading to a large number of instances being launched in a short period. This, in turn, caused the ELB to become overwhelmed, resulting in the 503 Service Unavailable errors.
Contributing Factors
Insufficient monitoring: While we had monitoring in place for the ELB and ASG, we did not have sufficient metrics to detect the aggressive scaling behavior. Lack of automation testing: We did not have automated tests in place to validate the ASG configuration and warm-up procedure. Inadequate documentation: The ASG configuration and warm-up procedure were not well-documented, making it difficult for the on-call engineer to understand the intended behavior. Recent changes to the ASG configuration: A recent change to the ASG configuration had introduced the misconfiguration, which was not caught by our review process. Limited visibility into AWS Auto Scaling: Our monitoring and logging tools did not provide sufficient visibility into the AWS Auto Scaling service, making it difficult to diagnose the issue.
What Went Well
Datadog monitoring: Our Datadog monitoring system detected the spike in 5xx errors and alerted the on-call engineer. Runbook execution: The on-call engineer followed our runbook for investigating ELB errors, which helped to identify the issue with the ASG. Collaboration between teams: The on-call engineer was able to collaborate with other teams, including the DevOps team, to resolve the issue quickly.
Action Items
Action
Owner
Due
Priority
Update ASG configuration to correct values
DevOps Engineer
2023-02-22
High
Implement automated testing for ASG configuration
DevOps Engineer
2023-03-01
Medium
Improve monitoring and logging for AWS Auto Scaling
Cloud Engineer
2023-03-08
Medium
Document ASG configuration and warm-up procedure
DevOps Engineer
2023-02-22
Low
Review and update runbook for investigating ELB errors
On-call Engineer
2023-02-25
Low
Lessons Learned
  1. The importance of automated testing: Automated testing can help catch misconfigurations and ensure that changes to infrastructure configurations are valid.
  1. The need for sufficient monitoring and logging: Sufficient monitoring and logging can help detect issues quickly and provide visibility into the behavior of cloud services.
  1. The value of collaboration and documentation: Collaboration between teams and good documentation can help resolve issues quickly and ensure that knowledge is shared across the organization.
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article