Watching my Auto Scaling group replace the same instance over and over

To practice high availability, I set up the standard thing: a launch template, an Auto Scaling group across two Availability Zones and an Application Load Balancer in front. The app is a small Flask API, and user data installs its dependencies and starts it when the instance boots.

I refreshed the Auto Scaling activity history a few minutes later and watched it launch an instance, mark it unhealthy, terminate it and launch another one. Over and over. It was like watching a very expensive loading spinner.

There were two separate problems.

The health check path. The target group checks / by default, and my API doesn't serve anything at /, so every check got a 404. The default success code is 200, so the load balancer marked every instance unhealthy. I added a /health route that returns 200 and pointed the health check at it.

The grace period. Even after that, some instances still got replaced. I had turned on Elastic Load Balancing health checks for the Auto Scaling group, which you want, because otherwise it only looks at EC2 status checks and never notices the app is broken. But my user data took a few minutes to install packages and start Flask, and I had set the health check grace period  way shorter than that. So the group was checking health before the app even existed, seeing failures and replacing the instance, and the new one went through the exact same thing.

I timed a fresh instance from launch to the first successful /health response and set the grace period comfortably above that. The loop stopped.

The longer-term fix is baking an AMI with the dependencies already installed so boot takes seconds instead of minutes. That's what I'm working on next.

The thing that made this debuggable was the activity history. Every termination has a reason written next to it, and just reading them in order told the whole story. I wish I'd looked there first instead of guessing for twenty minutes.