AWS Builder Center

I Set Auto Scaling Max to 1000. Woke Up to 200 Instances and a ₹1.2 Lakh Bill.

A misconfigured auto scaling maximum and a rogue background worker turned a quiet Saturday into a 200-instance runaway. Here's the mistake, the panic, and the guardrails that now keep my AWS bill sane.

I thought auto scaling was the responsible thing to do. Instead, it turned a routine product launch into a financial horror story.
Here's the short version: I set a maximum instance count of 1000 because the console asked for a number, and I thought "more headroom = safer." Then a background worker went rogue at 2 AM. My scaling policy saw high CPU and started adding instances to bring it down. But the workload was CPU-bound and load-balanced, so every new instance also got hammered. CPU stayed high. The policy added more. And more.
By 8 AM, I had 200 EC2 instances running—all doing useless work, all billing me in real time.

The Exact Mistake

I used target tracking on CPU. That works fine for normal traffic, but it's useless when a single broken endpoint is pegging every core. Scaling out doesn't fix bad code; it just multiplies the damage. My max of 1000 meant there was no ceiling. If I had set max to 10, the policy would have stopped after a few instances, and my bill would have been a fraction of the ₹1.2 lakh I ended up with.
But the real mistake was thinking "max is just a safety net." It's not. It's a cost ceiling. I had essentially authorized AWS to spend up to 1000 instances' worth of money without asking me again.

The 8 AM Panic

My phone buzzed with a budget alert—but it was a forecast alert, not a real-time one. By the time it fired, the damage was already done. I opened the EC2 console and saw 200 instances. My first thought: "Did we get hacked?" Then I saw the auto scaling group name. Nope. It was me.
I suspended scaling, set desired capacity back to 2, and terminated the extra instances. The background worker was fixed an hour later. The bill? ₹1.2 lakh. For a small startup, that hurt. But it could have been ten times worse if I hadn't had any alerts at all.

What I Do Now

  • Max capacity = actual peak + 20%, not 1000. If I need more, I'll deliberately raise it.
  • Real-time billing alerts, not daily forecasts. Hard stop at ₹5,000.
  • Never scale on CPU alone for CPU-bound work. I use request count or latency now.
  • Rate limit every expensive endpoint. No anonymous traffic gets to burn CPU.

The Takeaway

Auto scaling is powerful, but it can also automate your bankruptcy. A maximum of 1000 isn't a safety net—it's a blank cheque. Next time you configure an ASG, ask yourself: "If this goes wrong, what's the worst bill I'm willing to explain to my team?"
For me, that number is now 10. Not 1000.
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article