I thought auto scaling was the responsible thing to do. Instead, it turned a routine product launch into a financial horror story.
Here's the short version: I set a maximum instance count of 1000 because the console asked for a number, and I thought "more headroom = safer." Then a background worker went rogue at 2 AM. My scaling policy saw high CPU and started adding instances to bring it down. But the workload was CPU-bound and load-balanced, so every new instance also got hammered. CPU stayed high. The policy added more. And more.
By 8 AM, I had 200 EC2 instances running—all doing useless work, all billing me in real time.
The Exact Mistake
I used target tracking on CPU. That works fine for normal traffic, but it's useless when a single broken endpoint is pegging every core. Scaling out doesn't fix bad code; it just multiplies the damage. My max of 1000 meant there was no ceiling. If I had set max to 10, the policy would have stopped after a few instances, and my bill would have been a fraction of the ₹1.2 lakh I ended up with.
But the real mistake was thinking "max is just a safety net." It's not. It's a cost ceiling. I had essentially authorized AWS to spend up to 1000 instances' worth of money without asking me again.
The 8 AM Panic
My phone buzzed with a budget alert—but it was a forecast alert, not a real-time one. By the time it fired, the damage was already done. I opened the EC2 console and saw 200 instances. My first thought: "Did we get hacked?" Then I saw the auto scaling group name. Nope. It was me.
I suspended scaling, set desired capacity back to 2, and terminated the extra instances. The background worker was fixed an hour later. The bill? ₹1.2 lakh. For a small startup, that hurt. But it could have been ten times worse if I hadn't had any alerts at all.
What I Do Now
- Max capacity = actual peak + 20%, not 1000. If I need more, I'll deliberately raise it.
- Real-time billing alerts, not daily forecasts. Hard stop at ₹5,000.
- Never scale on CPU alone for CPU-bound work. I use request count or latency now.
- Rate limit every expensive endpoint. No anonymous traffic gets to burn CPU.
The Takeaway
Auto scaling is powerful, but it can also automate your bankruptcy. A maximum of 1000 isn't a safety net—it's a blank cheque. Next time you configure an ASG, ask yourself: "If this goes wrong, what's the worst bill I'm willing to explain to my team?"
For me, that number is now 10. Not 1000.
Loading article