AWS Builder Center
A $2.5B Estimate Taught Me to Distrust EstimatedCharges

A $2.5B Estimate Taught Me to Distrust EstimatedCharges

A CloudWatch billing alarm and a $2.5B estimate. What actually broke on July 17, and what I now check before automating on AWS billing data.

Series: AWS Cloud Operation (4 articles)

  1. 1
    A $2.5B Estimate Taught Me to Distrust EstimatedCharges This article
Cloud Engineer
On July 17, 2026, AWS customers started posting screenshots of the Billing and Cost Management console with numbers that looked like national budgets. One user whose actual usage came to $0.19 for the month was staring at an estimated charge of nearly $2.5 billion. The root cause, according to AWS, was a unit-pricing bug in the projection layer that powers Cost Explorer and the console's estimated charges. Actual invoices and payment processing were never touched — the bug lived entirely in the number that guesses what your month will look like, and for a few hours it guessed very, very wrong.
AWS fixed it fast. Root cause identified in about 90 minutes, estimated bill updates paused, corrected data backfilled by the next day. As incidents go, a clean one. But the part I kept turning over afterward wasn't the incident response. It was a much more boring question: what exactly is watching that number in my own accounts, and would it have noticed?

The metric nobody treats as untrusted input

CloudWatch billing alarms are built on the EstimatedCharges metric — the same projection layer that broke on July 17. Plenty of teams wire more than an alarm to it. Cost-threshold automation, Slack pings when spend crosses a line, and in more aggressive setups, auto-remediation that throttles or shuts down a service when the number looks scary.
I hadn't confirmed any reports of automation actually firing because of this incident, so I won't claim that happened. But the design gap is easy to see once you look for it. Any automation built on EstimatedCharges inherits every failure mode of that metric, and on July 17 the metric briefly asserted that a $0.19 account was spending billions. A false alarm is annoying. An automated "emergency cost response" tearing down production because a forecasting subsystem had a unit-pricing bug is a different kind of morning.
I went and checked my own setup after reading about the incident. I don't have anything destructive wired to billing thresholds, mostly Slack notifications, but I hadn't actually thought about what "trust" meant for that metric until this forced the question.

What I now check before automating on billing data

A few things came out of re-reading my own setup, roughly in order of how much effort each one takes.
Estimates and invoices are different sources of truth, and it's worth treating them that way explicitly. When a number looks wrong, the Invoices page and the Cost and Usage Report are the facts. Cost Explorer and the console's estimated charges are a forecast, and forecasts have bugs that facts don't.
The AWS Health Dashboard is the first place to look, not the last. I checked my own account balance before I checked Health Dashboard, which is backwards — a spike that correlates with an open incident is a data problem, not a spending problem, and that distinction changes what you should do next.
Anything automated that keys off EstimatedCharges deserves a sanity bound. An estimate that jumps four orders of magnitude in an hour is far more likely to be a bug than an actual bill, and a bound that catches "this number moved by 1000x since the last check" costs almost nothing to add.
And the one that matters most: if the response to a cost anomaly is destructive — stopping instances, deleting resources, scaling something to zero — it should require a human to click something. This is the same principle I keep coming back to when I write about giving AI agents autonomy: reversible actions can run on their own, destructive ones earn a human in the loop. It turns out that principle has nothing to do with AI specifically. It applies to any system that acts on a signal it never verified.

Where this leaves me

My own billing automation is genuinely low-stakes, so nothing here was an emergency fix. But it was a useful nudge to go look at a piece of infrastructure I'd basically stopped thinking about since the day I set it up. The metric had never lied to me before, so I'd quietly started treating "hasn't lied yet" as "won't lie." July 17 was a reminder that a forecast is still a forecast, no matter how many months it's been boringly correct.

Series: AWS Cloud Operation (4 articles)

  1. 1
    A $2.5B Estimate Taught Me to Distrust EstimatedCharges This article
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article