The FinOps Agent That Knows When Not to Act
A rightsizing agent should earn the right to act alone. I tested a simple evidence gate on 200 instances with known answers: 71 wrong downsizes became 4.
Every FinOps agent demo ends the same way: the agent finds idle instances, proposes smaller ones and shows a big monthly saving. The question nobody asks on stage is how many of those downsizes would have taken production down.
I built a synthetic fleet of 200 EC2 instances where I know the answer for every one of them, and compared two ways of letting an agent act. A threshold rule on p95 CPU downsized 152 instances, and 71 of them would have run above 90% CPU in the next 30 days. An evidence gate that is allowed to say "I am not sure" downsized 65, got 4 wrong, and handed 57 to a person with the evidence attached. With a 93-day lookback, it got none wrong.
The problem: savings you can see, risk you cannot
A rightsizing decision looks like a pick from a short list: downsize, keep, or upsize. That makes it a perfect job for an agent, and a dangerous one, because the agent writes an unsure recommendation the same way it writes a sure one.
The usual safety net is a threshold. If p95 CPU over the last 14 days is below 40%, the smaller instance would sit below 80%, so downsize. Fourteen days is the default lookback of AWS Compute Optimizer, so it is a fair starting point.
The trouble is that p95 means "ignore the busiest 5% of the time". A nightly report that runs for 20 minutes is 1.4% of the day. A weekly batch is less. A month-end close may not be in the window at all. Those are exactly the workloads that break when you halve their CPUs.
The idea: one bad day blocks automation
The gate does not look at percentiles. It looks at days. For each day in the history it asks one yes/no question: would the smaller instance have peaked above 90% on this day?
Then it turns the answer into a confidence with the oldest trick in statistics, Laplace's rule of succession: (safe days + 1) / (days + 2). Fourteen safe days out of fourteen gives 0.94. Three safe days on a brand-new instance gives 0.80, which is not enough, no matter how idle those three days were.
1
2
3
4
5
6
safe = sum(2 * day_peak < 90 for day_peak in daily_peaks)
confidence = (safe + 1) / (len(daily_peaks) + 2)
if confidence >= 0.90 and safe == len(daily_peaks): action = "downsize"
elif confidence < 0.50: action = "keep"
else: action = "defer" # to a person, with the evidenceThat is the whole gate. The agent can still reason, explain and propose. It just cannot act unless every day in the history agrees with it. When it defers, the ticket carries the confidence, the number of safe days and the day that failed, so the person reviewing it starts from evidence, not from a hunch.
A fleet with known answers
I did not have a labelled fleet, so I built one where the answer is known by construction. Each instance gets 14 observed days of 5-minute CPU and 30 hidden future days from the same workload. A downsize is wrong if the smaller instance would spend 15 minutes or more above 90% CPU in those 30 days.
The 200 instances mix seven workloads: 60 steady and idle, 40 steady and busy, 20 noisy at moderate load, 25 with a 20-minute nightly report, 20 with a Sunday batch, 15 with a two-hour month-end close, and 20 new instances with only 2 to 5 days of history. Pricing is the on-demand rate for m7g.xlarge in Frankfurt, $0.1955 an hour, so each correct downsize to m7g.large saves about $71 a month.
Results
| Threshold on p95 | Evidence gate, 14 days | Evidence gate, 93 days | |
|---|---|---|---|
| Downsized automatically | 152 | 65 | 61 |
| Wrong downsizes | 71 (47%) | 4 | 0 |
| Kept | 48 | 78 | 77 |
| Deferred to a person | 0 | 57 | 62 |
| Saving captured automatically | $5,780 / month | $4,353 / month | $4,353 / month |
| Saving waiting in the review queue | 0 | $1,427 / month | $1,427 / month |
| Saving the dashboard would claim | $10,846 / month | $4,353 / month | $4,353 / month |
I read four things from this.
The threshold rule is right about idle machines and wrong about everything interesting. It downsized all 60 idle instances correctly. It also downsized all 25 nightly reports, all 20 weekly batches and all 15 month-end jobs. Half its "savings" are incidents waiting for a date.
The gate threw nothing away. 81 instances in the fleet were truly safe to downsize. The gate took 61 of them on its own and put the other 20 in the review queue, mostly new instances that simply had not been around long enough. It never kept an instance that was safe to shrink.
The gate cannot defer what it never saw. Its 4 mistakes were month-end jobs whose month-end fell outside the 14-day window. Every day it looked at was clean, so it acted. With 93 days of history, the extended lookback Compute Optimizer offers with enhanced infrastructure metrics, those jobs moved to the review queue and the mistakes went to zero.
More data did not fix the threshold rule. With 93 days it got 72 downsizes wrong instead of 71. A percentile averages a rare peak away, and a longer window only gives it more quiet days to average with. Longer history helps the gate because the gate counts bad days instead of diluting them.
The decision I would argue for
I would rather have a FinOps agent that captures 75% of the savings with 4 incidents than one that claims 100% of them with 71. The remaining 25% is not lost; it sits in a queue with the evidence attached, and a person clears it in the time it takes to read one chart.
The honest part of this test is its limits. The fleet is synthetic, CPU is the only signal, and real workloads have memory, network and seasonal peaks that no 93-day window sees, such as a yearly audit. The gate does not solve those. It makes sure the agent says so instead of acting.
What I would do next
Add memory and network to the same day-by-day test, since a downsize halves those too. Feed the gate's deferrals back as labels, so the queue becomes training data for the next version. And keep the rule that matters: the agent may explain anything, but it may only act on what every day in the history supports.
Try it
The script, the fleet generator and the raw results are in CODE . It runs in a few seconds on a laptop with Python and NumPy, and the seed is fixed, so you get the same 71, 4 and 0.
A FinOps agent earns autonomy the same way a new colleague does: by saying "I am not sure" before it touches production. One bad day in the history should be enough to stop it.
Sources
- AWS Compute Optimizer, Enhanced infrastructure metrics (14-day default lookback, up to 93 days): https://docs.aws.amazon.com/compute-optimizer/latest/ug/enhanced-infrastructure-metrics.html
- Amazon EC2 m7g.xlarge on-demand price in eu-central-1 ($0.1955/hour): https://www.doit.com/compute/spot/eu-central-1/m7g.xlarge
- Amazon EC2 On-Demand pricing: https://aws.amazon.com/ec2/pricing/on-demand/
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article