Your AWS Bill Is a Codebase — Treat It Like One
Agentic AI workloads can rack up unexpected AWS costs in hours. This post shows how to treat your bill like code — with version-controlled CloudFormation alarms, a four-tier escalation pattern, and an automated kill switch that stops tagged resources when thresholds breach. Includes a deployable template that costs under $1/month
Your AWS Bill Is a Codebase
I checked my AWS bill on a Monday morning and found $258 in Bedrock charges from a single agent that ran over the weekend. An autonomous task — running every ten minutes, accumulating context with each call — had quietly turned a $12 experiment into a $3,875/month projection. Nobody paged me. No alarm fired. I found it by accident.
You know that feeling when you return from holiday abroad and discover your phone bill is three figures because data roaming was on? Same shock, same preventability, same "why didn't something stop this?"
The frustrating part isn't the money. It's that you knew this could happen. You just didn't have a system to catch it before it did.
The Real Problem: Cost Control Is an Afterthought
Most AWS accounts treat cost monitoring the way most codebases treat logging — bolted on after something breaks. You set a single budget alert at some round number, get an email you ignore because it fires every month, and hope for the best.
Here's what that looks like in practice:
- One alarm, one threshold — a single budget alert at $200/month tells you nothing about daily anomalies
- No escalation path — the same email goes to the same inbox whether you're $1 over or $500 over
- No automatic response — by the time a human reads the alert, the damage is done
- No visibility into what's burning — Bedrock tokens, idle EKS clusters, and orphaned EBS volumes all look the same in a monthly total
Before this project, my account had a single monthly budget alarm and a hope-for-the-best attitude. The alarm fired so regularly it became noise — I stopped reading it.
This is the gap between "we monitor costs" and "we control costs." Monitoring tells you what happened. Control prevents it from happening.
How Fast Can an Autonomous Agent Burn Money?
Let me give you a concrete example. Take an autonomous agent built with Claude Sonnet on Amazon Bedrock — a common pattern for scheduled tasks like monitoring, reporting, or maintenance automation. Configure it to run every 10 minutes. Give it memory so it can learn from previous runs. But forget to cap that memory.
Here's what happens over 48 hours:
| Metric | Value |
|---|---|
| Invocations | 288 (6/hour × 48 hours) |
| Final context size | 578,000 tokens |
| Total input tokens | 83.2 million |
| Total output tokens | 576,000 |
| Total cost (2 days) | $258.34 |
| Projected 30-day cost | $3,875 |
Now the same agent with a 4,000-token memory cap:
| Metric | Value |
|---|---|
| Total input tokens | 1.15 million |
| Total cost (2 days) | $12.10 |
| Savings from one config line | $246.24 (95%) |
The difference between a $12 experiment and a $3,875 disaster is a single parameter:
max_context_tokens. And the default? No limit.The math: 288 calls × average 289,000 input tokens per call = 83.2M input tokens × $3/M = $249.70, plus 576K output tokens × $15/M = $8.64. Total: $258.34. With a 4K cap: 288 × 4,000 = 1.15M tokens × $3/M = $3.46 input. Same output = $12.10 total.
Treating Your Bill Like a Codebase
The shift that changed everything for me was simple: I found that treating my AWS bill the same way I treat application code made the problem disappear. Version-controlled configuration. Automated testing. Graduated responses. Rollback capability.
If your application has health checks, circuit breakers, and auto-scaling, why doesn't your billing?
The system I built has three layers:
- Multi-tier alarms — four escalation levels, each with a different response
- Composite alarms — aggregate signals that catch patterns individual alarms miss
- Kill switches — automated remediation that stops the bleeding without human intervention
All of it defined in CloudFormation. All of it config-driven. Change a threshold in
settings.json, redeploy, done.Architecture
Four-tier CloudWatch alarm system with automated kill switch — from email nudge to resource shutdown in a single CloudFormation template.
Implementation: The Four-Tier Pattern
Tier 1: The Nudge ($20/day)
The first tier is informational. It fires when daily spend rate crosses $20 — low enough to catch anomalies early, high enough to avoid noise on normal days.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
DailySpendWarning:
Type: AWS::CloudWatch::Alarm
Properties:
AlarmName: !Sub '${ProjectPrefix}-daily-spend-tier1-warning'
AlarmDescription: !Sub 'Daily spend rate exceeded $${DailyBudgetTier1}'
Metrics:
- Id: charges
MetricStat:
Metric:
Namespace: AWS/Billing
MetricName: EstimatedCharges
Dimensions:
- Name: Currency
Value: USD
Period: 21600 # 6 hours
Stat: Maximum
- Id: daily_rate
Expression: "RATE(charges) * 86400"
Label: "Estimated Daily Spend Rate"
EvaluationPeriods: 1
Threshold: !Ref DailyBudgetTier1
ComparisonOperator: GreaterThanThreshold
AlarmActions:
- !Ref AlertTopicThe
RATE() function computes the per-second change in cumulative charges, then multiplies by 86,400 to project a daily rate. This catches spend spikes within hours — not at end-of-month when it's too late.Tier 2: The Alert ($50/day)
Tier 2 means something unusual is happening. Maybe you spun up a larger instance for testing, or a Bedrock experiment is running longer than planned. This one deserves a look today.
The CloudFormation is identical to Tier 1 except for the threshold. That's the beauty of parameterized templates — the escalation logic lives in configuration, not code.
Tier 3: The Critical ($100/day)
At Tier 3, the composite alarm kicks in. A composite alarm aggregates multiple signals into a single "something is seriously wrong" indicator:
1
2
3
4
5
6
7
8
9
10
CompositeAlarm:
Type: AWS::CloudWatch::CompositeAlarm
Properties:
AlarmName: !Sub '${ProjectPrefix}-composite-any-critical'
AlarmDescription: Fires when any tier 3+ alarm is active
AlarmRule: !Sub >
ALARM("${DailySpendCritical}") OR
ALARM("${DailySpendKillSwitch}")
AlarmActions:
- !Ref AlertTopicWhy a composite alarm? Because in production, you'll add service-specific alarms — Bedrock token spend, EKS cluster costs, Lambda concurrency spikes. The composite alarm catches any of them without requiring you to subscribe to each one individually.
Tier 4: The Kill Switch ($500/day)
This is where it gets interesting. Tier 4 doesn't just notify — it acts. When daily spend crosses $500, a Lambda function automatically stops tagged resources:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
def handler(event, context):
"""Kill switch — stop tagged resources when spend exceeds threshold."""
stopped = []
# Stop EC2 instances tagged with this project
ec2 = boto3.client('ec2')
instances = ec2.describe_instances(Filters=[
{'Name': 'tag:Project', 'Values': [PREFIX]},
{'Name': 'instance-state-name', 'Values': ['running']}
])
for r in instances['Reservations']:
for i in r['Instances']:
ec2.stop_instances(InstanceIds=[i['InstanceId']])
stopped.append(f"EC2:{i['InstanceId']}")
# Scale down ECS services tagged with this project
ecs = boto3.client('ecs')
for cluster_arn in ecs.list_clusters()['clusterArns']:
for svc_arn in ecs.list_services(cluster=cluster_arn)['serviceArns']:
svc = ecs.describe_services(
cluster=cluster_arn, services=[svc_arn]
)['services'][0]
tags = {t['key']: t['value'] for t in svc.get('tags', [])}
if tags.get('Project') == PREFIX and svc['desiredCount'] > 0:
ecs.update_service(
cluster=cluster_arn, service=svc_arn, desiredCount=0
)
stopped.append(f"ECS:{svc['serviceName']}")
return {'stopped': stopped, 'count': len(stopped)}For Bedrock specifically — which is where agentic costs accumulate fastest — the kill switch also revokes invoke permissions:
1
2
3
4
5
6
7
8
9
10
11
# Block Bedrock invocations by attaching a deny policy
iam = boto3.client('iam')
iam.put_role_policy(
RoleName=f'{PREFIX}-bedrock-role',
PolicyName='emergency-deny-bedrock',
PolicyDocument=json.dumps({
"Version": "2012-10-17",
"Statement": [{"Effect": "Deny", "Action": "bedrock:InvokeModel*", "Resource": "*"}]
})
)
stopped.append("Bedrock:invoke-denied")Three design decisions matter here:
- Tag-scoped — the kill switch only touches resources tagged with your project prefix. It won't accidentally stop your production database.
- SQS-buffered — the alarm publishes to SNS, which delivers to SQS, which triggers Lambda. This decouples the alarm from the action and gives you a dead-letter queue for failures.
- Concurrency of 1 —
ReservedConcurrentExecutions: 1prevents the kill switch from running in parallel. One execution, one clean sweep.
And here's where the concurrency setting matters beyond the kill switch: Lambda functions without reserved concurrency have no upper bound. If your alarm fires repeatedly (as it will during a sustained cost event), you could trigger dozens of parallel kill switch executions. Setting concurrency to 1 is a single line that prevents a remediation loop from becoming its own runaway.
The Mistakes That Built This System
I didn't start with four tiers and a kill switch. I started with a single CloudWatch alarm and a bad architecture. We built this system one incident at a time.
The most expensive lesson was an EventBridge Scheduler cleanup failure. I built an automation that created one-time schedules for resource lifecycle management — "delete this in 24 hours." The creation worked perfectly. The cleanup didn't. Over a weekend, the system created 112,382 individual schedules, none of which were ever deleted. The bill for that oversight: $887.80 in EventBridge charges alone.
The root cause was embarrassingly simple: I had a
create_schedule() call but no corresponding delete_schedule() in any code path — not on success, not on failure, not on timeout. Resources accumulated silently.That incident taught me three rules that are now baked into the system:
- Event-driven, not polling — CloudWatch alarms fire on state transitions (OK → ALARM), not on every evaluation. A scheduled Lambda that checks "is cost > threshold?" will fire every single execution while the condition is true. An alarm fires once.
- Every create needs a delete — if your code creates a resource, it must delete that resource in the success path, the failure path, and the timeout path. The scheduler incident created 112,382 EventBridge schedules because nothing cleaned them up.
- No Lambda recursive patterns — a Lambda function must never publish to an SNS topic it's subscribed to. I learned this when a notification handler subscribed to its own alert topic produced 6,649 invocations in 2 hours before I noticed. The fix was one line: remove the Lambda subscription, use email-only for alert topics.
These aren't theoretical concerns. They're scars.
Results
Before: one alarm, monthly cadence, $887.80 surprise. After: four tiers, 6-hour detection, $15/month to run.
After eight weeks of iteration, here's where the system landed:
- Monthly cost: ~$15/month for the entire monitoring stack (CloudWatch alarms, SNS, Lambda, DynamoDB)
- ROI: 2,000%+ — the system costs less than a single hour of the runaway spend it prevents
- Alert activity in 90 days: 14 Tier 1 (awareness), 3 Tier 2 (investigated same-day), 1 Tier 3 (caught a runaway Bedrock session projecting to $3,600/month — stopped within 6 hours)
- 91 CloudFormation resources managed from a single template and one config file
- 39 automated tests covering alarm deployment, kill switch behavior, failure modes, and rollback
- Caught $258 in projected runaway spend from an agentic workload within 6 hours of anomaly — early detection saved an estimated $3,600 over the month
The entire system deploys in under 3 minutes:
1
2
3
4
5
aws cloudformation deploy \
--template-file cost-control.yaml \
--stack-name cost-control \
--capabilities CAPABILITY_NAMED_IAM \
--parameter-overrides NotificationEmail=you@example.comAnd tears down just as fast:
1
aws cloudformation delete-stack --stack-name cost-controlNo orphaned resources. No manual cleanup. That's what "treating your bill like a codebase" means in practice.
Advanced Extensions
Once the base system is running, two additions make it significantly more powerful.
First, add Bedrock token monitoring as a custom metric. Publish per-model token counts from your agent's invocation logs to CloudWatch, then alarm on those directly. The base billing metric won't show you which model or which conversation is burning — custom metrics will. At $3 per million input tokens, 83 million tokens over a weekend adds up fast, and standard alarms can't break down by model.
Second, set Lambda concurrency limits on every function from day one. The default is no limit. A misconfigured trigger can spawn hundreds of parallel executions before the first CloudWatch data point arrives. One line in your template —
ReservedConcurrentExecutions: 10 — would have prevented two of the three incidents described above. Treat it as mandatory, not optional.Try It Yourself
The companion CloudFormation template is available on GitHub. It deploys the four-tier alarm pattern with a kill switch in about 3 minutes, costs under $1/month, and tears down cleanly.
The template is live at the link above. Clone it, adjust the thresholds in the parameters, and deploy. In the meantime, the CloudFormation snippets in this post are complete enough to assemble your own.
Clone the repo. Point your coding assistant at it and ask: "What parts of this would protect my account?" The patterns are modular — take the tiers you need, adjust the thresholds to match your spending, and deploy.
The next time you check your AWS bill on a Monday morning, you'll already know what it says.
Chris Church is a Solutions Architect at AWS, working with enterprise customers on cloud architecture and cost optimization. He builds tools that make cloud operations less surprising.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article