The First Copy Is Free: How We Cut CloudTrail Costs by 90% After Finding Trails Nobody Remembered
A story about duplicate trails, a "zombie" trail nobody could explain, a well-meant compliance trail, and why cleanup alone didn't keep the bill down.
TL;DR
- One AWS account was spending triple digits per day on CloudTrail. Nothing was broken and no alert had fired.
- Root cause: two trails delivering the same management events to the same bucket, plus broadly scoped S3 data events and a legacy trail with no owner.
- After the fix, daily spend dropped to single digits, a reduction of more than 90%.
- A few weeks later the cost jumped again in other accounts, because a well-meaning compliance trail duplicated management events.
- What made the fix durable was an architecture change: an organization-level trail.
- A copy-paste audit checklist is at the end.
The symptom: a bill that only moved one way
There was no incident, no failed deploy, and no new workload. A cost-anomaly view showed one account's CloudTrail line sitting in the triple digits per day and refusing to come down.
That kind of cost is hard to catch because it isn't a spike. It's a plateau, and plateaus get treated as "just what this account costs."
A quick refresher: what CloudTrail actually charges for
Most of the surprise comes from one phrase on the pricing page: "after the first free copy."
| Event type | Pricing (at time of writing) | Free? |
|---|---|---|
| Management events, first copy per region | $0 | Yes |
| Management events, additional copies | $2.00 per 100,000 events | No |
| Data events (S3 object-level, Lambda invoke, etc.) | $0.10 per 100,000 events | No, charged from the first event |
| CloudTrail Insights | $0.35 per 100,000 management events analyzed, per insight type | No |
Always check the CloudTrail pricing page for current rates.
Here is what that means in practice. Take a hypothetical account that produces 5 million management events a month:
- Trail #1 costs $0.
- Trail #2, logging the same events, costs $100.
- Trail #3, perhaps added by another team, costs another $100.
Nothing in the console warns you. A second trail looks exactly like the first.
Pricing mechanism
The investigation: five trails, three questions
I audited the account without changing anything, using a few read-only CLI calls:
bash
1
2
3
4
5
6
7
8
9
10
11
12
13
14
# 1. Inventory every trail, including shadow trails from other regions
aws cloudtrail describe-trails --include-shadow-trails
# 2. What does each trail actually log?
aws cloudtrail get-event-selectors --trail-name <trail-name>
# 3. Is it logging, and since when?
aws cloudtrail get-trail-status --name <trail-name>
# 4. Who owns it?
aws cloudtrail list-tags --resource-id-list <trail-arn>
# 5. Is anyone managing log retention?
aws s3api get-bucket-lifecycle-configuration --bucket <log-bucket>Then I opened Cost Explorer, filtered to CloudTrail, and grouped by usage type. The usage types explain a lot on their own:
PaidEventsRecorded: additional copies of management events.DataEventsRecorded: data events.FreeEventsRecorded: the free first copy.
Seeing
PaidEventsRecorded next to a big number means someone is paying for duplicate copies.What we found
Five trails, each with a different story:
| Trail | What it did | Verdict |
|---|---|---|
| Trail A (single-region) | Management events plus all-bucket S3 data events, with CloudWatch Logs delivery | Duplicate and too broad, remove |
| Trail B (second region) | Identical configuration to Trail A | Duplicate and too broad, remove |
| Central security trail | Multi-region, management events only, KMS-encrypted, feeding the SIEM | Keep, this was our baseline |
| Purpose-built compliance trail | Data events for one specific bucket | Keep, a good example of scoped logging |
| Legacy trail | Had been running for years, no tags, no known owner | Zombie, confirm nobody uses it, then retire |
Trails A and B were logging management events that the central trail already captured, and writing them to the same shared bucket. That bucket had no lifecycle policy, so storage kept growing as well.
The data-event scope made it worse. Logging S3 data events across all buckets is priced per event with no free copy.
Audit findings
The fix, in three passes
Pass 1: Remove the duplicates. After confirming ownership and getting approval, we deleted the two duplicate trails. This stopped most of the bleeding.
Pass 2: Trim the scope.
- Data events: we scoped data events to only what justified object-level auditing and turned them off elsewhere.
- Management events: we kept them, because regional management logging was needed and, after checking, the cost was negligible. We measured that rather than assuming it.
- The zombie: nobody could explain why the legacy trail existed, and it wasn't being used, so I removed it.
A word of caution on data events: scoping them down was right for our environment. Yours may be different. If you hold regulated data, customer data, or anything an auditor will ask about, keep data events on for those critical buckets and scope the selectors tightly. The goal is "log what matters," not "log nothing."
Pass 3: Close the door. I'll cover this after the results, because it turned out to be the pass that mattered most.
The result
| Before | After | |
|---|---|---|
| Daily CloudTrail spend | Triple digits, climbing | Single digits, flat |
| Trails in the account | 5 (2 duplicates, 1 zombie) | Only the ones with a purpose and an owner |
| Data event scope | All buckets | Only what justified object-level auditing |
| Overall reduction | More than 90% |
Results
The plot twist: it came back
A few weeks later, the same cost pattern appeared in other accounts, with a sudden jump in daily spend.
This time nobody had been careless. A team supporting a compliance initiative had enabled an additional trail with management events turned on. That is a perfectly legitimate reason. But management events were already being delivered by an existing trail, and only the first copy is free. The new trail created a second, billed copy of every management event, and management events are high volume.
The cleanup had fixed the symptom, but nothing stopped the next trail from being created, for good reasons, by someone who couldn't see the cost.
AWS documents the mechanic: if the management account has an organization trail that delivers management events, the same events delivered by trails created in member accounts are charged as additional copies. In a multi-account organization, duplication is easy to create by accident.
Recurrence / second wave
Pass 3: an organization trail as the single source of truth
The durable fix was architectural:
- One organization-level trail, created from the management account, covering every account and region.
- The organization trail is automatically applied to member accounts, giving the organization a centrally managed baseline for management-event logging.
- Teams with compliance or audit needs can build on that centrally managed baseline instead of creating their own trails.
- Data events are added deliberately and narrowly, never as a blanket setting.
With a single source of truth, teams no longer need to create separate trails for baseline management-event logging, which removes one of the most common ways duplication creeps in.
For extra protection, I'd also recommend a Service Control Policy that restricts
cloudtrail:CreateTrail in member accounts to the roles that genuinely need it. An org trail removes the need for duplicate trails, and the SCP prevents someone from recreating the problem by accident.Everything before Pass 3 was cleanup. Pass 3 was the architecture change.
Architectural solution
Lessons you can apply to your own organization
- "Free" has a boundary. Only the first copy of management events per region is free. A second trail is a second bill.
- Quiet doesn't mean healthy. Nothing alerted and nothing failed. Duplication is invisible until you audit for it.
- Compliance trails are a common culprit. Audit and compliance work often leads to new trails. Talk to your security and compliance teams before they create one, and point them to the central trail.
- Data events are the other lever. There is no free copy, so broad scope is expensive. Log what you need, not everything, and protect your critical buckets.
- Check for zombies. A trail with no tags and no owner is a cost and a compliance question at the same time.
- Tag trails and set bucket lifecycle rules. Ownership and retention should not be afterthoughts.
- Fix the architecture, not just the symptom. In a multi-account organization, an org trail plus guardrails beats periodic cleanup.
A 10-minute CloudTrail cost audit checklist
- List all trails in every region with
describe-trails --include-shadow-trails. - Identify trails that log the same management events.
- Check each trail's event selectors for broad S3 data-event scope.
- Verify
IsLoggingand the last delivery time for each trail. - Check that every trail has an owner tag.
- In Cost Explorer, group CloudTrail by usage type and by linked account, and look for
PaidEventsRecordedandDataEventsRecorded. - Ask your security and compliance teams whether they have added any trails recently.
- Confirm S3 lifecycle rules exist on log buckets.
- Check whether Insights or CloudWatch Logs delivery is enabled where it isn't needed.
- In a multi-account organization, evaluate an organization trail and SCP guardrails, and delete redundant member-account trails first.
Practical checklist
Closing thought
CloudTrail is essential and I'm not suggesting anyone log less than they need. The point is that the first copy is free, and everything after that is a design decision, not a discount.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article