AWS Builder Center

CloudWatch Builder Series - Part 1 : Quick Win Playbook for Optimization

A practitioner's guide to reclaiming CloudWatch spend using log class selection, intelligent tiering, retention tuning, and hygiene automation. Hands-on CLI, audit scripts, and decision tables.

Series: CloudWatch Builder Series (3 articles)

  1. 2
    CloudWatch Builder Series - Part 1 : Quick Win Playbook for Optimization This article

Overview

Log ingestion and storage make up the majority of the CloudWatch bills. This guide covers four quick wins — each implementable in hours with no application architecture changes — that together reduce CloudWatch spend by 20–50%. (Figures based on ~500 GB/month ingestion. Scale linearly for larger workloads.)



1. Log Class Selection: Standard vs. Infrequent Access

Log ingestion is typically 60–70% of a CloudWatch bill. The Infrequent Access (IA) log class offers a 50% lower ingestion cost ($0.25/GB vs. $0.50/GB in us-east-1) for logs that don't require real-time streaming or subscription filters.
IA Capabilities (as of March 2026)
  • ✅ Logs Insights queries (same speed as Standard)
  • ✅ Cross-account log analytics
  • ✅ KMS encryption
  • ✅ Managed ingestion and storage
  • ✅ S3 export
  • ✅ Data Protection (Mar 2026)
  • ✅ OpenSearch PPL/SQL queries (Mar 2026)
  • ❌ Subscription filters (no streaming to Lambda/Kinesis/OpenSearch)
  • ❌ Metric filters (cannot extract metrics from log lines)
  • ❌ Live Tail
Decision Matrix
Log Type
Class
Rationale
EKS/ECS control plane logs
IA
Queried only during troubleshooting
VPC Flow Logs (if sent to CW)
IA
Typically archived for compliance
CloudTrail management events
IA
Audit/compliance, infrequent queries
Application debug/trace logs
IA
Only needed during incidents
Lambda logs (dev/staging)
IA
Low query frequency
Production application error logs
Standard
May need real-time alerts via metric filters
Security/WAF logs with real-time alerting
Standard
Requires subscription filters
Logs feeding real-time dashboards
Standard
Needs metric filter extraction
Implementation
# Create a new log group with IA class
aws logs create-log-group \
--log-group-name /eks/my-cluster/control-plane \
--log-group-class INFREQUENT_ACCESS

# Note: Existing log groups cannot change class.
# You must create a new IA log group and redirect log sources.

ROI Example — Team ingesting 500 GB/month of EKS control plane logs:

Standard
Infrequent Access
Savings
Ingestion cost
$250/month
$125/month
$125/month
Annual savings
—
—
$1,500/year
Multiply across 10+ clusters = $15,000+/year from this single change.
Savings: 50% on ingestion cost



2. Intelligent Tiering

Most log data is queried intensively for the first few days, then rarely touched. Intelligent Tiering automatically moves data to cheaper storage tiers based on access patterns — with zero change to your query experience.
How It Works
Tier
Trigger
Query Experience
Storage Cost
Standard (default)
All new data starts here
Full speed
Base rate
Infrequent Access
No access for 30 days
No degradation
Lower
Archive Instant Access
No access for 90 days
No degradation
Lowest
Key difference from S3 Intelligent Tiering: This is a billing-layer optimization only. No retrieval delay, no restore request, no retrieval fee. Queries run at the same speed regardless of tier. When you query older data, it automatically promotes back to Standard for 30 days.
Implementation
# Enable for your entire account in a region
aws logs put-storage-tier-policy \
--storage-tier INTELLIGENT_TIERING

# Or via Console: CloudWatch > Logs > Settings > Enable Intelligent Tiering

That's it — one API call, applies to all log groups in the account within that Region.
Why this matters:
  • Zero operational overhead — no lifecycle policies to manage
  • No query degradation
  • Set-and-forget — works on existing log groups with existing data
ROI Example — 2 TB stored logs, 80% older than 30 days, 60% older than 90 days:
Tier
Data
Rate ($/GB/month)
Monthly Cost
Standard (<30 days)
410 GB (20%)
$0.03
$12.29
Infrequent Access (30-90 days)
410 GB (20%)
$0.0125
$5.12
Archive Instant Access (>90 days)
1,229 GB (60%)
$0.004
$4.92
With tiering, the monthly cost drops to $22.32 — versus $61.44 without it. That's an annual savings of $469.44. For an enterprise scale (5–10 TB of stored logs), the annual savings would reach ~$2,350.
Savings: 63.7% on storage costs



3. Log Retention Policies

The default retention is "Never Expire" — every byte stored forever. Most teams have years of debug logs they'll never query again.
The Compounding Cost
Storage pricing: ~$0.03/GB/month (Standard class)
  • 100 GB/day × 365 days × $0.03/GB = $1,095/year for one year's logs
  • After 3 years: $3,285/year in ongoing storage for data nobody queries
Recommended Retention
Log Type
Retention
Rationale
Application debug logs
3–7 days
Only useful during active development
Lambda execution logs (dev)
7 days
Short-lived troubleshooting
Application info/warn (prod)
30 days
Covers most incident investigations
Error logs (prod)
90 days
Post-mortem and trending analysis
Security/audit logs
365 days
Compliance requirements
CloudTrail
90–365 days
Compliance (also stored in S3 trail)
Implementation Options
Option 1 — AWS CLI Batch Update:
# Set 30-day retention on all log groups matching a pattern
aws logs describe-log-groups --log-group-name-prefix "/aws/lambda/dev-" \
--query 'logGroups[].logGroupName' --output text | \
xargs -I {} aws logs put-retention-policy \
--log-group-name {} \
--retention-in-days 30

Option 2 — CloudWatch Agent Config:
{
"logs": {
"logs_collected": {
"files": {
"collect_list": [{
"file_path": "/var/log/app/*.log",
"log_group_name": "/app/production",
"retention_in_days": 30
}]
}
}
}
}

Option 3 — AWS CDK:
new logs.LogGroup(this, 'AppLogs', {
logGroupName: '/app/production',
retention: logs.RetentionDays.ONE_MONTH,
});

ROI Example — 50 log groups, 10 GB/day ingestion, Never Expire, 2 years accumulated:

Before
After (30-day)
Savings
Stored data
~7,300 GB
~300 GB
96% reduction
Monthly cost
$219/month
$9/month
$210/month
Annual savings
—
—
$2,520/year



4. Log Group Hygiene

Over time, environments accumulate orphaned log groups, redundant subscription filters, and empty log streams. These contribute to API call costs, make troubleshooting harder, and create unnecessary cross-account data transfer.
How CloudWatch API Costs Add Up
CloudWatch charges $0.01 per 1,000 API requests. Automated tools amplify this:
Source
Impact
Grafana dashboards
28,800 calls/day per dashboard (20 widgets × 1,440 min)
CI/CD health checks
Per-pipeline, per-deploy
Dead subscription filters
Lambda invocations even with zero events
Third-party monitoring
Discovery scans across ALL groups including dead ones
Overlooked cost: Dead subscription filters and orphaned groups don't appear under "CloudWatch" in your bill — they show up under Lambda, Kinesis, or Data Transfer. Subscription filters are free to configure. The cost risk is indirect: a filter that's no longer serving a purpose is often a sign that an idle downstream resource (a Kinesis stream, OpenSearch domain, or provisioned Firehose) is still running and billing under its service, not under CloudWatch.
Find Orphaned Log Groups
aws logs describe-log-groups --query 'logGroups[?lastIngestionTime!=null]' | \
python3 -c "
import json, sys, time
groups = json.load(sys.stdin)
cutoff = (time.time() - 90*86400) * 1000
for g in groups:
if g.get('lastIngestionTime', 0) < cutoff:
print(f\"{g['logGroupName']} - stale {int((time.time()*1000 - g['lastIngestionTime'])/86400000)}d\")
"

Find Empty Log Streams
LOG_GROUP="/aws/lambda/my-function"
CUTOFF=$(( $(date +%s000) - 30*86400*1000 ))

aws logs describe-log-streams \
--log-group-name "$LOG_GROUP" \
--order-by LastEventTime --descending \
--query "logStreams[?lastEventTimestamp < \`$CUTOFF\`].{Stream: logStreamName, LastEvent: lastEventTimestamp}" \
--output table

Find Broad Subscription Filters
aws logs describe-log-groups --query 'logGroups[].logGroupName' --output text | \
xargs -I {} aws logs describe-subscription-filters --log-group-name {} \
--query 'subscriptionFilters[].{LogGroup: logGroupName, Destination: destinationArn, Pattern: filterPattern}'

Look for:
  • Multiple filters on the same log group sending to similar destinations
  • Filters pointing to deleted destinations (dead Kinesis streams, decommissioned Lambdas)
  • Filters with overly broad patterns (empty string = match everything)
Cleanup Actions (Post-Approval)
Action
Effort
Impact
Delete orphaned log groups
30 min
Eliminates ongoing storage cost
Remove dead subscription filters
30 min
Reduces failed Lambda invocations
Consolidate duplicate filters
1–2 hrs
Reduces Lambda/Kinesis costs
Delete empty log streams
15 min
Faster API responses
ROI Example — 200 orphaned log groups, 5 GB each:
Category
Monthly Savings
Storage (orphaned groups)
$30
Lambda from dead filters
$15
API calls from monitoring dead groups
$10
Total annual
$660/year
In large enterprises (1,000+ log groups), hidden costs often exceed storage costs.



Audit Script: Assess Before You Act

Run this read-only script first. It generates a CSV findings report with risk levels and estimated savings. Your team reviews, assesses risk, and approves actions before any deletion.
What It Does:
  • Discovers all log groups in your account
  • Identifies orphaned groups (no ingestion in 90+ days)
  • Finds groups with "Never Expire" retention
  • Scans for empty/stale log streams
  • Flags broad subscription filters
  • Calculates estimated savings per finding
  • Assigns risk levels (LOW = safe to act, MEDIUM = needs discussion)
  • Outputs CSV for review — does NOT delete anything
Required IAM Permissions (Read-Only)
  •  logs:DescribeLogGroups 
  •  logs:DescribeLogStreams 
  •  logs:DescribeSubscriptionFilters 
Python Audit Script
#!/usr/bin/env python3
"""CloudWatch Log Hygiene Audit - Read-only, generates CSV findings report.

Usage: python3 cw_log_audit.py
Requires: pip install boto3
Permissions: logs:DescribeLogGroups, logs:DescribeLogStreams,
logs:DescribeSubscriptionFilters (read-only)
"""
import boto3, time, csv, os
from datetime import datetime

REGION = os.environ.get('AWS_REGION', 'us-east-1')
OUTPUT_DIR = './audit-results'
DAYS_STALE = 90
DAYS_EMPTY_STREAM = 30

client = boto3.client('logs', region_name=REGION)

def get_all_log_groups():
groups, paginator = [], client.get_paginator('describe_log_groups')
for page in paginator.paginate():
groups.extend(page['logGroups'])
return groups

def audit_orphaned(groups):
findings, now = [], int(time.time() * 1000)
for lg in groups:
last = lg.get('lastIngestionTime')
if last is None or (now - last) > DAYS_STALE * 86400000:
gb = lg.get('storedBytes', 0) / (1024**3)
findings.append({
'log_group': lg['logGroupName'], 'type': 'ORPHANED',
'stored_gb': round(gb, 3), 'monthly_cost': round(gb * 0.03, 4),
'days_stale': 'Never' if not last else int((now - last) / 86400000),
'risk': 'LOW', 'action': 'DELETE or set 1-day retention'
})
return findings

def audit_no_retention(groups):
findings = []
for lg in groups:
if 'retentionInDays' not in lg:
gb = lg.get('storedBytes', 0) / (1024**3)
findings.append({
'log_group': lg['logGroupName'], 'type': 'NO_RETENTION',
'stored_gb': round(gb, 3), 'monthly_cost': round(gb * 0.03, 4),
'risk': 'MEDIUM', 'action': 'Set retention (30/90/365 days)'
})
return findings

def audit_empty_streams(groups, max_groups=50):
findings, now = [], int(time.time() * 1000)
for lg in [g for g in groups if g.get('lastIngestionTime')][:max_groups]:
try:
paginator = client.get_paginator('describe_log_streams')
empty = []
for page in paginator.paginate(
logGroupName=lg['logGroupName'], orderBy='LastEventTime',
descending=False, PaginationConfig={'MaxItems': 100}):
for s in page.get('logStreams', []):
evt = s.get('lastEventTimestamp')
if evt is None or (now - evt) > DAYS_EMPTY_STREAM * 86400000:
empty.append(s['logStreamName'])
if empty:
findings.append({
'log_group': lg['logGroupName'], 'type': 'EMPTY_STREAMS',
'empty_count': len(empty), 'risk': 'LOW',
'action': 'DELETE stale streams'
})
except Exception:
continue
return findings

def audit_filters(groups):
findings = []
for lg in groups:
try:
for sf in client.describe_subscription_filters(
logGroupName=lg['logGroupName']).get('subscriptionFilters', []):
if sf.get('filterPattern', '') == '':
findings.append({
'log_group': lg['logGroupName'], 'type': 'BROAD_FILTER',
'filter_name': sf['filterName'],
'destination': sf['destinationArn'],
'risk': 'MEDIUM', 'action': 'Add filter pattern or remove'
})
except Exception:
continue
return findings

# === Run Audit ===
print("=" * 60)
print(" CLOUDWATCH LOG HYGIENE AUDIT (Read-Only)")
print("=" * 60)
print(f" Region: {REGION}")
print(f" Stale threshold: {DAYS_STALE} days")
print(f" Empty stream threshold: {DAYS_EMPTY_STREAM} days\n")

print("[1/5] Discovering log groups...")
groups = get_all_log_groups()
print(f" Found {len(groups)} log groups")

print("[2/5] Finding orphaned groups...")
f1 = audit_orphaned(groups)
print(f" -> {len(f1)} orphaned")

print("[3/5] Checking retention policies...")
f2 = audit_no_retention(groups)
print(f" -> {len(f2)} without retention policy")

print("[4/5] Scanning empty streams (may take 1-2 min)...")
f3 = audit_empty_streams(groups)
print(f" -> {len(f3)} groups with empty streams")

print("[5/5] Auditing subscription filters...")
f4 = audit_filters(groups)
print(f" -> {len(f4)} broad filters\n")

# === Generate Report ===
all_findings = f1 + f2 + f3 + f4
os.makedirs(OUTPUT_DIR, exist_ok=True)
ts = datetime.now().strftime('%Y%m%d_%H%M%S')
csv_path = f'{OUTPUT_DIR}/cw_audit_{ts}.csv'

if all_findings:
keys = ['log_group', 'type', 'risk', 'action', 'stored_gb',
'monthly_cost', 'days_stale', 'empty_count',
'filter_name', 'destination']
with open(csv_path, 'w', newline='') as f:
writer = csv.DictWriter(f, fieldnames=keys, extrasaction='ignore')
writer.writeheader()
writer.writerows(all_findings)

total = sum(x.get('monthly_cost', 0) for x in all_findings)
low = sum(1 for x in all_findings if x.get('risk') == 'LOW')
med = sum(1 for x in all_findings if x.get('risk') == 'MEDIUM')

print("=" * 60)
print(" AUDIT COMPLETE - FINDINGS SUMMARY")
print("=" * 60)
print(f" Total findings: {len(all_findings)}")
print(f" LOW risk (safe to action): {low}")
print(f" MEDIUM risk (needs review): {med}")
print(f" Est. monthly savings: ${total:.2f}")
print(f" Est. annual savings: ${total*12:.2f}")
print(f" Report saved: {csv_path}")
print("=" * 60)
print("\n NEXT STEPS:")
print(" 1. Open the CSV and sort by 'risk'")
print(" 2. LOW risk items -> safe to clean up immediately")
print(" 3. MEDIUM risk items -> discuss with team")
print(" 4. After approval, run cleanup with targeted scripts")
print("=" * 60)

Sample Output:
============================================================
CLOUDWATCH LOG HYGIENE AUDIT (Read-Only)
============================================================
Region: us-east-1
Stale threshold: 90 days
Empty stream threshold: 30 days

[1/5] Discovering log groups... Found 183 log groups
[2/5] Finding orphaned groups... -> 31 orphaned
[3/5] Checking retention policies... -> 47 without retention policy
[4/5] Scanning empty streams... -> 12 groups with empty streams
[5/5] Auditing subscription filters.. -> 4 broad filters

============================================================
AUDIT COMPLETE - FINDINGS SUMMARY
============================================================
Total findings: 47
LOW risk (safe to action): 31
MEDIUM risk (needs review): 16
Est. monthly savings: $142.50
Est. annual savings: $1710.00
Report saved: ./audit-results/cw_audit_20260803_143022.csv
============================================================

Recommended Workflow:
  1. Run audit (read-only) — generates CSV with findings, costs, and risk levels
  1. Review in Excel/Sheets — filter by risk = LOW for immediate wins
  1. Team discussion — MEDIUM risk items need business decisions
  1. Execute cleanup — only after explicit approval



Summary

All four quick wins, implementable in under one day:
#
Action
Savings
Risk
1
IA Log Class
50% ingestion
LOW
2
Intelligent Tiering
40–60% storage
LOW
3
Retention Policies
Up to 96% storage
MEDIUM
4
Log Group Hygiene
Eliminates hidden costs
LOW–MED
Combined ROI:
Quick Win
Effort
Annual Savings
1. IA Log Class
1–2 hrs
$1,500+
2. Intelligent Tiering
5 min
$1,000–$3,000+
3. Retention Policies
2–4 hrs
$2,520+
4. Log Group Hygiene
2–3 hrs
$660+
TOTAL
< 1 day
$5,680–$7,680+
Enterprise accounts (5–50 TB/month) will see 10–100× these numbers.
Start here. Before investing in architectural changes, implement these four quick wins. No application code changes, no infrastructure redesign. This can be implemented in one day and can deliver thousands in annual savings — with zero impact on observability quality.



Resources

Series: CloudWatch Builder Series (3 articles)

  1. 2
    CloudWatch Builder Series - Part 1 : Quick Win Playbook for Optimization This article
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article