CloudWatch Builder Series - Part 1 : Quick Win Playbook for Optimization
A practitioner's guide to reclaiming CloudWatch spend using log class selection, intelligent tiering, retention tuning, and hygiene automation. Hands-on CLI, audit scripts, and decision tables.
Series: CloudWatch Builder Series (3 articles)
- 2CloudWatch Builder Series - Part 1 : Quick Win Playbook for Optimization This article
Overview
Log ingestion and storage make up the majority of the CloudWatch bills. This guide covers four quick wins — each implementable in hours with no application architecture changes — that together reduce CloudWatch spend by 20–50%. (Figures based on ~500 GB/month ingestion. Scale linearly for larger workloads.)
1. Log Class Selection: Standard vs. Infrequent Access
Log ingestion is typically 60–70% of a CloudWatch bill. The Infrequent Access (IA) log class offers a 50% lower ingestion cost ($0.25/GB vs. $0.50/GB in us-east-1) for logs that don't require real-time streaming or subscription filters.
IA Capabilities (as of March 2026)
- ✅ Logs Insights queries (same speed as Standard)
- ✅ Cross-account log analytics
- ✅ KMS encryption
- ✅ Managed ingestion and storage
- ✅ S3 export
- ✅ Data Protection (Mar 2026)
- ✅ OpenSearch PPL/SQL queries (Mar 2026)
- ❌ Subscription filters (no streaming to Lambda/Kinesis/OpenSearch)
- ❌ Metric filters (cannot extract metrics from log lines)
- ❌ Live Tail
Decision Matrix
Log Type | Class | Rationale |
|---|---|---|
EKS/ECS control plane logs | IA | Queried only during troubleshooting |
VPC Flow Logs (if sent to CW) | IA | Typically archived for compliance |
CloudTrail management events | IA | Audit/compliance, infrequent queries |
Application debug/trace logs | IA | Only needed during incidents |
Lambda logs (dev/staging) | IA | Low query frequency |
Production application error logs | Standard | May need real-time alerts via metric filters |
Security/WAF logs with real-time alerting | Standard | Requires subscription filters |
Logs feeding real-time dashboards | Standard | Needs metric filter extraction |
Implementation
# Create a new log group with IA classaws logs create-log-group \ --log-group-name /eks/my-cluster/control-plane \ --log-group-class INFREQUENT_ACCESS
# Note: Existing log groups cannot change class.# You must create a new IA log group and redirect log sources.
ROI Example — Team ingesting 500 GB/month of EKS control plane logs:
| Standard | Infrequent Access | Savings |
|---|---|---|---|
Ingestion cost | $250/month | $125/month | $125/month |
Annual savings | — | — | $1,500/year |
Multiply across 10+ clusters = $15,000+/year from this single change.
Savings: 50% on ingestion cost
2. Intelligent Tiering
Most log data is queried intensively for the first few days, then rarely touched. Intelligent Tiering automatically moves data to cheaper storage tiers based on access patterns — with zero change to your query experience.
How It Works
Tier | Trigger | Query Experience | Storage Cost |
|---|---|---|---|
Standard (default) | All new data starts here | Full speed | Base rate |
Infrequent Access | No access for 30 days | No degradation | Lower |
Archive Instant Access | No access for 90 days | No degradation | Lowest |
Key difference from S3 Intelligent Tiering: This is a billing-layer optimization only. No retrieval delay, no restore request, no retrieval fee. Queries run at the same speed regardless of tier. When you query older data, it automatically promotes back to Standard for 30 days.
Implementation
# Enable for your entire account in a regionaws logs put-storage-tier-policy \ --storage-tier INTELLIGENT_TIERING
# Or via Console: CloudWatch > Logs > Settings > Enable Intelligent Tiering
That's it — one API call, applies to all log groups in the account within that Region.
Why this matters:
- Zero operational overhead — no lifecycle policies to manage
- No query degradation
- Set-and-forget — works on existing log groups with existing data
ROI Example — 2 TB stored logs, 80% older than 30 days, 60% older than 90 days:
Tier | Data | Rate ($/GB/month) | Monthly Cost |
|---|---|---|---|
Standard (<30 days) | 410 GB (20%) | $0.03 | $12.29 |
Infrequent Access (30-90 days) | 410 GB (20%) | $0.0125 | $5.12 |
Archive Instant Access (>90 days) | 1,229 GB (60%) | $0.004 | $4.92 |
With tiering, the monthly cost drops to $22.32 — versus $61.44 without it. That's an annual savings of $469.44. For an enterprise scale (5–10 TB of stored logs), the annual savings would reach ~$2,350.
Savings: 63.7% on storage costs
3. Log Retention Policies
The default retention is "Never Expire" — every byte stored forever. Most teams have years of debug logs they'll never query again.
The Compounding Cost
Storage pricing: ~$0.03/GB/month (Standard class)
- 100 GB/day × 365 days × $0.03/GB = $1,095/year for one year's logs
- After 3 years: $3,285/year in ongoing storage for data nobody queries
Recommended Retention
Log Type | Retention | Rationale |
|---|---|---|
Application debug logs | 3–7 days | Only useful during active development |
Lambda execution logs (dev) | 7 days | Short-lived troubleshooting |
Application info/warn (prod) | 30 days | Covers most incident investigations |
Error logs (prod) | 90 days | Post-mortem and trending analysis |
Security/audit logs | 365 days | Compliance requirements |
CloudTrail | 90–365 days | Compliance (also stored in S3 trail) |
Implementation Options
Option 1 — AWS CLI Batch Update:
# Set 30-day retention on all log groups matching a patternaws logs describe-log-groups --log-group-name-prefix "/aws/lambda/dev-" \ --query 'logGroups[].logGroupName' --output text | \ xargs -I {} aws logs put-retention-policy \ --log-group-name {} \ --retention-in-days 30
Option 2 — CloudWatch Agent Config:
{ "logs": { "logs_collected": { "files": { "collect_list": [{ "file_path": "/var/log/app/*.log", "log_group_name": "/app/production", "retention_in_days": 30 }] } } }}
Option 3 — AWS CDK:
new logs.LogGroup(this, 'AppLogs', { logGroupName: '/app/production', retention: logs.RetentionDays.ONE_MONTH,});
ROI Example — 50 log groups, 10 GB/day ingestion, Never Expire, 2 years accumulated:
| Before | After (30-day) | Savings |
|---|---|---|---|
Stored data | ~7,300 GB | ~300 GB | 96% reduction |
Monthly cost | $219/month | $9/month | $210/month |
Annual savings | — | — | $2,520/year |
4. Log Group Hygiene
Over time, environments accumulate orphaned log groups, redundant subscription filters, and empty log streams. These contribute to API call costs, make troubleshooting harder, and create unnecessary cross-account data transfer.
How CloudWatch API Costs Add Up
CloudWatch charges $0.01 per 1,000 API requests. Automated tools amplify this:
Source | Impact |
|---|---|
Grafana dashboards | 28,800 calls/day per dashboard (20 widgets × 1,440 min) |
CI/CD health checks | Per-pipeline, per-deploy |
Dead subscription filters | Lambda invocations even with zero events |
Third-party monitoring | Discovery scans across ALL groups including dead ones |
Overlooked cost: Dead subscription filters and orphaned groups don't appear under "CloudWatch" in your bill — they show up under Lambda, Kinesis, or Data Transfer. Subscription filters are free to configure. The cost risk is indirect: a filter that's no longer serving a purpose is often a sign that an idle downstream resource (a Kinesis stream, OpenSearch domain, or provisioned Firehose) is still running and billing under its service, not under CloudWatch.
Find Orphaned Log Groups
aws logs describe-log-groups --query 'logGroups[?lastIngestionTime!=null]' | \ python3 -c "import json, sys, timegroups = json.load(sys.stdin)cutoff = (time.time() - 90*86400) * 1000for g in groups: if g.get('lastIngestionTime', 0) < cutoff: print(f\"{g['logGroupName']} - stale {int((time.time()*1000 - g['lastIngestionTime'])/86400000)}d\")"
Find Empty Log Streams
LOG_GROUP="/aws/lambda/my-function"CUTOFF=$(( $(date +%s000) - 30*86400*1000 ))
aws logs describe-log-streams \ --log-group-name "$LOG_GROUP" \ --order-by LastEventTime --descending \ --query "logStreams[?lastEventTimestamp < \`$CUTOFF\`].{Stream: logStreamName, LastEvent: lastEventTimestamp}" \ --output table
Find Broad Subscription Filters
aws logs describe-log-groups --query 'logGroups[].logGroupName' --output text | \ xargs -I {} aws logs describe-subscription-filters --log-group-name {} \ --query 'subscriptionFilters[].{LogGroup: logGroupName, Destination: destinationArn, Pattern: filterPattern}'
Look for:
- Multiple filters on the same log group sending to similar destinations
- Filters pointing to deleted destinations (dead Kinesis streams, decommissioned Lambdas)
- Filters with overly broad patterns (empty string = match everything)
Cleanup Actions (Post-Approval)
Action | Effort | Impact |
|---|---|---|
Delete orphaned log groups | 30 min | Eliminates ongoing storage cost |
Remove dead subscription filters | 30 min | Reduces failed Lambda invocations |
Consolidate duplicate filters | 1–2 hrs | Reduces Lambda/Kinesis costs |
Delete empty log streams | 15 min | Faster API responses |
ROI Example — 200 orphaned log groups, 5 GB each:
Category | Monthly Savings |
|---|---|
Storage (orphaned groups) | $30 |
Lambda from dead filters | $15 |
API calls from monitoring dead groups | $10 |
Total annual | $660/year |
In large enterprises (1,000+ log groups), hidden costs often exceed storage costs.
Audit Script: Assess Before You Act
Run this read-only script first. It generates a CSV findings report with risk levels and estimated savings. Your team reviews, assesses risk, and approves actions before any deletion.
What It Does:
- Discovers all log groups in your account
- Identifies orphaned groups (no ingestion in 90+ days)
- Finds groups with "Never Expire" retention
- Scans for empty/stale log streams
- Flags broad subscription filters
- Calculates estimated savings per finding
- Assigns risk levels (LOW = safe to act, MEDIUM = needs discussion)
- Outputs CSV for review — does NOT delete anything
Required IAM Permissions (Read-Only)
-
logs:DescribeLogGroups
-
logs:DescribeLogStreams
-
logs:DescribeSubscriptionFilters
Python Audit Script
#!/usr/bin/env python3"""CloudWatch Log Hygiene Audit - Read-only, generates CSV findings report.
Usage: python3 cw_log_audit.pyRequires: pip install boto3Permissions: logs:DescribeLogGroups, logs:DescribeLogStreams, logs:DescribeSubscriptionFilters (read-only)"""import boto3, time, csv, osfrom datetime import datetime
REGION = os.environ.get('AWS_REGION', 'us-east-1')OUTPUT_DIR = './audit-results'DAYS_STALE = 90DAYS_EMPTY_STREAM = 30
client = boto3.client('logs', region_name=REGION)
def get_all_log_groups(): groups, paginator = [], client.get_paginator('describe_log_groups') for page in paginator.paginate(): groups.extend(page['logGroups']) return groups
def audit_orphaned(groups): findings, now = [], int(time.time() * 1000) for lg in groups: last = lg.get('lastIngestionTime') if last is None or (now - last) > DAYS_STALE * 86400000: gb = lg.get('storedBytes', 0) / (1024**3) findings.append({ 'log_group': lg['logGroupName'], 'type': 'ORPHANED', 'stored_gb': round(gb, 3), 'monthly_cost': round(gb * 0.03, 4), 'days_stale': 'Never' if not last else int((now - last) / 86400000), 'risk': 'LOW', 'action': 'DELETE or set 1-day retention' }) return findings
def audit_no_retention(groups): findings = [] for lg in groups: if 'retentionInDays' not in lg: gb = lg.get('storedBytes', 0) / (1024**3) findings.append({ 'log_group': lg['logGroupName'], 'type': 'NO_RETENTION', 'stored_gb': round(gb, 3), 'monthly_cost': round(gb * 0.03, 4), 'risk': 'MEDIUM', 'action': 'Set retention (30/90/365 days)' }) return findings
def audit_empty_streams(groups, max_groups=50): findings, now = [], int(time.time() * 1000) for lg in [g for g in groups if g.get('lastIngestionTime')][:max_groups]: try: paginator = client.get_paginator('describe_log_streams') empty = [] for page in paginator.paginate( logGroupName=lg['logGroupName'], orderBy='LastEventTime', descending=False, PaginationConfig={'MaxItems': 100}): for s in page.get('logStreams', []): evt = s.get('lastEventTimestamp') if evt is None or (now - evt) > DAYS_EMPTY_STREAM * 86400000: empty.append(s['logStreamName']) if empty: findings.append({ 'log_group': lg['logGroupName'], 'type': 'EMPTY_STREAMS', 'empty_count': len(empty), 'risk': 'LOW', 'action': 'DELETE stale streams' }) except Exception: continue return findings
def audit_filters(groups): findings = [] for lg in groups: try: for sf in client.describe_subscription_filters( logGroupName=lg['logGroupName']).get('subscriptionFilters', []): if sf.get('filterPattern', '') == '': findings.append({ 'log_group': lg['logGroupName'], 'type': 'BROAD_FILTER', 'filter_name': sf['filterName'], 'destination': sf['destinationArn'], 'risk': 'MEDIUM', 'action': 'Add filter pattern or remove' }) except Exception: continue return findings
# === Run Audit ===print("=" * 60)print(" CLOUDWATCH LOG HYGIENE AUDIT (Read-Only)")print("=" * 60)print(f" Region: {REGION}")print(f" Stale threshold: {DAYS_STALE} days")print(f" Empty stream threshold: {DAYS_EMPTY_STREAM} days\n")
print("[1/5] Discovering log groups...")groups = get_all_log_groups()print(f" Found {len(groups)} log groups")
print("[2/5] Finding orphaned groups...")f1 = audit_orphaned(groups)print(f" -> {len(f1)} orphaned")
print("[3/5] Checking retention policies...")f2 = audit_no_retention(groups)print(f" -> {len(f2)} without retention policy")
print("[4/5] Scanning empty streams (may take 1-2 min)...")f3 = audit_empty_streams(groups)print(f" -> {len(f3)} groups with empty streams")
print("[5/5] Auditing subscription filters...")f4 = audit_filters(groups)print(f" -> {len(f4)} broad filters\n")
# === Generate Report ===all_findings = f1 + f2 + f3 + f4os.makedirs(OUTPUT_DIR, exist_ok=True)ts = datetime.now().strftime('%Y%m%d_%H%M%S')csv_path = f'{OUTPUT_DIR}/cw_audit_{ts}.csv'
if all_findings: keys = ['log_group', 'type', 'risk', 'action', 'stored_gb', 'monthly_cost', 'days_stale', 'empty_count', 'filter_name', 'destination'] with open(csv_path, 'w', newline='') as f: writer = csv.DictWriter(f, fieldnames=keys, extrasaction='ignore') writer.writeheader() writer.writerows(all_findings)
total = sum(x.get('monthly_cost', 0) for x in all_findings)low = sum(1 for x in all_findings if x.get('risk') == 'LOW')med = sum(1 for x in all_findings if x.get('risk') == 'MEDIUM')
print("=" * 60)print(" AUDIT COMPLETE - FINDINGS SUMMARY")print("=" * 60)print(f" Total findings: {len(all_findings)}")print(f" LOW risk (safe to action): {low}")print(f" MEDIUM risk (needs review): {med}")print(f" Est. monthly savings: ${total:.2f}")print(f" Est. annual savings: ${total*12:.2f}")print(f" Report saved: {csv_path}")print("=" * 60)print("\n NEXT STEPS:")print(" 1. Open the CSV and sort by 'risk'")print(" 2. LOW risk items -> safe to clean up immediately")print(" 3. MEDIUM risk items -> discuss with team")print(" 4. After approval, run cleanup with targeted scripts")print("=" * 60)
Sample Output:
============================================================ CLOUDWATCH LOG HYGIENE AUDIT (Read-Only)============================================================ Region: us-east-1 Stale threshold: 90 days Empty stream threshold: 30 days
[1/5] Discovering log groups... Found 183 log groups[2/5] Finding orphaned groups... -> 31 orphaned[3/5] Checking retention policies... -> 47 without retention policy[4/5] Scanning empty streams... -> 12 groups with empty streams[5/5] Auditing subscription filters.. -> 4 broad filters
============================================================ AUDIT COMPLETE - FINDINGS SUMMARY============================================================ Total findings: 47 LOW risk (safe to action): 31 MEDIUM risk (needs review): 16 Est. monthly savings: $142.50 Est. annual savings: $1710.00 Report saved: ./audit-results/cw_audit_20260803_143022.csv============================================================
Recommended Workflow:
- Run audit (read-only) — generates CSV with findings, costs, and risk levels
- Review in Excel/Sheets — filter by risk = LOW for immediate wins
- Team discussion — MEDIUM risk items need business decisions
- Execute cleanup — only after explicit approval
Summary
All four quick wins, implementable in under one day:
# | Action | Savings | Risk |
|---|---|---|---|
1 | IA Log Class | 50% ingestion | LOW |
2 | Intelligent Tiering | 40–60% storage | LOW |
3 | Retention Policies | Up to 96% storage | MEDIUM |
4 | Log Group Hygiene | Eliminates hidden costs | LOW–MED |
Combined ROI:
Quick Win | Effort | Annual Savings |
|---|---|---|
1. IA Log Class | 1–2 hrs | $1,500+ |
2. Intelligent Tiering | 5 min | $1,000–$3,000+ |
3. Retention Policies | 2–4 hrs | $2,520+ |
4. Log Group Hygiene | 2–3 hrs | $660+ |
TOTAL | < 1 day | $5,680–$7,680+ |
Enterprise accounts (5–50 TB/month) will see 10–100× these numbers.
Start here. Before investing in architectural changes, implement these four quick wins. No application code changes, no infrastructure redesign. This can be implemented in one day and can deliver thousands in annual savings — with zero impact on observability quality.
Resources
Series: CloudWatch Builder Series (3 articles)
- 2CloudWatch Builder Series - Part 1 : Quick Win Playbook for Optimization This article
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article