AWS Builder Center

CloudWatch Builder Series - Part 2 : Centralize, Query, and Instrument at Scale

A hands on guide to Log Centralization. Set up cross-account, cross-region log centralization using the AWS CLI — no Kinesis, no Lambda, no cross-account IAM roles. Includes decision matrix (centralization vs. cross-account observability), data source filtering, backup region DR, cost analysis, and three deployment patterns.

Series: CloudWatch Builder Series (3 articles)

  1. 1
    CloudWatch Builder Series - Part 2 : Centralize, Query, and Instrument at Scale This article

Overview

This is Part 2 of the CloudWatch Builder Series. While Part 1 focused on cost optimization quick wins, this guide covers how you operate and manage multi account logs at scale — with hands-on CLI implementations and decision frameworks.

1. Cross-Account Cross-Region Log Centralization

Centralization rules automatically replicate log data from multiple AWS accounts and regions into a single destination account. Unlike cross-account observability (which federates queries), this physically copies logs into one account — giving you full ownership, independent retention, and local query performance.
Capability
Details
Scope
Entire org, specific OUs, or specific accounts
Filtering
By data source: VPC Flow Logs, CloudTrail, EKS, custom apps
Backup region
Built-in DR — replicate to a second region
Pricing
First copy FREE; additional copies $0.05/GB
Historical data
New data only (post-rule creation)
Integration
Works with AWS Organizations

When to use: Centralization vs. Cross-Account Observability

Criteria
Log Centralization
Cross-Account Observability
Data location
Physically copies to destination
Stays in source; queries federated
Query performance
Fast (local data)
Depends on source account load
Independent retention
Yes — set your own policies
No — governed by source account
Compliance/audit
All logs in one account for auditors
Auditors need multi-account access
Cost (query)
Standard Logs Insights pricing
Cross-account query charges apply
Cost (storage)
Double storage (source + destination)
No duplication
Real-time
Near real-time (seconds to minutes)
Real-time (federated query)
Setup complexity
One rule covers entire org
Per-account link setup
Best for
Compliance, security, SIEM feeds, analytics
Day-to-day troubleshooting, dashboards
Rule of thumb:
  • Security/compliance team — Centralization (need physical custody)
  • Platform/SRE team — Cross-Account Observability (need real-time federated view)
  • Both — Use both. They are not mutually exclusive.

Implementation Steps


Step 1: Register delegated administrator
Grants central logging account permission to create org-wide rules.
Run from the management account (one-time setup)
The target account ( 123456789012 ) must be a member account in your organization.
The caller needs the  organizations:RegisterDelegatedAdministrator  IAM permission
aws organizations enable-aws-service-access \
--service-principal observabilityadmin.amazonaws.com
aws organizations register-delegated-administrator \
--account-id 123456789012 \
--service-principal observabilityadmin.amazonaws.com

Step 2: Create centralization rule
Defines scope (accounts/OUs/regions) and destination. Replication starts automatically.
Run from the delegated admin account. Choose a scope:
Option A — Entire organization, all regions:
aws observabilityadmin create-centralization-rule-for-organization \
--rule-name "org-wide-centralization-rule" \
--rule '{
"Source": {
"Regions": ["*"],
"Scope": "*",
"SourceLogsConfiguration": {
"EncryptedLogGroupStrategy": "SKIP"
}
},
"Destination": {
"Region": "us-east-1",
"Account": "123456789012"
}
}'

Option B — Specific OUs, specific regions:
aws observabilityadmin create-centralization-rule-for-organization \
--rule-name "prod-staging-centralization-rule" \
--rule '{
"Source": {
"Regions": ["us-east-1", "eu-west-1"],
"Scope": "ou-abc1-production,ou-abc2-staging",
"SourceLogsConfiguration": {
"EncryptedLogGroupStrategy": "SKIP"
}
},
"Destination": {
"Region": "us-east-1",
"Account": "123456789012"
}
}'

Option C — Filter by data source + backup region:
aws observabilityadmin create-centralization-rule-for-organization \
--rule-name "org-wide-centralization-rule" \
--rule '{
"Source": {
"Regions": ["*"],
"Scope": "*",
"SourceLogsConfiguration": {
"DataSourceSelectionCriteria": "DataSourceType IN (\"VPC_FLOW_LOGS\", \"CLOUDTRAIL\", \"EKS\")",
"EncryptedLogGroupStrategy": "SKIP"
}
},
"Destination": {
"Region": "us-east-1",
"Account": "123456789012",
"DestinationLogsConfiguration": {
"BackupConfiguration": {
"Region": "eu-west-1"
}
}
}
}'

Step 3: Verify rule status
Confirms the rule is active. Run from the delegated admin account.
aws observabilityadmin get-centralization-rule-for-organization \
--rule-identifier "prod-staging-centralization-rule" \
--query '[RuleName, RuleHealth, CreatedTimeStamp, RuleArn]' \
--output table

Step 4: Verify logs arriving
Checks centralized log groups are receiving data. Run from Destination account

aws logs describe-log-groups \
--log-group-name-prefix "/centralized/" \
--query 'logGroups[].{Name: logGroupName, StoredBytes: storedBytes, RetentionDays: retentionInDays, Class: logGroupClass, Arn: arn}' \
--output table


Cost Analysis: Centralization vs. Subscription Filters

Before centralization, teams used subscription filters + Kinesis/Firehose to move logs cross-account.
Scenario: 500 GB/month across 10 accounts into 1 central account
Cost Component
Old Way (Sub Filters + Kinesis)
Centralization
Kinesis Data Streams
~$36/month
$0
Kinesis Data Firehose
~$18/month
$0
Lambda processing
~$15/month
$0
Cross-region data transfer
~$10/month
$0
Centralization fee (first copy)
N/A
FREE
Ingestion at destination
$250/month
$250/month
Total monthly
~$329/month
~$250/month
Annual
$3,948/year
$3,000/year
Savings: $948/year + eliminated operational overhead (no Kinesis, Lambda, IAM cross-account roles to manage).
When the old way still makes sense:
  • You need to transform logs before they land (strip PII, enrich fields)
  • You need logs in S3, not CloudWatch
  • You are sending to third-party SIEM (Splunk, Datadog)
Key Constraints:
Limitation
Workaround
Only processes new data (post-rule creation)
For historical: create-export-task to S3, then re-ingest
Backup region costs $0.05/GB
Use only for compliance-critical log types
Cannot filter by log group name — only data source
Use OU structure + data source filter to narrow scope

Monitoring Your Centralization

# Alert if centralization stops flowing (no data in 6 hours)
aws cloudwatch put-metric-alarm \
--alarm-name "CentralizationFlowStopped" \
--evaluation-periods 6 \
--comparison-operator LessThanOrEqualToThreshold \
--threshold 0 \
--treat-missing-data "breaching" \
--metrics '[
{
"Id": "e1",
"Expression": "SUM(METRICS(\"m1\"))",
"Label": "Total Incoming Bytes"
},
{
"Id": "m1",
"MetricStat": {
"Metric": {
"Namespace": "AWS/Logs",
"MetricName": "IncomingBytes",
"Dimensions": [
{"Name": "LogGroupName", "Value": "/centralized/vpc-flow-logs"}
]
},
"Period": 3600,
"Stat": "Sum"
},
"ReturnData": false
}
]' \
--alarm-actions arn:aws:sns:us-east-1:123456789012:ops-alerts \
--alarm-description "No centralized logs received in 6 hours"




References

Series: CloudWatch Builder Series (3 articles)

  1. 1
    CloudWatch Builder Series - Part 2 : Centralize, Query, and Instrument at Scale This article
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article