Overview
This is Part 2 of the CloudWatch Builder Series. While Part 1 focused on cost optimization quick wins, this guide covers how you operate and manage multi account logs at scale — with hands-on CLI implementations and decision frameworks.
1. Cross-Account Cross-Region Log Centralization
Centralization rules automatically replicate log data from multiple AWS accounts and regions into a single destination account. Unlike cross-account observability (which federates queries), this physically copies logs into one account — giving you full ownership, independent retention, and local query performance.
Capability | Details |
|---|
Scope | Entire org, specific OUs, or specific accounts |
Filtering | By data source: VPC Flow Logs, CloudTrail, EKS, custom apps |
Backup region | Built-in DR — replicate to a second region |
Pricing | First copy FREE; additional copies $0.05/GB |
Historical data | New data only (post-rule creation) |
Integration | Works with AWS Organizations |
When to use: Centralization vs. Cross-Account Observability
Criteria | Log Centralization | Cross-Account Observability |
|---|
Data location | Physically copies to destination | Stays in source; queries federated |
Query performance | Fast (local data) | Depends on source account load |
Independent retention | Yes — set your own policies | No — governed by source account |
Compliance/audit | All logs in one account for auditors | Auditors need multi-account access |
Cost (query) | Standard Logs Insights pricing | Cross-account query charges apply |
Cost (storage) | Double storage (source + destination) | No duplication |
Real-time | Near real-time (seconds to minutes) | Real-time (federated query) |
Setup complexity | One rule covers entire org | Per-account link setup |
Best for | Compliance, security, SIEM feeds, analytics | Day-to-day troubleshooting, dashboards |
Rule of thumb:
- Security/compliance team — Centralization (need physical custody)
- Platform/SRE team — Cross-Account Observability (need real-time federated view)
- Both — Use both. They are not mutually exclusive.
Implementation Steps
Step 1: Register delegated administrator
Grants central logging account permission to create org-wide rules.
Run from the management account (one-time setup)
The target account ( 123456789012 ) must be a member account in your organization.
The caller needs the organizations:RegisterDelegatedAdministrator IAM permission
aws organizations enable-aws-service-access \
--service-principal observabilityadmin.amazonaws.com
aws organizations register-delegated-administrator \
--account-id 123456789012 \
--service-principal observabilityadmin.amazonaws.com
Step 2: Create centralization rule
Defines scope (accounts/OUs/regions) and destination. Replication starts automatically.
Run from the delegated admin account. Choose a scope:
Option A — Entire organization, all regions:
aws observabilityadmin create-centralization-rule-for-organization \
--rule-name "org-wide-centralization-rule" \
--rule '{
"Source": {
"Regions": ["*"],
"Scope": "*",
"SourceLogsConfiguration": {
"EncryptedLogGroupStrategy": "SKIP"
}
},
"Destination": {
"Region": "us-east-1",
"Account": "123456789012"
}
}'
Option B — Specific OUs, specific regions:
aws observabilityadmin create-centralization-rule-for-organization \
--rule-name "prod-staging-centralization-rule" \
--rule '{
"Source": {
"Regions": ["us-east-1", "eu-west-1"],
"Scope": "ou-abc1-production,ou-abc2-staging",
"SourceLogsConfiguration": {
"EncryptedLogGroupStrategy": "SKIP"
}
},
"Destination": {
"Region": "us-east-1",
"Account": "123456789012"
}
}'
Option C — Filter by data source + backup region:
aws observabilityadmin create-centralization-rule-for-organization \
--rule-name "org-wide-centralization-rule" \
--rule '{
"Source": {
"Regions": ["*"],
"Scope": "*",
"SourceLogsConfiguration": {
"DataSourceSelectionCriteria": "DataSourceType IN (\"VPC_FLOW_LOGS\", \"CLOUDTRAIL\", \"EKS\")",
"EncryptedLogGroupStrategy": "SKIP"
}
},
"Destination": {
"Region": "us-east-1",
"Account": "123456789012",
"DestinationLogsConfiguration": {
"BackupConfiguration": {
"Region": "eu-west-1"
}
}
}
}'
Step 3: Verify rule status
Confirms the rule is active. Run from the delegated admin account.
aws observabilityadmin get-centralization-rule-for-organization \
--rule-identifier "prod-staging-centralization-rule" \
--query '[RuleName, RuleHealth, CreatedTimeStamp, RuleArn]' \
--output table
Step 4: Verify logs arriving
Checks centralized log groups are receiving data. Run from Destination account
aws logs describe-log-groups \
--log-group-name-prefix "/centralized/" \
--query 'logGroups[].{Name: logGroupName, StoredBytes: storedBytes, RetentionDays: retentionInDays, Class: logGroupClass, Arn: arn}' \
--output table
Cost Analysis: Centralization vs. Subscription Filters
Before centralization, teams used subscription filters + Kinesis/Firehose to move logs cross-account.
Scenario: 500 GB/month across 10 accounts into 1 central account
Cost Component | Old Way (Sub Filters + Kinesis) | Centralization |
|---|
Kinesis Data Streams | ~$36/month | $0 |
Kinesis Data Firehose | ~$18/month | $0 |
Lambda processing | ~$15/month | $0 |
Cross-region data transfer | ~$10/month | $0 |
Centralization fee (first copy) | N/A | FREE |
Ingestion at destination | $250/month | $250/month |
Total monthly | ~$329/month | ~$250/month |
Annual | $3,948/year | $3,000/year |
Savings: $948/year + eliminated operational overhead (no Kinesis, Lambda, IAM cross-account roles to manage).
When the old way still makes sense:
- You need to transform logs before they land (strip PII, enrich fields)
- You need logs in S3, not CloudWatch
- You are sending to third-party SIEM (Splunk, Datadog)
Key Constraints:
Limitation | Workaround |
|---|
Only processes new data (post-rule creation) | For historical: create-export-task to S3, then re-ingest |
Backup region costs $0.05/GB | Use only for compliance-critical log types |
Cannot filter by log group name — only data source | Use OU structure + data source filter to narrow scope |
Monitoring Your Centralization
# Alert if centralization stops flowing (no data in 6 hours)
aws cloudwatch put-metric-alarm \
--alarm-name "CentralizationFlowStopped" \
--evaluation-periods 6 \
--comparison-operator LessThanOrEqualToThreshold \
--threshold 0 \
--treat-missing-data "breaching" \
--metrics '[
{
"Id": "e1",
"Expression": "SUM(METRICS(\"m1\"))",
"Label": "Total Incoming Bytes"
},
{
"Id": "m1",
"MetricStat": {
"Metric": {
"Namespace": "AWS/Logs",
"MetricName": "IncomingBytes",
"Dimensions": [
{"Name": "LogGroupName", "Value": "/centralized/vpc-flow-logs"}
]
},
"Period": 3600,
"Stat": "Sum"
},
"ReturnData": false
}
]' \
--alarm-actions arn:aws:sns:us-east-1:123456789012:ops-alerts \
--alarm-description "No centralized logs received in 6 hours"
References
Loading article