CloudWatch Unified Data Management, Part 1: One Log Store You Query as Iceberg Tables
Part 1 of a two-part series. This part covers what the feature is, how logs get in, how they're normalized, and how to query them in place with Athena. Part 2 covers cross-source correlation, the query languages, and metrics.
Series: CloudWatch Unified Agent (2 articles)
- 1CloudWatch Unified Data Management, Part 1: One Log Store You Query as Iceberg Tables This article
The last incident I worked started with a VPC Flow Log and ended in a SIEM I had to log into separately. The flow log said an internal IP was talking to somewhere it shouldn't. To find out whether that same IP had touched the AWS control plane, I needed CloudTrail, which lived in a different tool, in a different format, behind a different query language. By the time I'd exported both, lined up the timestamps by hand, and matched the IPs in a spreadsheet, the useful part of the hour was gone.
The data existed. It was just scattered, duplicated, and shaped differently everywhere it landed. That's the problem CloudWatch's unified data management is built to kill.
Announced at re:Invent 2025 and now live in nearly all commercial Regions, it turns CloudWatch from an AWS monitoring service into one store for operational, security, and compliance data. It ingests logs from AWS and third parties, normalizes them into open schemas as they arrive, and lets you query across all of it in one place. Because it exposes the data as Apache Iceberg tables, you can point Athena, Redshift, or Spark at the same logs without copying them anywhere.
This first part is about getting data in and querying it in place. The correlation payoff, the part that solved my incident, is in Part 2.
What It Actually Is
Before this, "observability" and "security data" lived in separate worlds. Ops logs sat in CloudWatch. Security logs went to a SIEM. Compliance data went somewhere a third team could audit it. Each tool kept its own copy, in its own format, and you paid to store and move all of it.
Unified data management collapses that into a single store with four moving parts:
- Managed data sources that collect logs automatically across accounts and Regions
- Pipelines that normalize and transform the data as it lands
- A query layer that speaks multiple languages over one dataset
- Iceberg-compatible access through S3 Tables, so external tools read the same data in place
The unified data store console, with Collect, Curate, and Analyze sections that map to the moving parts above
You still pay through normal CloudWatch Logs pricing for ingestion, storage, and queries. There's no new upfront commitment. The shift isn't a new bill, it's that the data stops being scattered.
Ingestion: The Logs Collect Themselves
The first job is getting data in without building collectors. CloudWatch now auto-collects AWS vended logs across your accounts and Regions through AWS Organizations. Out of the box that covers CloudTrail, VPC Flow Logs, WAF access logs, and Route 53 resolver logs, with collection policies applied uniformly at the org level instead of account by account.
The Data sources tab showing VPC Flow Logs and CloudTrail as AWS sources, each with an active S3 Tables association
The part that surprised me is the third-party connector list. These aren't AWS sources:
- Endpoint: CrowdStrike, SentinelOne
- Identity: Okta, Entra ID
- Cloud security: Wiz
- Network security: Zscaler, Palo Alto Networks
- Productivity: Microsoft Office 365, Windows Event Logs, GitHub
- ITSM: ServiceNow CMDB
That connector list is the tell. AWS isn't positioning this as "better CloudWatch Logs." It's going after the SIEM ingestion layer, the place where teams currently pay a per-gigabyte tax to land Okta and CrowdStrike data next to their cloud logs
The managed connector catalog: CrowdStrike, Zscaler, SentinelOne, Microsoft 365, Entra ID, Windows Events
More connectors: Okta, Palo Alto NGFW, GitHub Audit Log, ServiceNow CMDB, Wiz, Cisco Umbrella, Akamai
Normalization: OCSF and OTel at Ingest Time
Collecting logs is easy. Making them comparable is the hard part, and it's where most of the manual work used to live.
As data is ingested, CloudWatch can normalize it into open schemas. For security and compliance data that's the Open Cybersecurity Schema Framework (OCSF); for telemetry it's OpenTelemetry (OTel). A
parseToOCSF processor converts raw logs into OCSF events so a CrowdStrike alert and a CloudTrail event describe an actor, an action, and a resource using the same field names.You configure this with pipelines. A pipeline connects a source, runs a chain of processors, and writes to a destination. Processors cover the usual work: OCSF conversion, Grok parsing for custom formats, field-level operations, string manipulation, filtering, enrichment. Log pipelines support up to 20 processors applied in order, and there's a wizard, so you're picking processors from a library, not writing collector configs by hand.
The payoff is that a field means the same thing everywhere. When source IP is
src_endpoint.ip in every dataset, joining across sources stops being a data-cleaning project.The Iceberg Angle: Read the Data Without Owning It Twice
The feature I keep coming back to is S3 Tables integration. When you enable it, CloudWatch makes your logs available in a managed
aws-cloudwatch S3 Tables bucket as Apache Iceberg tables.Any Iceberg-compatible engine can then query the same logs in place: Athena, Redshift, SageMaker Unified Studio, Spark, or a third-party analytics tool. No ETL job copies the data into a warehouse. No second storage bill for the duplicate.
Two facts worth knowing before you lean on it:
- It's read-only. The
aws-cloudwatchbucket is a serving layer for analytics engines, not somewhere you write back. - The copy is not independent storage. Retention in the S3 Tables bucket matches the log group's retention. Set a log group to one day and the mirrored records expire after one day too. Delete the log group and the table's data goes with it.
This is what makes the "unified store" claim more than marketing. Your SOC analyst uses the console. Your data team runs Spark against the Iceberg tables. Your compliance auditor queries through Athena. One copy underneath, three tools on top.
The two log sources as Apache Iceberg tables in the managed aws-cloudwatch S3 Tables bucket
Try It Yourself: Query Flow Logs as an Iceberg Table in Athena
I ran this end to end in one account. It's more involved than the marketing "few clicks," mostly because of a catalog-registration step that's easy to miss, so here's the real sequence.
1. Get flow logs into CloudWatch Logs. Either enable VPC Flow Logs as a managed data source, or create a flow log pointed at a log group directly:
1
2
3
4
5
6
7
aws ec2 create-flow-logs \
--resource-type VPC --resource-ids <your-vpc-id> \
--traffic-type ALL \
--log-destination-type cloud-watch-logs \
--log-group-name /demo/vpc-flowlogs \
--deliver-logs-permission-arn arn:aws:iam::<acct>:role/<flow-logs-role> \
--max-aggregation-interval 602. Turn on the S3 Tables integration and associate the source. The integration creates the managed
aws-cloudwatch table bucket. The association tells it which sources to mirror into Iceberg:1
2
3
4
5
6
7
aws observabilityadmin create-s3-table-integration \
--encryption SseAlgorithm=AES256 \
--role-arn arn:aws:iam::<acct>:role/<integration-role>
aws logs associate-source-to-s3-table-integration \
--integration-arn <integration-arn> \
--data-source name=*,type=*3. Register the Iceberg catalog in Glue. This is the step the quick-start guides skip. Athena can't see the tables until the S3 Tables bucket is registered as a federated Glue catalog. Without it you get
Catalog not found:1
2
3
4
5
6
7
8
aws glue create-catalog --name s3tablescatalog --catalog-input '{
"FederatedCatalog": {
"Identifier": "arn:aws:s3tables:<region>:<acct>:bucket/*",
"ConnectionName": "aws:s3tables"
},
"CreateDatabaseDefaultPermissions": [{"Principal": {"DataLakePrincipalIdentifier": "IAM_ALLOWED_PRINCIPALS"}, "Permissions": ["ALL"]}],
"CreateTableDefaultPermissions": [{"Principal": {"DataLakePrincipalIdentifier": "IAM_ALLOWED_PRINCIPALS"}, "Permissions": ["ALL"]}]
}'IAM_ALLOWED_PRINCIPALS here keeps access on IAM instead of forcing separate Lake Formation grants for a demo. For anything real, drop that and grant table and column access through Lake Formation.4. Query the flow logs in place. The table lands in the
logs namespace as amazon_vpc__flow. This finds the top rejected connections:1
2
3
4
5
6
7
SELECT srcaddr, dstaddr, dstport, action,
count(*) AS flows, sum(bytes) AS bytes
FROM "s3tablescatalog/aws-cloudwatch"."logs"."amazon_vpc__flow"
WHERE action = 'REJECT'
GROUP BY srcaddr, dstaddr, dstport, action
ORDER BY flows DESC
LIMIT 25;No export, no copy. Athena reads the same records CloudWatch is storing.
The flow-logs query running in Athena against the amazon_vpc__flow Iceberg table, returning traffic by source IP and action
Two things that will trip you up:
- There's no backfill. Only events that arrive after you create the association land in the tables. Generate some traffic, then wait.
- Materialization lags, and unevenly. When I ran this, the CloudTrail table filled in within a few minutes while the flow-logs table took longer to commit its per-flow columns like
srcaddrandaction. Budget 15 to 30 minutes before the schema and data settle. The "about 8 minutes" you'll see quoted is optimistic once you account for delivery plus Iceberg commit time.
When you're done, delete the flow log, the S3 Tables integration and association, the Glue catalog, and the log group so ingestion charges stop.
What's in Part 2
Querying one table in place is the foundation. The reason the feature matters is what happens when you query across sources. Part 2 picks up there: the query languages and the Facets panel, the correlation that answers "is this IP touching both my network and my control plane," and how metrics get the same open-schema treatment through OTel and PromQL.
Series: CloudWatch Unified Agent (2 articles)
- 1CloudWatch Unified Data Management, Part 1: One Log Store You Query as Iceberg Tables This article
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article