
Taming Iceberg Table Sprawl with AWS Glue Table Optimizers
AWS Glue's Iceberg table optimizers quietly fix tiny files, dead snapshots, and orphaned data. Here's how to turn them on with IAM policy and CLI calls.
Taming Iceberg Table Sprawl with AWS Glue Table Optimizers
I got pulled into a support thread a while back where a team's Athena query on an "orders" table had gone from a few seconds to over a minute, and nobody had touched the query itself. Turned out the table had accumulated close to 40,000 files in a single partition, most of them a few KB each, from a Kafka Connect job that had been quietly committing small batches for months. Nobody had a compaction job running against it. Nobody had thought about it since the pipeline went live.
That's basically the point of this post. If you're running Apache Iceberg on the Glue Data Catalog and you're not using AWS Glue's table optimizers, you probably have some version of this problem already — you just haven't gone looking.
Why this happens
Iceberg tables age in three ways that nobody notices day to day:
Small files pile up. Every micro-batch write — Kinesis, Kafka Connect, a Glue streaming job, Flink, whatever's feeding the table — adds a new Parquet file. A few hundred writes a day over a few months and a partition that should have a few hundred well-sized files ends up with tens of thousands of tiny ones. More files means more S3 GET/LIST calls and more metadata for the query engine to chew through before it even reads a row.
Snapshots never expire. Iceberg gives you time travel by keeping a version history — every commit is a new snapshot. Nobody sets an expiration policy by default, so the history just grows, and so does the storage held by data files that only exist because an old snapshot still points at them.
Orphan files accumulate. A failed job, a retried write, a dropped partition — these leave files sitting in S3 that the table's metadata no longer references. They don't show up in query results, but they're still billed.
AWS Glue has a managed answer to all three, and it's genuinely low-effort to turn on. That's the part worth writing up, because the actual setup — the IAM policy, the exact CLI calls, where people get it wrong — isn't laid out anywhere in one place.

What you're actually turning on
Three separate features, each independently switchable per table:
Compaction merges small files into properly sized ones. The default strategy is binpack — just merge, no reordering. You can also ask it to sort by a column, or use Z-order across multiple columns if your query patterns filter on more than one field at a time.
Snapshot retention expires old snapshots past a retention window or count, and — if you tell it to — deletes the data files that only those expired snapshots were holding onto.
Orphan file deletion scans a table's S3 location for files the Iceberg metadata doesn't know about, and removes the ones old enough to be safe to delete.
One thing worth flagging up front: this is an Iceberg-only feature. It doesn't apply to Hudi or Delta Lake tables in the Glue Catalog, and compaction specifically only handles Parquet. If you're on a different format, this whole feature set isn't for you.
They all run as managed background jobs — Glue schedules and scales them, you don't spin up a cluster or write a Spark job. You point an IAM role at the table and that's most of the work.
Setting up the IAM role
This is the part that actually takes effort, and it's worth getting right once.
Trust policy, so Glue can assume the role:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": { "Service": "glue.amazonaws.com" },
"Action": "sts:AssumeRole"
}
]
}Permissions policy for S3, Glue metadata, and logging:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": ["s3:PutObject", "s3:GetObject", "s3:DeleteObject"],
"Resource": ["arn:aws:s3:::your-iceberg-bucket/orders_table/*"]
},
{
"Effect": "Allow",
"Action": ["s3:ListBucket"],
"Resource": ["arn:aws:s3:::your-iceberg-bucket"]
},
{
"Effect": "Allow",
"Action": ["glue:UpdateTable", "glue:GetTable"],
"Resource": [
"arn:aws:glue:us-east-1:123456789012:table/analytics_db/orders_table",
"arn:aws:glue:us-east-1:123456789012:database/analytics_db",
"arn:aws:glue:us-east-1:123456789012:catalog"
]
},
{
"Effect": "Allow",
"Action": ["logs:CreateLogGroup", "logs:CreateLogStream", "logs:PutLogEvents"],
"Resource": [
"arn:aws:logs:us-east-1:123456789012:log-group:/aws-glue/iceberg-compaction/logs:*",
"arn:aws:logs:us-east-1:123456789012:log-group:/aws-glue/iceberg-retention/logs:*",
"arn:aws:logs:us-east-1:123456789012:log-group:/aws-glue/iceberg-orphan-file-deletion/logs:*"
]
}
]
}A few things I'd have wanted someone to tell me before I set this up myself:
Scope that S3 resource to the table's own prefix, not the bucket root. It's not just tidiness — orphan file deletion scans everything under the table's configured location, so if two tables happen to share a prefix, one table's optimizer can delete the other table's files. This is the one real way to lose data with this feature, and it's entirely avoidable by giving every table its own path.
If the table's data is registered with Lake Formation, you don't use raw S3 IAM permissions at all — swap them for
lakeformation:GetDataAccess and the relevant Lake Formation grants (ALTER, DESCRIBE, INSERT, DELETE) on the table.If the bucket uses SSE-KMS, the role needs
kms:Decrypt and kms:GenerateDataKey , but that grant has to live on the KMS key policy, not just the IAM role. Miss this and compaction just fails quietly — no obvious "access denied" pointing at KMS, just a failed run in the logs.Also add
iam:PassRole on the optimizer role, granted to whatever principal is going to call create-table-optimizer — otherwise that call itself gets rejected.Orphan file deletion doesn't need
glue:UpdateTable or s3:PutObject — it only reads and deletes. If you want to keep its blast radius smaller, give it a separate, narrower role instead of reusing the compaction role for everything.Turning it on
Using a table
analytics_db.orders_table in account 123456789012 , region us-east-1 , with a role arn:aws:iam::123456789012:role/iceberg-optimizer-role .
Compaction
aws glue create-table-optimizer \
--catalog-id 123456789012 \
--database-name analytics_db \
--table-name orders_table \
--table-optimizer-configuration '{
"roleArn": "arn:aws:iam::123456789012:role/iceberg-optimizer-role",
"enabled": true,
"compactionConfiguration": {
"icebergConfiguration": { "strategy": "binpack" }
}
}' \
--type compactionOnce this is on, Glue watches the table itself — it kicks off a run when a partition crosses 100 files where each one is under 75% of the target file size (512 MB by default, from the
write.target-file-size-bytes table property). There's nothing to schedule.On strategy: stick with
binpack unless you have a specific reason not to. Switch to sort if your queries almost always filter on one particular column — it clusters similar values together so the engine can skip more files. Reach for zorder only if queries commonly filter across two or more columns at once, since it weights all of them evenly instead of favoring one. It won't work on Decimal or TimestampWithoutZone columns, so check your schema before picking it.If compaction fails four runs in a row, Glue suspends the optimizer on its own rather than keep burning compute on something broken. It's worth knowing that threshold exists so a string of failures doesn't just look like the feature "stopped working" — check the CloudWatch log group for the real error, fix it, then re-enable.
Snapshot retention
aws glue create-table-optimizer \
--catalog-id 123456789012 \
--database-name analytics_db \
--table-name orders_table \
--table-optimizer-configuration '{
"roleArn": "arn:aws:iam::123456789012:role/iceberg-optimizer-role",
"enabled": true,
"retentionConfiguration": {
"icebergConfiguration": {
"snapshotRetentionPeriodInDays": 7,
"numberOfSnapshotsToRetain": 3,
"cleanExpiredFiles": true
}
}
}' \
--type retentionsnapshotRetentionPeriodInDays defaults to 5 and controls how far back you can time-travel. numberOfSnapshotsToRetain defaults to 1 and acts as a floor — Glue keeps at least that many snapshots no matter how old they are. cleanExpiredFiles defaults to true and is what actually frees up S3 storage; setting it to false just removes the metadata pointer and leaves the underlying files behind as orphans for the other optimizer to eventually pick up.The one thing that confuses people: a snapshot only gets deleted once it's both past the retention window and beyond the minimum count. Set the minimum too high relative to your write frequency and you'll be staring at snapshots that look expired but aren't going anywhere. Also, if the table uses Iceberg branches or tags, their own retention rules take precedence over this config, so don't be surprised when a tagged snapshot outlives your retention window.
Orphan file deletion
aws glue create-table-optimizer \
--catalog-id 123456789012 \
--database-name analytics_db \
--table-name orders_table \
--table-optimizer-configuration '{
"roleArn": "arn:aws:iam::123456789012:role/iceberg-optimizer-role",
"enabled": true,
"orphanFileDeletionConfiguration": {
"icebergConfiguration": {
"orphanFileRetentionPeriodInDays": 3,
"location": "s3://your-iceberg-bucket/orders_table/"
}
}
}' \
--type orphan_file_deletionIt only ever deletes files created after the optimizer itself was enabled, so switching this on for an existing table is safe — anything already sitting there is left alone. The
location field can narrow the scan to a sub-path if the table has nested locations you want excluded, though the better fix is making sure table locations don't overlap in the first place. Left on its default, it runs about once a day.If you're rolling this out to a lot of tables
Configuring optimizers table by table is fine for a handful of tables, not for a whole data lake. AWS Glue also supports catalog-level defaults, so new and updated Iceberg tables pick up compaction, retention, and orphan deletion automatically without anyone touching them individually. You set this up through the Lake Formation console's "Table optimization" option on the default catalog, or the
UpdateCatalog API — whoever runs it needs glue:UpdateCatalog or Lake Formation's ALTER CATALOG permission.A few things worth knowing before you flip that switch: table-level settings always beat the catalog default, so a table that already has its own config is untouched by anything you change at the catalog level, including disabling it. Tables without their own config are supposed to inherit a catalog-level disable too, but there's a known AWS issue where that inheritance doesn't always take — don't just trust the toggle, check
get-table-optimizer on a sample of tables to confirm what's actually running. And if Data Catalog metadata encryption is turned on, catalog-level inheritance doesn't apply at all — every table needs to be configured on its own, full stop.Checking what's actually happening
aws glue get-table-optimizer \
--catalog-id 123456789012 \
--database-name analytics_db \
--table-name orders_table \
--type compactionaws glue list-table-optimizer-runs \
--catalog-id 123456789012 \
--database-name analytics_db \
--table-name orders_table \
--type compactionAnd if you're checking a bunch of tables at once — up to 20 per call:
aws glue batch-get-table-optimizer \
--entries '[
{"catalogId":"123456789012","databaseName":"analytics_db","tableName":"orders_table","type":"compaction"},
{"catalogId":"123456789012","databaseName":"analytics_db","tableName":"orders_table","type":"retention"},
{"catalogId":"123456789012","databaseName":"analytics_db","tableName":"orders_table","type":"orphan_file_deletion"}
]'Compaction runs report files and bytes compacted; retention runs report data files, manifest files, and manifest lists deleted. If you're managing more than a couple of tables, it's worth wiring these into a scheduled check that flags anything reporting
fail , since compaction going quiet after four failed attempts is easy to miss otherwise.Things that will actually get you
- Two tables sharing an S3 prefix is the real failure mode here — with optimizers on for both, one table's retention or orphan-deletion run can delete files the other still needs. There's no cross-table awareness built in, so this has to be enforced at the pipeline or infrastructure level, not just by convention.
- S3 lifecycle rules don't know anything about Iceberg snapshots. A generic bucket lifecycle policy can delete a manifest or data file an active snapshot still points to. If your bucket has lifecycle rules, exclude the Iceberg table paths.
- Retention and orphan deletion cap out at a million file deletions per run. On a table that's been neglected for a long time, one run won't clear the backlog — expect it to take a few passes before it catches up.
- Z-order doesn't support
DecimalorTimestampWithoutZonecolumns. Worth checking before you commit to it as a strategy.
- Compaction doesn't work across accounts or regions, on resource links, or on S3 Express One Zone tables. Cross-account Iceberg setups need a different approach — a Glue ETL job calling Iceberg's own
rewrite_data_filesprocedure directly.
- If you hit
OversizedAllocationExceptionduring a compaction run, drop the vectorized reader's batch size via theread.parquet.vectorization.batch-sizetable property (default 5000), and disableread.parquet.vectorization.enabledentirely if that's still not enough.
Where to start
If you're not running any of this yet, don't try to roll it out everywhere at once. Pick the one table everyone already complains about, give it its own S3 prefix if it doesn't have one, set up the IAM role, and turn compaction on first. Measure the query time before and after — that's usually the number that gets the rest of the rollout approved.
References
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article