AWS Builder Center
Taming Iceberg Table Sprawl with AWS Glue Table Optimizers

Taming Iceberg Table Sprawl with AWS Glue Table Optimizers

AWS Glue's Iceberg table optimizers quietly fix tiny files, dead snapshots, and orphaned data. Here's how to turn them on with IAM policy and CLI calls.

Senior Data & AI Engineer

Taming Iceberg Table Sprawl with AWS Glue Table Optimizers

I got pulled into a support thread a while back where a team's Athena query on an "orders" table had gone from a few seconds to over a minute, and nobody had touched the query itself. Turned out the table had accumulated close to 40,000 files in a single partition, most of them a few KB each, from a Kafka Connect job that had been quietly committing small batches for months. Nobody had a compaction job running against it. Nobody had thought about it since the pipeline went live.
That's basically the point of this post. If you're running Apache Iceberg on the Glue Data Catalog and you're not using AWS Glue's table optimizers, you probably have some version of this problem already — you just haven't gone looking.

Why this happens

Iceberg tables age in three ways that nobody notices day to day:
Small files pile up. Every micro-batch write — Kinesis, Kafka Connect, a Glue streaming job, Flink, whatever's feeding the table — adds a new Parquet file. A few hundred writes a day over a few months and a partition that should have a few hundred well-sized files ends up with tens of thousands of tiny ones. More files means more S3 GET/LIST calls and more metadata for the query engine to chew through before it even reads a row.
Snapshots never expire. Iceberg gives you time travel by keeping a version history — every commit is a new snapshot. Nobody sets an expiration policy by default, so the history just grows, and so does the storage held by data files that only exist because an old snapshot still points at them.
Orphan files accumulate. A failed job, a retried write, a dropped partition — these leave files sitting in S3 that the table's metadata no longer references. They don't show up in query results, but they're still billed.
AWS Glue has a managed answer to all three, and it's genuinely low-effort to turn on. That's the part worth writing up, because the actual setup — the IAM policy, the exact CLI calls, where people get it wrong — isn't laid out anywhere in one place.


What you're actually turning on

Three separate features, each independently switchable per table:
Compaction merges small files into properly sized ones. The default strategy is binpack — just merge, no reordering. You can also ask it to sort by a column, or use Z-order across multiple columns if your query patterns filter on more than one field at a time.
Snapshot retention expires old snapshots past a retention window or count, and — if you tell it to — deletes the data files that only those expired snapshots were holding onto.
Orphan file deletion scans a table's S3 location for files the Iceberg metadata doesn't know about, and removes the ones old enough to be safe to delete.
One thing worth flagging up front: this is an Iceberg-only feature. It doesn't apply to Hudi or Delta Lake tables in the Glue Catalog, and compaction specifically only handles Parquet. If you're on a different format, this whole feature set isn't for you.
They all run as managed background jobs — Glue schedules and scales them, you don't spin up a cluster or write a Spark job. You point an IAM role at the table and that's most of the work.

Setting up the IAM role

This is the part that actually takes effort, and it's worth getting right once.
Trust policy, so Glue can assume the role:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": { "Service": "glue.amazonaws.com" },
"Action": "sts:AssumeRole"
}
]
}
Permissions policy for S3, Glue metadata, and logging:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": ["s3:PutObject", "s3:GetObject", "s3:DeleteObject"],
"Resource": ["arn:aws:s3:::your-iceberg-bucket/orders_table/*"]
},
{
"Effect": "Allow",
"Action": ["s3:ListBucket"],
"Resource": ["arn:aws:s3:::your-iceberg-bucket"]
},
{
"Effect": "Allow",
"Action": ["glue:UpdateTable", "glue:GetTable"],
"Resource": [
"arn:aws:glue:us-east-1:123456789012:table/analytics_db/orders_table",
"arn:aws:glue:us-east-1:123456789012:database/analytics_db",
"arn:aws:glue:us-east-1:123456789012:catalog"
]
},
{
"Effect": "Allow",
"Action": ["logs:CreateLogGroup", "logs:CreateLogStream", "logs:PutLogEvents"],
"Resource": [
"arn:aws:logs:us-east-1:123456789012:log-group:/aws-glue/iceberg-compaction/logs:*",
"arn:aws:logs:us-east-1:123456789012:log-group:/aws-glue/iceberg-retention/logs:*",
"arn:aws:logs:us-east-1:123456789012:log-group:/aws-glue/iceberg-orphan-file-deletion/logs:*"
]
}
]
}
A few things I'd have wanted someone to tell me before I set this up myself:
Scope that S3 resource to the table's own prefix, not the bucket root. It's not just tidiness — orphan file deletion scans everything under the table's configured location, so if two tables happen to share a prefix, one table's optimizer can delete the other table's files. This is the one real way to lose data with this feature, and it's entirely avoidable by giving every table its own path.
If the table's data is registered with Lake Formation, you don't use raw S3 IAM permissions at all — swap them for  lakeformation:GetDataAccess  and the relevant Lake Formation grants (ALTER, DESCRIBE, INSERT, DELETE) on the table.
If the bucket uses SSE-KMS, the role needs  kms:Decrypt  and  kms:GenerateDataKey , but that grant has to live on the KMS key policy, not just the IAM role. Miss this and compaction just fails quietly — no obvious "access denied" pointing at KMS, just a failed run in the logs.
Also add  iam:PassRole  on the optimizer role, granted to whatever principal is going to call  create-table-optimizer  — otherwise that call itself gets rejected.
Orphan file deletion doesn't need  glue:UpdateTable  or  s3:PutObject  — it only reads and deletes. If you want to keep its blast radius smaller, give it a separate, narrower role instead of reusing the compaction role for everything.

Turning it on

Using a table  analytics_db.orders_table  in account  123456789012 , region  us-east-1 , with a role  arn:aws:iam::123456789012:role/iceberg-optimizer-role .


Compaction

aws glue create-table-optimizer \
--catalog-id 123456789012 \
--database-name analytics_db \
--table-name orders_table \
--table-optimizer-configuration '{
"roleArn": "arn:aws:iam::123456789012:role/iceberg-optimizer-role",
"enabled": true,
"compactionConfiguration": {
"icebergConfiguration": { "strategy": "binpack" }
}
}' \
--type compaction
Once this is on, Glue watches the table itself — it kicks off a run when a partition crosses 100 files where each one is under 75% of the target file size (512 MB by default, from the  write.target-file-size-bytes  table property). There's nothing to schedule.
On strategy: stick with  binpack  unless you have a specific reason not to. Switch to  sort  if your queries almost always filter on one particular column — it clusters similar values together so the engine can skip more files. Reach for  zorder  only if queries commonly filter across two or more columns at once, since it weights all of them evenly instead of favoring one. It won't work on  Decimal  or  TimestampWithoutZone  columns, so check your schema before picking it.
If compaction fails four runs in a row, Glue suspends the optimizer on its own rather than keep burning compute on something broken. It's worth knowing that threshold exists so a string of failures doesn't just look like the feature "stopped working" — check the CloudWatch log group for the real error, fix it, then re-enable.

Snapshot retention

aws glue create-table-optimizer \
--catalog-id 123456789012 \
--database-name analytics_db \
--table-name orders_table \
--table-optimizer-configuration '{
"roleArn": "arn:aws:iam::123456789012:role/iceberg-optimizer-role",
"enabled": true,
"retentionConfiguration": {
"icebergConfiguration": {
"snapshotRetentionPeriodInDays": 7,
"numberOfSnapshotsToRetain": 3,
"cleanExpiredFiles": true
}
}
}' \
--type retention
 snapshotRetentionPeriodInDays  defaults to 5 and controls how far back you can time-travel.  numberOfSnapshotsToRetain  defaults to 1 and acts as a floor — Glue keeps at least that many snapshots no matter how old they are.  cleanExpiredFiles  defaults to true and is what actually frees up S3 storage; setting it to false just removes the metadata pointer and leaves the underlying files behind as orphans for the other optimizer to eventually pick up.
The one thing that confuses people: a snapshot only gets deleted once it's both past the retention window and beyond the minimum count. Set the minimum too high relative to your write frequency and you'll be staring at snapshots that look expired but aren't going anywhere. Also, if the table uses Iceberg branches or tags, their own retention rules take precedence over this config, so don't be surprised when a tagged snapshot outlives your retention window.

Orphan file deletion

aws glue create-table-optimizer \
--catalog-id 123456789012 \
--database-name analytics_db \
--table-name orders_table \
--table-optimizer-configuration '{
"roleArn": "arn:aws:iam::123456789012:role/iceberg-optimizer-role",
"enabled": true,
"orphanFileDeletionConfiguration": {
"icebergConfiguration": {
"orphanFileRetentionPeriodInDays": 3,
"location": "s3://your-iceberg-bucket/orders_table/"
}
}
}' \
--type orphan_file_deletion
It only ever deletes files created after the optimizer itself was enabled, so switching this on for an existing table is safe — anything already sitting there is left alone. The  location  field can narrow the scan to a sub-path if the table has nested locations you want excluded, though the better fix is making sure table locations don't overlap in the first place. Left on its default, it runs about once a day.

If you're rolling this out to a lot of tables

Configuring optimizers table by table is fine for a handful of tables, not for a whole data lake. AWS Glue also supports catalog-level defaults, so new and updated Iceberg tables pick up compaction, retention, and orphan deletion automatically without anyone touching them individually. You set this up through the Lake Formation console's "Table optimization" option on the default catalog, or the  UpdateCatalog  API — whoever runs it needs  glue:UpdateCatalog  or Lake Formation's  ALTER CATALOG  permission.
A few things worth knowing before you flip that switch: table-level settings always beat the catalog default, so a table that already has its own config is untouched by anything you change at the catalog level, including disabling it. Tables without their own config are supposed to inherit a catalog-level disable too, but there's a known AWS issue where that inheritance doesn't always take — don't just trust the toggle, check  get-table-optimizer  on a sample of tables to confirm what's actually running. And if Data Catalog metadata encryption is turned on, catalog-level inheritance doesn't apply at all — every table needs to be configured on its own, full stop.

Checking what's actually happening

aws glue get-table-optimizer \
--catalog-id 123456789012 \
--database-name analytics_db \
--table-name orders_table \
--type compaction
aws glue list-table-optimizer-runs \
--catalog-id 123456789012 \
--database-name analytics_db \
--table-name orders_table \
--type compaction
And if you're checking a bunch of tables at once — up to 20 per call:
aws glue batch-get-table-optimizer \
--entries '[
{"catalogId":"123456789012","databaseName":"analytics_db","tableName":"orders_table","type":"compaction"},
{"catalogId":"123456789012","databaseName":"analytics_db","tableName":"orders_table","type":"retention"},
{"catalogId":"123456789012","databaseName":"analytics_db","tableName":"orders_table","type":"orphan_file_deletion"}
]'
Compaction runs report files and bytes compacted; retention runs report data files, manifest files, and manifest lists deleted. If you're managing more than a couple of tables, it's worth wiring these into a scheduled check that flags anything reporting  fail , since compaction going quiet after four failed attempts is easy to miss otherwise.

Things that will actually get you

  • Two tables sharing an S3 prefix is the real failure mode here — with optimizers on for both, one table's retention or orphan-deletion run can delete files the other still needs. There's no cross-table awareness built in, so this has to be enforced at the pipeline or infrastructure level, not just by convention.
  • S3 lifecycle rules don't know anything about Iceberg snapshots. A generic bucket lifecycle policy can delete a manifest or data file an active snapshot still points to. If your bucket has lifecycle rules, exclude the Iceberg table paths.
  • Retention and orphan deletion cap out at a million file deletions per run. On a table that's been neglected for a long time, one run won't clear the backlog — expect it to take a few passes before it catches up.
  • Z-order doesn't support  Decimal  or  TimestampWithoutZone  columns. Worth checking before you commit to it as a strategy.
  • Compaction doesn't work across accounts or regions, on resource links, or on S3 Express One Zone tables. Cross-account Iceberg setups need a different approach — a Glue ETL job calling Iceberg's own  rewrite_data_files  procedure directly.
  • If you hit  OversizedAllocationException  during a compaction run, drop the vectorized reader's batch size via the  read.parquet.vectorization.batch-size  table property (default 5000), and disable  read.parquet.vectorization.enabled  entirely if that's still not enough.

Where to start

If you're not running any of this yet, don't try to roll it out everywhere at once. Pick the one table everyone already complains about, give it its own S3 prefix if it doesn't have one, set up the IAM role, and turn compaction on first. Measure the query time before and after — that's usually the number that gets the rest of the rollout approved.



References

Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article