One Iceberg Table for Ninety Sites, on S3 Tables

For a few years the IIoT platform I work on stored sensor readings the obvious way. Every site got its own S3 bucket, its own Glue database, its own Firehose, its own crawler and its own set of IAM roles. Terraform made that cheap to stamp out ninety times. It did not make it cheap to use. Any question that spanned the fleet meant a job that looped over ninety sites, or a UNION ALL over ninety tables, and every consuming team kept its own copy of the site list that went stale the moment a site was added.

This year we replaced all of that with one Apache Iceberg table per region on Amazon S3 Tables. Three tables, three regions, ninety sites, one WHERE clause. I wrote the whole journey up as a six part series, and this post is the short version so you can decide whether the long one is worth your time.

Sites publish readings through a Kinesis stream in their region. Firehose reads the stream, runs a small Go Lambda that maps every payload onto one schema, and commits the batch straight into the Iceberg table. The sensor_readings table has eight columns, is partitioned by site_id and by day of the measurement time, and is sorted by sensor inside each partition. A single site, single day query prunes about 99 percent of the files before reading a byte. Compaction and snapshot expiry are done by S3 Tables, so there is nothing to crawl and nothing to schedule.

Consumers never touch the bucket. They read through Lake Formation. Fleet wide consumers get the whole table. A partner that only needs a handful of sites gets a data cells filter on the same table, site_id IN (...), created from a list that Terraform already knew about, so they see their sites and nothing else. Adding a consumer is one entry in a map.

The sharing part cost us a month and an AWS support case. An S3 Tables catalog in Glue is a federation layer. When a consumer account queries a shared table, every call is authorised twice, once by Lake Formation and once more at the S3 Tables layer with the consumer's own identity. So the consumer account needs a table bucket policy on the producer side, seven s3tables actions including GetTableData, which happens to be a data plane action whose denials never show up in CloudTrail event history.

That policy on its own lets a consumer read around Lake Formation. The fix AWS suggested was an aws:CalledVia condition on Glue. That broke Athena, because at query time Lake Formation itself makes the S3 Tables calls under lakeformation.amazonaws.com. The two service condition is now confirmed by the service team as the complete, supported set for Athena and Glue ETL, and a documentation update is pending.

A few smaller things caught us too. IAM_ALLOWED_PRINCIPALS on the table silently defeats a version 5 cross account share. A shared data cells filter is not visible in the consumer account, so the consumer grants SELECT on the table and Lake Formation intersects it with the filter. And S3 Tables rejects Requester Pays, so the Athena workgroup must have it off.

Ninety stacks became three. Fleet questions became a WHERE clause. Freshness went from next day to about five minutes. The jobs that stitched ninety buckets together have nothing left to do once the history is loaded. And onboarding a new site or a new consumer is now a Terraform change that a reviewer can read in a minute.

Each part stands on its own, but they build on each other.

Everything is Terraform, and the patterns are not specific to our platform. If you run any multi tenant IoT platform on AWS and are staring at a bucket per tenant, I hope this saves you the month.