AWS Builder Center

Builder Center

Build Lakehouse architecture on AWS with S3 Iceberg, AWS Services and Snowflake

A Transactional data lake on S3 using Apache Iceberg

Harpreet Dhanoa

A transactional data lake on S3 using Apache Iceberg tables registered in AWS Glue Data Catalog and governed by Lake Formation, consumable from Athena / Redshift and Snowflake (Iceberg Tables).
1) Core building blocks
  • Storage: Amazon S3 buckets (raw/curated/consumption) with versioning + default SSE-KMS.
  • Table format: Apache Iceberg for ACID, schema/partition evolution, and time travel on S3. Athena, Glue, EMR, and Redshift support it natively
  • Catalog & governance: AWS Glue Data Catalog (+ Lake Formation for fine-grained access). Glue can read/write Iceberg; Studio/Jobs show how to set it up.
  • Query engines (AWS-native):
  • Athena (read/write, DDL, time travel on Iceberg).
  • Redshift via Spectrum/Serverless querying Iceberg tables cataloged in Glue.
  • Snowflake: Iceberg Tables over S3 using external volumes + catalog integration with Glue (so Snowflake uses your Glue catalog + S3 data).

2) Reference layout (recommended)
Buckets
  • s3://company-lake/raw/...
  • s3://company-lake/curated/... (Iceberg tables live here)
  • s3://company-lake/consumption/...
Glue databases
  • raw_db, curated_db, consumption_db
Table format
  • All curated tables: Iceberg (Parquet files + Iceberg metadata).
Access
  • Lake Formation grants at database/table/column level; cross-account sharing where needed.

3) Step-by-step build
A) Foundation & security
  1. Create S3 buckets with versioning + default SSE-KMS.
  1. Enable Lake Formation and register the S3 locations; assign data permissions to producer/consumer roles.
  1. Create Glue databases (raw_db, curated_db).
Tip: Keep raw data immutable; all changes happen in curated Iceberg tables.
B) Ingestion → raw zone
  • Land data using DMS / DataSync / Kinesis / Firehose / Glue connectors into s3://.../raw/… with sensible partitioning (e.g., source=/dt=YYYY-MM-DD/).
  • Crawl or define schemas in Glue for discovery where helpful.
C) Curate into Iceberg (ETL/ELT)
  • Use AWS Glue (Spark) or EMR to transform from raw → curated.
  • In Glue Jobs, enable Iceberg support (parameter --datalake-formats=iceberg) and write to Glue-cataloged Iceberg tables in curated_db.
Example (Athena) to create an Iceberg table)
CREATE TABLE curated_db.orders_iceberg (
order_id bigint,
customer_id bigint,
amount decimal(12,2),
order_ts timestamp
)
PARTITIONED BY (date_trunc('day', order_ts))
LOCATION 's3://company-lake/curated/orders/'
TBLPROPERTIES (
'table_type'='ICEBERG',
'format'='PARQUET'
);
Athena supports DDL, writes, and time travel on Iceberg.
D) Query & serve (AWS-native)
  • Athena: interactive SQL, CTAS/INSERT/DELETE/MERGE, and time travel for audits/rollback.
  • Redshift Spectrum/Serverless: create an external schema pointing to Glue; query Iceberg alongside warehouse tables.
E) Wire up Snowflake to the same Iceberg
  1. In Snowflake, configure an EXTERNAL VOLUME for your S3 bucket/prefix that holds the Iceberg data/metadata.
  1. Create a CATALOG INTEGRATION to AWS Glue (Snowflake → Glue Data Catalog).
  1. CREATE ICEBERG TABLE … USING CATALOG GLUE to register the existing Iceberg table in Snowflake (no copying; it references S3 + Glue).
Snowflake’s official guide + AWS blog show this pattern (S3 + Glue catalog + Snowflake Iceberg) and note the pruning/metadata efficiency benefits.

4) Governance & sharing
  • Lake Formation is your policy source of truth for AWS consumers (Athena/Redshift/EMR/Glue).
  • For Snowflake, control S3 access via external volume IAM roles and rely on Glue catalog for consistency.
  • Tag data domains (PII, PCI, region) and apply column/row filters in Lake Formation.
  • Use Athena time travel on Iceberg for audits; keep CloudTrail for admin actions.

5) Performance & cost patterns
  • Iceberg gives snapshot isolation, partition & schema evolution, concurrent writers, and time travel—ideal for multi-engine access. Athena explicitly supports it; Redshift queries via Spectrum; Snowflake reads via Iceberg Tables. ( AWS Documentation )
  • Optimize S3 layout: partition pruning, Z-ordering/equality deletes (where supported), and compaction jobs to right-size files.
  • Athena/Redshift: use Glue statistics + partition metadata; schedule Iceberg compaction.
  • Snowflake on Iceberg: monitor file counts/sizes and metadata refresh cadence; follow Snowflake guidance for external volumes.

6) Minimal command checklist (you can adapt)
(Athena) Create Iceberg + insert
CREATE TABLE curated_db.customers_iceberg (...)
TBLPROPERTIES ('table_type'='ICEBERG');
INSERT INTO curated_db.customers_iceberg
SELECT ... FROM raw_db.customers_raw;
(Redshift) External schema to Glue
CREATE EXTERNAL SCHEMA lake_curated
FROM DATA CATALOG
DATABASE 'curated_db'
IAM_ROLE 'arn:aws:iam::<acct>:role/RedshiftSpectrumRole'
CREATE EXTERNAL DATABASE IF NOT EXISTS;
(Snowflake) Glue catalog integration → create Iceberg table reference
CREATE CATALOG INTEGRATION glue_int TYPE = GLUE
ENABLED = TRUE
STORAGE_ALLOWED_LOCATIONS = ('s3://company-lake/curated/');
CREATE OR REPLACE ICEBERG TABLE curated.orders
CATALOG='glue_int'
EXTERNAL_VOLUME='company_lake_ext_vol'
BASE_LOCATION='s3://company-lake/curated/orders/';

7) When to choose which engine
  • Athena: ad-hoc SQL, serverless analytics, ops simplicity.
  • Redshift: integrated BI with MPP performance; join Iceberg + warehouse tables.
  • Snowflake (Iceberg Tables): cross-org collaboration, Snowflake features (governance, SQL, tasks) while reading the same S3 Iceberg without duplicating data.

8) Common pitfalls (and fixes)
  • Mismatched catalogs: Keep Glue as the system of record for Iceberg; ensure Snowflake’s catalog integration points to the same DB/table.
  • Small/fragmented files: schedule compaction (Iceberg rewrite) to maintain query performance.
  • Permissions drift: Manage S3/Lake Formation/IAM consistently; test access with service-linked roles.
  • Metadata refresh: In Snowflake, refresh Iceberg metadata on schema changes or heavy writes.

9) Build recipe
  1. S3 buckets + Lake Formation + Glue DBs
  1. Land raw → transform with Glue/EMR → write Iceberg to curated_db
  1. Query with Athena (DDL, writes, time travel) and Redshift Spectrum
  1. Configure Snowflake external volume + Glue catalog integration → CREATE ICEBERG TABLE (no data copy)
  1. Add compaction, stats, and LF governance + cost/usage dashboards
10) Resources
 https://docs.aws.amazon.com/prescriptive-guidance/latest/apache-iceberg-on-aws/introduction.html 
 https://docs.snowflake.com/en/sql-reference/sql/create-iceberg-table-aws-glue 
 Using Apache Iceberg tables with Amazon Redshift - Amazon Redshift 
 CREATE ICEBERG TABLE (AWS Glue as the Iceberg catalog) | Snowflake Documentation 
 Query Apache Iceberg tables - Amazon Athena 
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article