A Beginner’s Guide to Apache Iceberg with Amazon S3 Tables
Learn Apache Iceberg and Amazon S3 Tables with this beginner's guide. Create real Iceberg tables, load data, query, update, and evolve schemas using SageMaker Unified Studio and Athena Spark. Discover ACID guarantees, time travel capabilities, and schema evolution without file rewrites. Perfect for analytics workloads requiring transactional reliability, point-in-time accuracy, and multi-engine support on S3.
If you have never worked with Apache Iceberg or Amazon S3 Tables before, this guide is a practical starting point. You will not need prior experience with either technology. Instead, you will build up a working understanding by doing. You will create a real Iceberg table, load data into it, query it, make corrections, and watch schema changes take effect, all from a single notebook in SageMaker Unified Studio.
Each step is explained in plain terms so you can see not just what to run, but why it works the way it does. The notebook uses SQL cells backed by Athena Spark, so if you are comfortable writing SQL you already have most of what you need to follow along.
So if you have been curious about Apache Iceberg or Amazon S3 tables, this is a good place to start.
About the Sample Data
For this example, we use a synthetic dataset modelled on Medicare Benefits Schedule (MBS) service records. The MBS is the Australian government schedule of medical services that are subsidized under Medicare, covering everything from general practitioner consultations to specialist visits, diagnostic imaging and pathology.
Each record in our dataset represents an aggregated set of services for a given item number over a period of time, and includes metrics like the number of services rendered, the total benefit paid by Medicare, the scheduled fee, etc.
This kind of data is a good fit for the techniques in this guide because it has the characteristics you see in many real government and health analytics workloads. It arrives in periodic extracts, corrections and revisions are common, reporting requirements often demand point-in-time accuracy, and the schema tends to grow over time as new policy dimensions are added.
Note: If you work outside health, you can substitute your own domain, but the MBS structure maps cleanly onto patterns you would find in insurance claims, billing records, program expenditure data and similar datasets.
Now, let’s start from the beginning…
What is Apache Iceberg?
Apache Iceberg is an open table format that adds a structured, transactional layer on top of files stored in Amazon S3. If you are already using Parquet files on S3, you may be querying them today through tables defined in the AWS Glue Data Catalog, either created manually or discovered by a Glue crawler. That approach works, but it has its limits. Crawlers infer schema from files, there is no transaction support, and updating or deleting specific rows means rewriting entire files yourself.
Iceberg takes a different approach. Rather than inferring structure from files at query time, Iceberg maintains its own metadata that precisely describes which files belong to a table, what their schemas are, and what state the table was in at each point in time. That metadata lives alongside your data files in S3 and is updated atomically as changes are made. Engines such as Amazon Athena, Amazon Redshift and Apache Spark read from that metadata to plan queries rather than scanning a prefix and guessing layout.
The practical result is that these Iceberg tables behave much more like a database table than a folder of Parquet files does. You get consistent reads, support for row‑level updates and deletes, schema evolution without file rewrites, and a snapshot history you can query.
Querying Iceberg with SQL
One of the most practical things about Iceberg is that it does not ask you to change how you work. If you already write SQL against tables in Athena, Redshift or Spark, you can query an Iceberg table backed by Amazon S3 Tables in exactly the same way. There is no new syntax to learn for basic operations. SELECT, INSERT, UPDATE and DELETE all work as you would expect, and the table appears in the catalog like any other table you have worked with before.
You can write SQL and queries the same way you would against a Glue catalog table backed by Parquet files.
The difference is not visible in the SQL itself, it shows up in what the table can do for you. You get consistent reads across concurrent writes, row‑level updates and deletes without file rewrites, and a snapshot history you can query when you need to. The SQL skills you already have get you further than they did with a table in a traditional data lake, without having to change your tooling or query patterns.
What is ACID and why does it matter?
ACID stands for Atomicity, Consistency, Isolation and Durability. These are the properties that relational databases use to ensure data stays reliable when things go wrong. Atomicity means a write either completes fully or not at all. Consistency means the data is always in a valid state. Isolation means concurrent reads and writes do not interfere with each other. Durability means a completed write stays completed. Standard S3 object storage was not designed with any of these properties in mind, which is what makes building reliable analytics on top of it harder than it looks.
The most common symptom of missing ACID guarantees is a job that fails halfway through writing a partition. The directory ends up with a mix of files from the previous run and the current one, downstream queries read all of them, and the results do not reflect any real state of the data. There is no transaction to roll back, so the fix is manual.
Iceberg restores these guarantees by treating every write as a snapshot, a precise record of exactly which files make up the table at that moment. When a write completes, the table moves atomically to the new snapshot. Readers always see a complete, consistent state. If a write fails, the snapshot does not advance and the table stays in the last known good state with nothing to clean up.
Dealing with Schema Evolution
If your tables are defined in the Glue Data Catalog, schema changes from source systems have historically required careful handling. Depending on the file format and query engine in use, adding a column or changing a data type can break pipelines, introduce unexpected nulls, or require manual intervention to keep table definitions in sync. Aligning historical partitions with a new schema has traditionally meant writing code to paper over inconsistencies or reprocessing large volumes of data.
Using Apache Iceberg changes this picture considerably. Iceberg tracks schema evolution as a first-class feature, recording column additions, renames, and compatible type changes in the table metadata rather than in the data files themselves. Engines like Athena, Spark, and Trino read that metadata and resolve the correct schema for each file transparently, so old data files remain untouched and pipelines continue to function without manual rewrites or table rebuilds.
Iceberg tracks columns by stable internal identifiers rather than by their position in the file. That means you can add, drop, rename and reorder columns without touching existing data files. Old files stay valid under the new schema. Queries written before the change continue to work, and queries written after can use the new columns straight away. Schema changes that previously required a maintenance window and a migration script become routine ALTER TABLE statements.
Built-in time travel
Audit requirements, regulatory queries and the need to reproduce past reports all create the same problem, in that you may need to see the data as it was at a specific point in time. With a conventional S3 data lake, the usual answer is to keep snapshots in separate prefixes or restore from backups, which adds storage costs and operational complexity.
Because Iceberg records a history of snapshots, time travel is a native capability of the table. You can query the snapshot history to see when changes were made and what operations created them. You can then query the table as of a timestamp or a specific snapshot identifier, targeting exactly the state that existed when a particular report was run or when a particular data issue was introduced. If a pipeline run introduces bad data, you can roll back to the last known good snapshot rather than restoring files from a backup.
Hidden partitioning
Partitioning in a Hive-style layout is a trade-off. It improves query performance by grouping related data into the same folders, but it also means partition values are encoded in directory names and users need to know to filter on them. Forget to include a partition column in a WHERE clause and you get a full table scan. Change the partitioning strategy and you typically need to rewrite historical data and update queries that relied on the old layout.
Iceberg stores partition transforms in metadata rather than in directory names. Users write queries against the logical columns in the table and Iceberg handles the partition pruning automatically. A query that filters on service_date benefits from date-based partitioning without needing to reference a year or month column that exists only for performance reasons. Partition strategies can also be changed going forward without requiring historical data to be rewritten or queries to be updated.
One table, many engines
Most data platforms are not built around a single engine. You might load and transform data using Spark or Glue, query it interactively with Athena, access it from Redshift for reporting and analytics and read it from Python notebooks for machine learning. With a traditional data lake, keeping all of those engines pointed at the same data and getting consistent results can require careful coordination. Engine-specific formats or catalogs sometimes lead to copies of the same data sitting in different places.
An Iceberg table can be read and written by multiple engines against the same physical files and the same metadata. You do not need to maintain separate copies for different workloads. A pipeline that writes through Spark creates a snapshot that Athena queries immediately, with the same schema and the same data, because they are both working from the same Iceberg metadata.
This multi-engine compatibility extends beyond the AWS ecosystem. Third-party platforms such as Databricks and Snowflake also support Apache Iceberg, which means teams that work across multiple platforms are not forced to maintain separate copies of the same data for different environments. For organizations that operate across more than one platform, or that are in the middle of a migration, the open nature of the Iceberg format means you can continue working with the tools your teams already use without being locked into a single engine or vendor.
Performance at scale
A common problem with traditional data lakes as they grow is that the number of small files increases, query planning slows down because engines have to list and evaluate large numbers of objects, and compaction jobs become a regular piece of maintenance work. At small scale this is manageable. At large scale it starts to consume engineering time that would be better spent elsewhere.
Iceberg addresses this with a layered metadata structure. Manifests describe which files belong to the table and include statistics such as min and max column values. Manifest lists describe which manifests are relevant to a given query. Engines use this hierarchy to skip files without reading them, which keeps query planning fast even when a table contains millions of files. Amazon S3 Tables runs background compaction automatically, which means small files are consolidated without you scheduling and monitoring that work yourself.
Introducing Amazon S3 Tables
Amazon S3 Tables is a managed service built specifically to deliver the Iceberg features described above without the operational overhead of running them yourself. You can run Iceberg on a standard S3 bucket today, but doing so at production scale means taking on responsibility for deploying and maintaining a catalog, scheduling compaction jobs, managing snapshot expiry, and ensuring that access policies are applied consistently across every table. For teams that are already managing pipelines and infrastructure, those tasks add up quickly and tend to consume engineering time that would be better spent on the data itself.
When you store tables in an S3 table bucket, the ACID guarantees, schema evolution, time travel snapshots, hidden partitioning, and multi-engine access all work as described throughout this post, because S3 Tables is an Iceberg-native service from the ground up. Rather than a generic S3 bucket with an Iceberg catalog layered on top, a table bucket is a purpose-built storage type that understands Iceberg tables as first-class objects. Tables have names, they appear in the catalog, and they integrate directly with AWS Lake Formation for access control. That means the same governance model you apply elsewhere in your data platform applies here too, without needing to configure a separate permissions layer or maintain a custom catalog alongside your data.
The performance and maintenance features that make Iceberg practical at scale also become automatic rather than something you need to schedule and monitor yourself. The background compaction that consolidates small files, the snapshot expiry that prevents metadata from growing unbounded over time, and the manifest management that keeps query planning fast even as tables grow to millions of files are all handled by the service. You do not need to write compaction jobs, set up schedulers, or build alerting around maintenance tasks that did not complete. S3 Tables runs that work in the background and keeps tables in good shape without any intervention required.
The practical result is a table that behaves like a database table, supports SQL from Athena, Redshift, and Spark without any changes to how you write queries, and stays performant and well-maintained as it grows.
Note: If you are already working with Parquet files on S3 and tables in the Glue Data Catalog, you can think of S3 Tables as the natural next step, one that gives you proper transactional semantics, consistent reads, and built-in governance, while your data stays in S3 and your existing SQL skills remain fully transferable.
Getting Hands-On with Apache Iceberg
To follow the rest of this guide we are going to use Amazon SageMaker Unified Studio (IAM Domain). SageMaker Unified Studio is a unified development experience that brings together AWS data, analytics, artificial intelligence (AI), and machine learning (ML) services. It provides a place to build, deploy, execute, and monitor workflows from a single interface.
Note: You will need access to an AWS account, and If this is your first time using SageMaker Unified Studio (IAM Domain), you may need to do some setup . If you are in a managed environment, your AWS administrator should be able to help with access and setup.
Prerequisite: Create an Amazon S3 Table bucket
Before we can get started, you will need to create a table bucket that will hold your example table. In the AWS Console , Open the Amazon S3 console and go to the table buckets section. Create a table bucket named health‑data, using configuration options that match your environment. After the table bucket is created it appears as part of an S3 Tables catalog that SageMaker Unified Studio can use. (If it doesn't, ask your AWS administrator to check the permissions that were used to setup SageMaker Unified Studio)
With the table bucket ready, we are going to move to SageMaker Unified Studio and create a new notebook. From the AWS Console, search for Amazon SageMaker and from the the SageMaker home page, open SageMaker Unified Studio.
From the SageMaker Unified Studio home page, select Build in the notebook to open a blank notebook. Notebooks are made up of individual cells, each of which contains a small piece of code or a query. For the examples in this post you will use the SQL cell type, configured to run on Athena Spark. This gives you serverless Spark compute behind a familiar SQL interface, with no cluster setup required.
Step 1: Configure a SQL cell for Athena Spark
In your new notebook, add a cell and set its type to SQL. From the cell’s connection selector, choose the Athena Spark connection associated with your project or workgroup.
Once selected, SQL you run in that cell is executed on Athena Spark and can access catalogs that include your S3 Tables. You do not need to create a Spark session manually; the connection handles that behind the scenes.
Step 2: Create the namespace in your table bucket
Create a namespace, similar to a database or schema, inside the health‑data table bucket. In a SQL cell connected to Athena Spark, run:
CREATE NAMESPACE IF NOT EXISTS "health-data"."mbs";This creates a namespace called mbs under the health‑data table bucket. The IF NOT EXISTS clause makes it safe to run this statement multiple times.
Step 3: Create the Iceberg table
Next, create an Iceberg table for your MBS service records. This table includes identifiers, dates, item codes, descriptions, provider state, age group and financial amounts. In a new SQL cell using the same Athena Spark connection, run:
CREATE TABLE IF NOT EXISTS "health-data"."mbs"."mbs_services" (
record_id BIGINT,
service_date DATE,
item_number VARCHAR,
item_description VARCHAR,
provider_state VARCHAR,
patient_age_group VARCHAR,
services_rendered BIGINT,
benefit_paid_aud DOUBLE,
schedule_fee_aud DOUBLE,
patient_out_of_pocket_aud DOUBLE
)
USING iceberg;This statement registers mbs_services as an Iceberg table in the S3 Tables catalog associated with your health‑data bucket. Athena Spark recognizes it as an Iceberg table and uses Iceberg’s metadata for planning and execution.
Step 4: Insert sample records with Athena Spark
With the table defined, insert some sample rows to work with. In another SQL cell on Athena Spark, run:
INSERT INTO "health-data"."mbs"."mbs_services" VALUES
(1001, DATE '2024-07-15', '23', 'Professional attendance, Level B',
'NSW', '45-54', 3840, 158880.00, 195000.00, 36120.00),
(1002, DATE '2024-07-15', '57', 'Professional attendance, Level D',
'VIC', '65-74', 1205, 87481.75, 102425.00, 14943.25),
(1003, DATE '2024-08-01', '105', 'Brief health assessment, 75 years+',
'QLD', '75+', 620, 112990.00, 112990.00, 0.00),
(1004, DATE '2024-08-22', '2798', 'Cervical screening test',
'WA', '25-34', 980, 77912.00, 86240.00, 8328.00),
(1005, DATE '2024-09-10', '5000', 'Pathology request, single determinant',
'SA', '55-64', 4500, 108000.00, 120600.00, 12600.00);Athena Spark writes these records into the S3 Tables‑backed Iceberg table and creates a new snapshot that reflects the inserted data.
Step 5: Verify the table contents from a SQL cell
To check that everything worked as expected, run a simple query in a SQL cell:
SELECT *
FROM "health-data"."mbs"."mbs_services";
The results grid in the notebook should show the five sample records with the columns you defined. You have now created and populated an Iceberg table in an S3 table bucket, using only SQL cells on Athena Spark.
Step 6: Run an analytical query with Athena Spark
Use Athena Spark to run a more interesting analytical query over the same table. The following example summarizes activity and benefits by provider state for a financial year. In a SQL cell:
SELECT
provider_state,
COUNT(DISTINCT item_number) AS distinct_items,
SUM(services_rendered) AS total_services,
SUM(benefit_paid_aud) AS total_benefit_aud,
ROUND(AVG(patient_out_of_pocket_aud), 2) AS avg_out_of_pocket_aud
FROM
"health-data"."mbs"."mbs_services"
WHERE
service_date BETWEEN DATE '2024-07-01' AND DATE '2025-06-30'
GROUP BY
provider_state
ORDER BY
total_benefit_aud DESC;
Athena Spark uses Iceberg’s metadata and partitioning to prune unnecessary files and plan the query efficiently, while you stay in the notebook and work in SQL.
Step 7: Apply corrections with UPDATE from the notebook
Corrections are a normal part of working with operational data. Suppose the benefit and out‑of‑pocket amounts for one record need to be updated. In a SQL cell connected to Athena Spark, run:
UPDATE "health-data"."mbs"."mbs_services"
SET benefit_paid_aud = 115200.00,
patient_out_of_pocket_aud = 5400.00
WHERE record_id = 1005
AND service_date = DATE '2024-09-10';Athena Spark applies this update through Iceberg, creating a new snapshot while preserving the ability to query earlier snapshots if you need to reconstruct the previous state.
Step 8: Delete obsolete data with SQL
If some records should no longer be included in analysis, for example because they were superseded by later extracts, you can delete them directly from the table. In a SQL cell, run:
DELETE FROM "health-data"."mbs"."mbs_services"
WHERE service_date < DATE '2023-07-01'
AND item_number = '23';
This delete operation is handled transactionally by Iceberg. The table remains consistent for all readers as Athena Spark carries out the change.
Step 9: Inspect snapshot history for time travel
To see how the table has evolved, query the snapshot history using Athena Spark. In a SQL cell, run:
SELECT snapshot_id, committed_at, operation
FROM "health-data"."mbs"."mbs_services".snapshots
ORDER BY committed_at;The results show each snapshot’s identifier, commit time and operation, giving you a timeline of inserts, updates and deletes. Depending on your engine’s syntax you can then use these snapshot identifiers or timestamps to query the table as of a particular point or to roll back to a previous snapshot.
Step 10: Evolve the schema with ALTER TABLE
As your analytic needs expand, you may want to add new attributes such as remoteness category, Indigenous status or referral flags. You can extend the schema without rewriting historical data. In a SQL cell on Athena Spark, run:
ALTER TABLE "health-data"."mbs"."mbs_services"
ADD COLUMNS (
remoteness_category VARCHAR,
indigenous_status VARCHAR,
referred_by_gp BOOLEAN
);
After this statement completes, new data can populate these columns. Existing rows show null values until you choose to backfill them.
Step 11: Rename an existing column
If your data dictionary changes, you can rename columns in place. To rename patient_age_group to age_cohort, use:
ALTER TABLE "health-data"."mbs"."mbs_services"
RENAME COLUMN patient_age_group TO age_cohort;
Iceberg resolves this change at the metadata level, so the underlying files remain valid and queries can be updated to use the new column name when you are ready.
All from a single notebook
And remember, through the entire lifecycle of your Iceberg table runs from a single notebook using SQL cells configured to use Athena Spark. You create namespaces and tables, insert and query data, apply updates and deletes, inspect snapshot history and evolve the schema without switching consoles or maintaining separate scripts. The same notebook can also include Python cells for visualization and downstream processing, while continuing to use Athena Spark for large‑scale SQL operations.
By combining Apache Iceberg, Amazon S3 Tables and SageMaker Unified Studio (IAM domain) with SQL cells on Athena Spark, you get a lakehouse environment that behaves like a warehouse, keeps data in S3 and allows multiple engines to work from the same source of truth, all orchestrated from a single notebook.
Summary
By working through this guide you have built and operated a real Iceberg table on Amazon S3 Tables from a single SageMaker Unified Studio notebook. Starting from the basics of what Iceberg is and why ACID guarantees matter on object storage, you created a table bucket, defined a schema, loaded data, ran analytical queries, applied row-level updates and deletes, inspected the snapshot history, and evolved the schema, all without leaving the notebook or switching between consoles.
The techniques here are not specific to health data. The same patterns apply whenever you are working with data that arrives in periodic extracts, requires corrections over time, needs to support point-in-time reporting, or is consumed by more than one engine. If you are currently managing Parquet files on S3 with tables in the Glue Data Catalog, this guide shows a practical path toward a more reliable foundation without changing the way you write queries or the tools your team already uses.
The next steps from here depend on what matters most for your use case. Below are some additional resources to help further your learning about Apache Iceberg and Amazon S3 Tables.
Additional Resources:
Apache Iceberg
Amazon S3 Tables
Amazon SageMaker Unified Studio
Related AWS Blog Posts
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article