AWS Builder Center

Builder Center

Your Data Lake Can Move from Hive to Iceberg Without Building a Migration Pipeline

On November 4, 2026, AWS introduced managed Hive-to-Iceberg migration for Amazon S3 Tables. It validates, converts, copies, and synchronizes existing datasets while production workloads continue running. Here's how the new workflow works and why it matters for data engineering.

The problem: Your data works, but your data lake is getting harder to manage.
Imagine an e-commerce company storing five years of transaction data in Amazon S3.
Its analytics team uses:
  • Apache Hive tables
  • Parquet files
  • AWS Glue Data Catalog
  • Amazon Athena
  • Spark-based processing jobs
Everything works.
But the company wants modern table capabilities:
  • Schema evolution
  • ACID transactions
  • Time travel queries
  • Better partition management
  • More efficient table maintenance
This is where Apache Iceberg comes in.
The problem is that moving an existing data lake to Iceberg isn't simply changing a file format.
You need to handle metadata, data types, compatibility, file organization, validation, and production cutover.
That can become a separate engineering project.
AWS has now made a large part of that migration a managed operation.

What's new in AWS?

On November 4, 2026, AWS announced managed migration of Apache Hive tables to Apache Iceberg in Amazon S3 Tables.
The new capability handles the migration workflow instead of requiring teams to build their own conversion infrastructure.
It supports:
  • Data validation
  • Compatibility checks
  • Data type conversions
  • File recompression
  • File compaction
  • Migration reports
  • Repeated synchronization before cutover
The important part is that production workloads can continue writing to the source table while the destination is being prepared.

First, understand the difference

Traditional Hive table

A typical Hive table in S3 looks like this:
text
Amazon S3
    |
    v
Parquet Files
    |
    v
AWS Glue Data Catalog
    |
    v
Hive Table
    |
    v
Athena / Spark Queries
Hive provides a way to define and manage tables over files.
But traditional file-based table management can become difficult when datasets grow, schemas evolve, and multiple applications read and write the same data.

Apache Iceberg table

Iceberg adds a metadata layer for managing analytical datasets.
text
S3 Table Bucket
       |
       v
Apache Iceberg Table
       |
       +---- Table Metadata
       |
       +---- Snapshots
       |
       +---- Schema History
       |
       +---- Data Files
       |
       v
Athena / Spark / Other Compatible Engines
This makes it possible to manage table changes more reliably.
For example, you can evolve schemas, change partition strategies, and query historical table snapshots.

How the new migration workflow works

Here's the interesting part.
AWS S3 Tables manages the migration through a staged process.
text
       EXISTING HIVE TABLE
                |
                v
        AWS Glue Data Catalog
                |
                v
        START MIGRATION
                |
                v
       COMPATIBILITY CHECK
                |
                v
         DATA VALIDATION
                |
                v
        COPY + CONVERSION
                |
                v
        DESTINATION TABLE
          (READ ONLY)
                |
                v
        SYNCHRONIZE CHANGES
                |
                v
         FINAL CUTOVER
                |
                v
        ICEBERG TABLE
          (READ / WRITE)
Let's break down the stages.

01. Start with your existing table

You select an eligible Hive table registered in AWS Glue Data Catalog.
The destination is an Amazon S3 table bucket.
You also choose the destination namespace and table name.

02. Compatibility checks

Before copying data, S3 Tables identifies the source objects and generates a compatibility report.
This helps identify issues before the migration proceeds.

03. Data conversion

S3 Tables manages tasks such as:
  • Converting supported data types
  • Recompressing files
  • Compacting files
  • Creating the Iceberg table structure
For example, when source files use Snappy compression, AWS can recompress them using Zstandard (ZSTD).

04. Synchronize ongoing changes

This is one of the most useful parts.
Your original Hive table can continue receiving writes during migration.
You can synchronize the destination repeatedly. Each synchronization copies only the changes since the previous sync.
That means you don't have to freeze your entire data platform while the initial migration runs.

05. Complete the migration

Once the destination is synchronized and validated, you complete the migration.
The destination table transitions from read-only to read-write.
There is still a short cutover window, so this should be planned rather than treated as a zero-downtime operation.

A practical example: E-commerce analytics

Imagine an online shopping platform with millions of order records.
Its existing data lake contains:
Dataset
Example data
Orders
Order ID, customer ID, amount
Products
Product ID, category, price
Payments
Transaction ID, payment status
Customers
Customer ID, location
Events
Clicks, page views, sessions
The analytics team wants to modernize these datasets.
Instead of building and operating a custom migration pipeline for every table, it can use the managed S3 Tables migration workflow for eligible Hive tables.

Before

text
Orders Hive Table
       |
       v
Parquet Files in S3
       |
       v
Glue Data Catalog
       |
       v
Athena Queries

After

text
Orders Iceberg Table
       |
       v
Amazon S3 Tables
       |
       +---- Snapshots
       |
       +---- Schema Evolution
       |
       +---- Table Maintenance
       |
       v
Athena / Spark / Other Iceberg Engines
The company can then use the new Iceberg table for analytical workloads while keeping the data accessible through compatible query engines.

Why Apache Iceberg matters

Iceberg isn't just another file format.
It is an open table format that manages how data files and table metadata work together.
Here are four important capabilities:
Feature
Why it matters
ACID transactions
Reliable table updates
Schema evolution
Add or modify columns without rebuilding everything
Hidden partitioning
Query data without depending on physical partition details
Time travel
Query earlier snapshots of a table
For example, suppose your orders table originally has:
text
order_id
customer_id
amount
Later, the business wants to add:
text
payment_method
With Iceberg, supported schema changes can be managed at the table level without forcing every downstream query to be rewritten.

What does this mean for cost?

AWS says the managed migration itself has no separate Hive-to-Iceberg conversion charge.
You pay for the data copied by the migration job.
There is also an important detail for S3 Intelligent-Tiering users: source objects remain in their current access tier.
This does not mean the entire data platform is free.
Storage, requests, queries, and other services can still incur their normal charges.

Can you automate it?

Yes.
AWS provides migration APIs, AWS CLI and SDK support.
The basic workflow follows this sequence:
text
1. Start migration
        |
        v
2. Review compatibility report
        |
        v
3. Monitor migration
        |
        v
4. Synchronize source changes
        |
        v
5. Validate destination
        |
        v
6. Complete migration
You can initiate a migration through the AWS CLI:
Bash
aws s3tables start-table-migration \
  --cli-input-json file://migration-plan.json
The migration plan contains details such as:
  • Source Glue table
  • Destination S3 table bucket
  • Namespace and table name
  • IAM role
  • Encryption configuration
  • Data conversion settings
  • Migration report location
The exact plan depends on your source table and destination configuration.
AWS also provides an Agent Toolkit skill called  migrating-hive-tables-to-s3-tables .
It can help a coding agent analyze IAM policies, validate a migration plan, execute the migration stages, and monitor progress.
The agent can assist with orchestration, but migration validation and production approval should remain part of your engineering workflow.

Important limitations to know

This is a new capability, not a universal migration tool for every data lake.
At launch, the supported scope is:
Requirement
Supported configuration
Source format
Apache Hive
File format
Parquet
Catalog
AWS Glue Data Catalog
Account
Same AWS account
Region
Same AWS Region
Destination
Amazon S3 Tables
It is available in selected AWS Regions, including Asia Pacific (Mumbai).
Check the latest AWS documentation before planning a production migration, especially if your tables use formats or catalog configurations outside this supported scope.

The bigger engineering lesson

Data modernization is not only about choosing a better technology.
The difficult part is moving existing production workloads without breaking the systems that already depend on them.
With managed Hive-to-Iceberg migration, AWS is reducing the amount of custom infrastructure teams need to operate for this particular migration.
The interesting workflow is:
Validate → Migrate → Sync → Verify → Cut over.
And that is a much better starting point than building an entirely separate migration pipeline for every table.


Further Reading

  1.  Amazon S3 Tables now offer managed migration of Apache Hive tables to Apache Iceberg — AWS News Blog 
  1.  Amazon S3 Tables managed migration — AWS announcement 
  1.  Working with Amazon S3 Tables — AWS Documentation 
  1.  Apache Iceberg — Official Documentation 
  1.  Amazon Athena — AWS Documentation 
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article