AWS Glue 6.0: Why the Safest Iceberg v3 Migration Starts with Compatibility, Not Features
AWS Glue 6.0 brings a 30% price reduction, Spark 4.1.1, Python 3.13, and full Apache Iceberg v3 support. This practical guide explains what changed and how to adopt the new runtime without creating cross-engine compatibility problems.
AWS Glue 6.0: A Safer Path to Modern Data Lakes
AWS Glue 6.0 is an important step forward for teams building modern data platforms. The release combines a 30% price reduction with a new runtime based on Apache Spark 4.1.1, Python 3.13, and Scala 2.13.17. It also expands support for open table formats, including Apache Iceberg v3, Hudi, and Delta Lake.
At first glance, the upgrade may look simple. A team can select Glue 6.0 for a job and immediately gain access to newer runtime capabilities. However, a production migration involves more than changing a version number. Runtime behavior, dependencies, table formats, query engines, and downstream applications may all be affected.
My recommendation is straightforward: upgrade the runtime first, preserve compatibility where necessary, and adopt Iceberg v3 gradually instead of changing every table at once.
What is new in AWS Glue 6.0?
Glue 6.0 includes several improvements for data engineering teams. Spark 4.1.1 introduces Spark Declarative Pipelines, which provide a more declarative way to define data transformations and dependencies. This can reduce repetitive orchestration work and make complex pipelines easier to understand.
The release also adds Arrow-native Python user-defined functions and user-defined table functions. These improvements can reduce serialization overhead between Python and the Spark engine, which may improve performance for workloads that rely heavily on custom Python transformations.
For streaming use cases, Glue 6.0 introduces a real-time mode for stateless streaming workloads. This capability is designed for applications that need lower-latency event processing, such as real-time routing, operational analytics, and time-sensitive data transformations.
The runtime upgrade also brings Python 3.13 and Scala 2.13.17. These newer versions provide access to updated language features and libraries, but they also mean that existing dependencies should be tested carefully. Custom JAR files compiled for Scala 2.12 may need to be recompiled, and Java applications using the older AWS SDK for Java may require code changes.
The price reduction is another attractive part of the release. However, the total cost of a data platform depends on more than the Glue hourly rate. Job duration, DPU consumption, crawlers, Amazon S3 storage and requests, Data Catalog usage, and related analytics services all contribute to the final bill. Teams should therefore measure the complete workload instead of assuming that a lower unit price will automatically produce the same percentage of savings.
Why Apache Iceberg v3 matters
Apache Iceberg v3 introduces capabilities that are particularly useful for semi-structured, high-frequency, and spatial data. One notable feature is the VARIANT data type with shredding support. It allows teams to work with JSON, logs, and event data without flattening every field into a fixed schema before analysis.
This can make data pipelines more resilient when upstream applications add new fields or change their structure. Instead of creating duplicate copies of the same data or writing custom parsing logic for every schema change, data engineers can preserve more of the original structure and query it when needed.
Iceberg v3 also introduces geometry and geography data types for spatial workloads. These features can support location intelligence, geographic analysis, logistics, mapping, and other use cases that previously required additional processing steps.
Another improvement is support for nanosecond-precision timestamps. This can be valuable for Internet of Things data, scientific workloads, high-frequency events, and any system where millisecond precision is not enough.
These capabilities are powerful, but they should not be enabled automatically for every table. AWS Glue 6.0 supports both Iceberg v2 and v3, and Iceberg v2 remains the default. This gives teams the flexibility to use v3 only where its features provide clear value.
The compatibility decision teams should not overlook
The most important consideration is compatibility with downstream services. Iceberg v3 tables created in AWS Glue 6.0 cannot currently be read by Athena SQL. New Iceberg v3 data types also have limitations in some AWS Glue features, including DynamicFrames and Visual ETL in Glue Studio.
This means that changing a table to Iceberg v3 is not just an internal ETL decision. It changes the interface between data producers and data consumers. Before adopting v3, teams should identify every service, application, dashboard, notebook, and external engine that reads or writes the table.
For shared tables used by Athena, business intelligence tools, multiple teams, or several analytics engines, Iceberg v2 may remain the safer choice. For isolated tables consumed only by Spark workloads that need semi-structured, spatial, or high-precision features, Iceberg v3 may be a good fit.
Another important point is that moving a table from Iceberg v2 to v3 is a one-way change. Teams should treat this decision as a long-term data-product commitment rather than a temporary experiment. If compatibility is uncertain, creating a separate v3 table may be safer than converting the only existing copy.
A practical migration approach
The first step is to create an inventory of existing Glue jobs and their dependencies. Record the current Glue version, Spark APIs, Python packages, custom JAR files, connectors, S3 configuration, and downstream consumers. Pay special attention to Scala 2.12 dependencies, Java SDK usage, EMRFS-specific settings, and SQL expressions that may behave differently under the new Spark runtime.
The second step is to upgrade a representative group of jobs to Glue 6.0 while keeping existing Iceberg tables on version 2. This separates runtime migration from table-format migration. If a job fails, the team can determine whether the problem comes from Spark, Python, Scala, a connector, or the table format.
Testing should go beyond checking whether a job completes successfully. Compare row counts, schemas, null handling, timestamps, partitioning, error records, output files, data quality results, and downstream query behavior. Spark 4.1.1 enables ANSI mode by default, so invalid casts and numeric overflows that previously returned NULL may now produce errors. These changes should be tested with realistic data, including malformed and unexpected records.
The third step is to measure operational and financial performance. Compare job duration, DPU usage, retry frequency, shuffle volume, input size, output size, and query performance before and after the upgrade. Use equivalent workloads and separate one-time migration costs from recurring production costs.
The final step is to evaluate Iceberg v3 selectively. Begin with a non-critical table whose consumer list is known and whose readers are confirmed to support the required features. Monitor query behavior, schema evolution, data quality, and operational processes before expanding the approach to additional tables.
A balanced architecture pattern
For many organizations, a mixed strategy will be the most practical. Shared data products can remain on Iceberg v2 as an interoperability layer for Athena and other consumers. Specialized Spark-native zones can adopt Iceberg v3 when they need the VARIANT type, spatial data types, or nanosecond timestamps.
This approach avoids turning one table-format upgrade into a platform-wide migration. It also gives data teams a controlled environment in which to learn the new capabilities, validate performance, and understand the operational implications.
Over time, the v3 zone can expand as support improves across analytics engines and as the organization becomes more comfortable managing format-specific compatibility. The goal is not to use the newest format everywhere. The goal is to use the right format for each data product.
Final thoughts
AWS Glue 6.0 is a meaningful release because it brings together lower pricing, a modern Spark runtime, updated programming languages, improved streaming capabilities, and stronger open table format support.
The release is especially relevant for teams modernizing data lakes, processing semi-structured events, building streaming pipelines, or preparing data for analytics and AI applications. However, the safest adoption strategy is not to enable every new feature immediately.
Start by upgrading and testing the runtime. Then measure the complete workload. Finally, adopt Iceberg v3 only where the full consumer ecosystem is ready for a one-way format change. Keeping shared tables on Iceberg v2 is not a sign that a team is falling behind. It is a deliberate compatibility decision.
This approach turns AWS Glue 6.0 from a risky upgrade-everything project into a controlled path toward a more capable, flexible, and cost-efficient data platform.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article