AWS Builder Center

Builder Center

AWS Glue in 2026: pick the version, worker, and engine that match the job

A 2026 checklist for AWS Glue Spark jobs: which version to run after 5.1 and 6.0, which workers and Flex settings cut cost without breaking SLAs, when Lambda or Python Shell beats Spark, and how to watch runs with Glue Observability, CloudWatch, and EventBridge.

Series: Glue in Production (1 article)

  1. 1
    AWS Glue in 2026: pick the version, worker, and engine that match the job This article

Felipe Rodrigues

Staff Engineer @ BHub.ai
AWS Glue is the serverless path for ETL, cataloging, and data integration on AWS. Spark, Python Shell, and the Glue Data Catalog cover everything from legacy batch jobs to Iceberg lakehouses.
Wrong defaults still cost real money: an EOL Glue version, a Spark cluster for a 200 MB JSON transform, Flex left off a nightly job, Auto Scaling miswired so you pay for idle DPUs, or jobs you cannot debug because metrics were never enabled. This post is a 2026 operating checklist: which Glue version to run, which worker to buy, when Flex helps, how to scale, which engine to pick, and how to observe jobs with AWS-native tooling.

Ray is omitted from the overview on purpose: Glue for Ray is closed to new customers as of 2026-04-30.

Glue versions: 3.0 through 6.0 (and what already EOL)

Version choice drives Spark/Python stacks, open table format support, Lake Formation FGAC behavior, default timeouts, and price. As of late 2026 the ladder looks like this:
Glue version
Spark
Python
Highlights
Watch-outs
6.0
4.1.1
3.13
~30% lower price vs prior Glue versions; Iceberg v3; Hudi 1.1.1; Delta 4.2.0; Spark Declarative Pipelines; Spark Connect
Breaking: EMRFS removed (S3A only), AWS SDK for Java v1 removed, Scala 2.13 JARs, Iceberg v3 is one-way and Athena SQL may fail on format-version 3
5.1 (omit default)
3.5.6
3.11
Docs still say jobs created without a Glue version default to 5.1 (checked against add-job / release notes after 6.0 GA); OTF bump (Hudi 1.0.2, Iceberg 1.10.0, Delta 3.3.2); FGAC writes (DML/DDL) for Spark DataFrames/SQL
Still Spark 3.x; plan a separate jump if you need Spark 4 / Iceberg v3
5.0
3.5.4
3.11
~32% faster than Glue 4.0 (AWS launch claim); Spark-native Lake Formation FGAC;  requirements.txt ; DataZone; S3 Table Buckets; Java 17
GlueContext/DynamicFrame table-level AC from Glue 4.0 and earlier is not supported; use Spark-native FGAC
4.0
3.3.0
3.10
Mature lakehouse connectors on Spark 3.3; Observability metrics available
No longer the tip of the tree
3.0
3.1.1
3.7
Still listed; Flex starts here
Python 3.7 era stack; keep only for true legacy
2.0 / 1.0 / 0.9
Spark 2.x
-
-
EOL 2026-04-01. Migrate.

Practical recommendations

  • New Spark jobs: start on Glue 5.1 (still the documented default when Glue version is omitted) unless you need Spark 4.1, Iceberg v3, or the 6.0 price cut enough to absorb the migration work.
  • Moving to 6.0: budget for S3A-only configs, Scala 2.13 rebuilds, SDK v1 removal, ANSI mode defaults, and Iceberg v3 compatibility with Athena and downstream readers. Read the migration guide before flipping production.
  • Still on Glue 2.0 or older: you are past EOL. Upgrade is not optional.
  • Python Shell: leave 3.6 (EOS/EOL in 2026). Use 3.9.
Default job timeout also changed: blank timeout is 2880 minutes on Glue 4.0 and earlier and 480 minutes on Glue 5.0+. After an upgrade, jobs that used to run for many hours can stop at the new default unless you set an explicit timeout ( add job ).

Worker types: G, R, and when vertical beats horizontal

Worker choice is DPU burn rate plus memory headroom. The old "only G.1X / G.2X / G.4X" table is incomplete in 2026.
Worker
DPU
vCPU
Memory
Disk (approx)
Notes
G.025X
0.25
2
4 GB
~84 GB
Streaming only, Glue 3.0+
G.1X
1
4
16 GB
~94 GB
Cost-efficient default; Flex OK
G.2X
2
8
32 GB
~138 GB
Heavier joins/aggs; Flex OK
G.4X
4
16
64 GB
~256 GB
Large joins / heavy transforms
G.8X
8
32
128 GB
~512 GB
Glue 3.0+; Region list
G.12X
12
48
192 GB
~768 GB
Glue 4.0+; higher startup latency; no Flex
G.16X
16
64
256 GB
~1024 GB
Same as G.12X at larger size
R.1X to R.8X
1-8
4-32
2x G peer memory
same disk class as G peer
Memory-optimized; Glue 4.0+; no Flex; public blog signal $0.52/DPU-hour

How I choose

Author guidance, not AWS-published thresholds:
  • Start on G.1X for common transforms and moderate data.
  • Move to G.2X / G.4X when Spark UI or Observability shows memory pressure, heavy shuffle, or large joins.
  • Use R.* when the bottleneck is memory per executor (skew, broadcast joins, fat query plans) and horizontal scale alone does not help.
  • Reserve G.12X / G.16X for the largest batch shapes after you have evidence from metrics.

Production example (author claim)

One production job processes about 2.3 million PostgreSQL rows, applies Spark transforms, serializes to Avro, and publishes to Kafka in under six minutes on G.1X with Auto Scaling. That shape is distributed ETL with a clear sink. It is the kind of workload Glue Spark is built for.

Flex execution class

Flex is for Glue 3.0+ Spark batch jobs on G.1X or G.2X only. AWS runs them on spare capacity, so start time varies. Public pricing examples still show roughly $0.29/DPU-hour Flex versus $0.44/DPU-hour Standard (~34% lower in those examples; FAQ says up to about 35%). Pricing varies by Region. Confirm the rate where you run.
Use Flex for nightly cleanup, weekly reports, and non-SLA batch. Skip Flex when start latency is part of an SLA, and remember Flex does not apply to G.12X, G.16X, or R workers.
Glue 6.0 separately advertises a ~30% price reduction versus prior Glue DPU-hour rates (AWS What's New claim). That is a version-level change, not a substitute for Flex. Confirm the live DPU-hour on the pricing page for your Region before you quote a dollar figure for 6.0.

Scaling workers: Auto Scaling vs fixed count

Wrong sizing either burns idle DPUs or leaves the job starved.

Auto Scaling

Best when volume swings: streaming-adjacent batch, monthly peaks, log volumes that move between tens of GB and a TB.
Configure Spark Auto Scaling with:
  •  --enable-auto-scaling true 
  •  NumberOfWorkers  as the maximum worker count
Current Glue Auto Scaling docs for Spark do not document  --min-workers  /  --max-workers  as the Spark ETL controls. Those names showed up in older notes and in Ray job parameters. Wire the maximum through  NumberOfWorkers  and verify in the console or CloudFormation/CDK you actually deploy.
Source:  Auto Scaling .
Supported Auto Scaling worker types include G.1X through G.16X, R.1X to R.8X, and G.025X for streaming.
After Auto Scaling is on, watch executor demand in CloudWatch with  glue.driver.ExecutorAllocationManager.executors.numberAllExecutors  and  glue.driver.ExecutorAllocationManager.executors.numberMaxNeededExecutors . Low  glue.driver.workerUtilization  is a signal to let Auto Scaling shrink the fleet or to cut a fixed worker count.
Tradeoff: scale-out is not instant. Provisioning adds latency. That is usually fine for batch; it can hurt tight near-real-time windows.

Fixed worker count

Best for stable daily volumes and pipelines where runtime must stay predictable. You get cost predictability and stable duration. You also risk over-provisioning on quiet days. Measure with history, then lock the count.
Optional resilience: enable job run queuing when concurrency quotas or capacity would otherwise fail the start ( add job ).

Pick the engine: Spark, Python Shell, or Lambda

Spark is powerful and expensive to boot. Paying for a distributed engine to run a sequential Python script is a common leak.

  • Lambda: small payloads, event-driven triggers, fast start. Stay inside Lambda limits.
  • Glue Spark: large-scale distributed ETL, joins, lakehouse table maintenance, Spark SQL.
  • Glue Python Shell: Catalog integration, Glue triggers, orchestration inside Glue, without Spark overhead. Use Python 3.9, not 3.6.
Glue for Ray closed to new customers on 2026-04-30. Existing customers keep running; AWS points new Ray work at Ray on Amazon EKS (KubeRay). Do not design greenfield Glue jobs around Ray.
Source:  Ray availability change .

Spark habits that keep jobs alive

  • Avoid  collect()  /  toPandas()  on large frames; keep work on executors.
  • Prefer distributed transforms ( select ,  filter ,  withColumn , joins done in Spark).
  • Fix skew with repartition/coalesce, sane partition sizes, and Spark Adaptive Query Execution when skew joins show up in Observability.
  • Prefer columnar formats (Parquet, and Iceberg/Hudi/Delta when you need table semantics).
  • Use checkpoints on long jobs.
  • Set retries with backoff on critical paths.
  • Set an explicit timeout slightly above expected runtime (especially on Glue 5.0+ defaults).
  • Validate on sampled data before full production volume.
  • Initialize  GlueContext  when you want job metrics and Observability to publish correctly.
For deeper tuning, use  AWS Prescriptive Guidance: Best practices for performance tuning AWS Glue for Apache Spark jobs  plus Spark UI and the metrics below.

Lake Formation FGAC on 5.x+

Glue 5.0 moved fine-grained access control to Spark-native Lake Formation integration. GlueContext/DynamicFrame table-level patterns from Glue 4.0 and earlier are not the path on 5.0+. Glue 5.1 expands FGAC to writes (DML/DDL) for Spark DataFrames/SQL. FGAC jobs need at least four workers.

Monitoring with Glue Observability and CloudWatch

Treat observability as part of the job definition, not an afterthought. On Glue 4.0+, enable both classic job metrics and Glue Observability metrics, keep continuous CloudWatch logging on, and turn on Spark UI event logs to S3 for deep dives.
Enable Observability in job parameters:
--enable-observability-metrics true
--enable-metrics true
--enable-continuous-cloudwatch-log true
--enable-spark-ui true
--spark-event-logs-path s3://your-bucket/spark-ui-logs/
( glueContext  must be initialized for Observability / throughput publishing.)
Observability groups metrics into four buckets you can graph in Amazon CloudWatch and in the Glue Studio job run pages:
Bucket
What you learn
Example signals
Reliability
Why jobs fail
 glue.error.ALL ,  glue.error.OUT_OF_MEMORY_ERROR ,  glue.error.PERMISSION_ERROR ,  glue.succeed.ALL 
Performance
Where time goes
 glue.driver.skewness.job ,  glue.driver.skewness.stage 
Throughput
Volume trends per source/sink
 glue.driver.bytesRead ,  glue.driver.recordsRead ,  glue.driver.bytesWrittten  (CloudWatch metric name as published in AWS docs, including the double  t )
Resource utilization
Idle or saturated capacity
 glue.driver.workerUtilization , driver/executor heap and disk  % 

Alarms that page people

Build Amazon CloudWatch alarms on those metrics, then notify through Amazon SNS (email, Chatbot to Slack/Teams, or your existing incident hook). Baseline a quiet week before you set thresholds so you page on real drift.
Useful alarm shapes:
  • Failures:  glue.error.ALL  (SUM) greater than 0 on a production job. Narrow with category metrics such as  glue.error.OUT_OF_MEMORY_ERROR ,  glue.error.PERMISSION_ERROR , and  glue.error.THROTTLING_ERROR  as listed in the Observability error-category table. Confirm the exact  MetricName  strings in the CloudWatch console for your Region before you ship alarms (AWS prose sometimes shortens the category labels).
  • Skew:  glue.driver.skewness.job  above your baseline for more than one evaluation period.
  • Waste:  glue.driver.workerUtilization  chronically low on fixed-size jobs (right-size or enable Auto Scaling).
  • Pressure: driver or executor  memory.*.used.percentage  /  disk.used.percentage  above a safe ceiling before OOM or disk-full failures.
  • SLA duration: classic  glue.driver.aggregate.elapsedTime  (or job run duration) above the agreed max for that pipeline.
  • Quota risk: Glue service resource usage metrics under the  AWS/Glue  namespace when you approach account concurrency or workflow limits ( resource metrics ).

Terminal state via EventBridge

Metric alarms catch OOM, skew, and waste while the job is still producing datapoints. For terminal outcomes, add an Amazon EventBridge rule on Glue Job State Change and target Amazon SNS for  FAILED ,  TIMEOUT , and  STOPPED  (optionally  SUCCEEDED  if you want success fans). That path does not wait for custom metric datapoints, so short-lived failures still notify.
Example event pattern:
{
  "source": ["aws.glue"],
  "detail-type": ["Glue Job State Change"],
  "detail": {
    "state": ["FAILED", "TIMEOUT", "STOPPED"]
  }
}
When an alarm or EventBridge notification fires, use this triage order: Observability error class → CloudWatch graph for the same JobRunId → Spark UI for the hot stage → CloudWatch Logs (and Logs Insights on  /aws-glue/jobs/  streams when you need text search). That path stays inside AWS and matches how Glue Studio surfaces run health today.


2026 operating checklist

Before you call a Glue Spark job "production ready":
  1. Glue version is 5.1+ (or a deliberate 6.0 migration with S3A / Scala 2.13 / Iceberg v3 checked). Nothing on EOL 2.0/1.0/0.9.
  1. Worker type matches evidence from a prior run (start G.1X; move to R.* or G.12X/G.16X only with metrics).
  1. Flex on for non-SLA G.1X/G.2X batch; off when start latency is part of the contract.
  1. Auto Scaling uses  --enable-auto-scaling  and  NumberOfWorkers  as max, or a fixed count justified by stable volume.
  1. Explicit timeout set for Glue 5.0+ (do not inherit 480 minutes by accident).
  1. Job run queuing considered where concurrency quotas bite.
  1. Engine choice is honest: Spark only when you need distribution; Python Shell 3.9 or Lambda otherwise. No new Glue Ray.
  1. Lake Formation FGAC paths use Spark DataFrames/SQL on 5.x+, with at least four workers when FGAC is on.
  1.  --enable-observability-metrics ,  --enable-metrics , continuous logs, and Spark UI logs are on; CloudWatch alarms cover skew/utilization/errors, and EventBridge Glue Job State Change notifies SNS on FAILED/TIMEOUT/STOPPED.
  1. A sampled dry run passed before full volume.

TL;DR

  • Prefer Glue 5.1 for new Spark jobs; evaluate 6.0 when you want Spark 4.1 / Iceberg v3 / the advertised price cut and can absorb breaking changes. Leave EOL versions immediately.
  • Size workers from Observability and Spark UI evidence. Use G.1X/G.2X for most batch; R.* or G.12X/G.16X when memory or plan size demands vertical scale.
  • Turn on Flex for non-SLA G.1X/G.2X batch; confirm regional price.
  • Auto Scaling:  --enable-auto-scaling  plus  NumberOfWorkers  as max.
  • Use Spark for distributed ETL, Python Shell 3.9 for non-Spark Glue Python, Lambda for small event-driven work. Do not start new Glue Ray jobs.
  • Monitor with Glue Observability metrics, CloudWatch alarms, and EventBridge job state change to SNS. Keep Spark UI logs on S3 for stage-level debugging.
# aws-glue# apache-spark# etl# data-engineering# aws-lake-formation

Series: Glue in Production (1 article)

  1. 1
    AWS Glue in 2026: pick the version, worker, and engine that match the job This article
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article