Inside AWS Glue: Understanding the Spark Engine and Using the Spark UI for Troubleshooting

Learn how AWS Glue uses the Spark engine to create scalable, performant data pipelines, and how to monitor background processes using the Spark UI.

David McAmis

Published Jan 18, 2026

Last modified Jan 30, 2026

 AWS Glue  is a fully managed, serverless data integration service that relies heavily on the  Apache Spark  engine to perform large-scale data processing efficiently. Understanding how the Spark engine works within AWS Glue, and how to monitor it using the Spark UI, helps clarify why Glue is well suited for extract, transform, and load (ETL) workloads and how engineers can troubleshoot performance or correctness issues.
At its core, AWS Glue uses Apache Spark as the distributed execution engine for ETL jobs. When a Glue job starts, AWS provisions a Spark environment on demand, including the required compute resources, networking, and storage access. Users do not manage clusters directly; instead, Glue abstracts this complexity and exposes Spark through higher-level constructs such as  DynamicFrames ,  Glue Data Catalog integration , and job parameters. Despite these abstractions, the underlying execution model is still Spark’s distributed, in-memory processing paradigm, and the Spark UI reflects this internal execution in a familiar way for Spark users.
AWS Glue ArchitectureThe Spark engine in AWS Glue operates using a driver-and-executor architecture. The driver program, created automatically by Glue, is responsible for orchestrating the job, building the logical execution plan, and scheduling tasks. Executors run on distributed worker nodes and execute transformations on partitions of data. In the Spark UI, this architecture is visible through separate views of the driver, executors, stages, and tasks, allowing users to see how work is distributed across the cluster and how resources are utilized.
To access the Spark UI: 
Login to the AWS Console
Navigate to the AWS Glue home page
Click on Go to ETL Jobs
Navigate to a Glue job that you have run and then click the Runs tab
You will see the Spark UI in the tabbed menu, as shown below:
﻿
Data ingestion in AWS Glue typically begins with reading from sources such as Amazon S3, relational databases, or streaming services. Spark reads this data in parallel by splitting it into partitions, which are then processed independently by executors. The Spark UI shows this parallelism through stages and tasks, making it easy to see how many partitions are being processed and whether tasks are evenly distributed. Skewed task durations in the UI often indicate data skew, which can be a root cause of slow-running Glue jobs.
Transformations in AWS Glue follow Spark’s lazy evaluation model. When transformations such as filters, joins, or aggregations are defined, Spark builds a directed acyclic graph (DAG) of operations rather than executing them immediately. Execution begins only when an action is triggered, such as writing output data. The Spark UI provides a visual representation of this DAG through the Jobs and Stages tabs, showing how logical transformations are broken into physical stages separated by shuffle boundaries. This view is particularly useful for understanding where expensive shuffles occur and how they impact performance.
﻿
Memory and disk usage are critical factors in Spark job performance, and the Spark UI is one of the most effective tools for diagnosing related issues in AWS Glue. The Executors tab (shown below) shows memory usage, garbage collection time, disk spill metrics, and shuffle read/write statistics for each executor. High spill values or excessive garbage collection time can indicate that the selected Glue worker type does not have sufficient memory for the workload, or that transformations such as wide joins are generating large intermediate datasets.
﻿
AWS Glue enhances Spark’s built-in fault tolerance by monitoring job execution and handling retries, but the Spark UI remains invaluable when failures occur. Failed tasks and stages are clearly marked in the UI, along with error messages and stack traces. This allows engineers to distinguish between data-related errors, such as schema mismatches or corrupt records, and infrastructure-related issues, such as executor loss. By correlating failure points in the Spark UI with CloudWatch logs generated by Glue, troubleshooting becomes significantly more precise.
In addition to the Spark UI for AWS Glue jobs is provided through the AWS Glue console, newer Glue versions provides links that expose the Spark UI while the job is running and for a limited time after completion. This temporary availability aligns with Glue’s serverless model, where clusters are created and destroyed on demand. Even with this ephemeral nature, the Spark UI provides deep visibility into job behavior during execution, enabling performance tuning and root cause analysis without managing long-lived infrastructure.
When a Glue job completes, the Spark environment is automatically torn down, and no compute resources remain allocated. Combined with the Spark UI’s detailed insights during execution, this serverless lifecycle allows AWS Glue to deliver scalable, cost-efficient ETL processing while still giving data engineers the transparency and diagnostic capabilities they expect from Apache Spark.