AWS Builder Center

Builder Center

Handling Data Skew in Apache Spark on AWS EMR: Detection and Mitigation

Data skew can significantly impact Spark performance by creating uneven workloads across partitions. This article explores how to identify skewed workloads on AWS EMR and discusses practical techniques such as broadcast joins, salting, and Adaptive Query Execution (AQE) to improve Spark performance.

Handling Data Skew in Apache Spark on AWS EMR: Detection and Mitigation

Introduction

Apache Spark is designed to process large datasets by distributing the workload across multiple partitions and executors. In an ideal situation, the data is evenly distributed, allowing all tasks to finish at roughly the same time.
In real-world data, however, this is often not the case.
One of the common problems in distributed data processing is data skew. Data skew occurs when a small number of keys contain a disproportionately large amount of data. During operations such as joins, aggregations, and group-bys, this can cause some Spark partitions to become significantly larger than others.
The result can be uneven task execution, excessive shuffle, executor memory pressure, and long-running jobs.
In this article, I will explain how data skew affects Spark workloads running on AWS EMR and discuss three practical approaches for handling it: broadcast joins, salting, and Adaptive Query Execution (AQE).


What Is Data Skew?

Spark divides data into partitions so that multiple tasks can process it in parallel.
Consider a dataset containing customer transactions. Suppose most customers have a few hundred transactions, but one customer has several million records.
If we perform a join or aggregation using  customer_id , Spark may send all records belonging to that customer to the same partition.
For example:
text
Partition 1 → 100 MB
Partition 2 → 110 MB
Partition 3 → 105 MB
Partition 4 → 2.5 GB

Although the first three partitions finish quickly, the fourth partition takes much longer.
This creates a straggler task.
The Spark application cannot complete the stage until that slow task finishes, meaning that having many executors does not necessarily solve the problem.


Why Does Data Skew Happen?

Data skew is usually caused by an uneven distribution of values.
Some common causes include:
  • Highly popular keys
  • Default or unknown values such as  NULL  or  UNKNOWN 
  • Uneven customer activity
  • IoT or event streams dominated by a small number of devices
  • Geographic or categorical data with highly unequal distributions
  • Large fact tables joined using low-cardinality keys
For example, imagine a dataset where:
text
customer_id = 1001 → 500 records
customer_id = 1002 → 700 records
customer_id = 1003 → 600 records
customer_id = 9999 → 50,000,000 records

A join involving  customer_id  can create a heavily skewed partition for  9999 .


How to Identify Data Skew

Before trying to optimize a Spark job, it is important to confirm that skew is actually the problem.
The Spark UI is particularly useful for this.
When examining a stage, some useful indicators are:
  • Large differences in task duration
  • Large differences in shuffle read
  • Large differences in input size
  • One or a few tasks taking significantly longer than the others
  • Executors processing highly different amounts of data
For example, if most tasks finish in a few seconds while one task continues running for several minutes, the stage may be suffering from skew.
It is also useful to examine the distribution of the join or aggregation key directly.
Python
from pyspark.sql.functions import count

df.groupBy("customer_id") \
  .agg(count("*").alias("record_count")) \
  .orderBy("record_count", ascending=False) \
  .show(20)

This helps identify keys that occur far more frequently than others.


Approach 1: Broadcast Join

One possible solution is a broadcast join.
If one side of the join is small enough to fit comfortably in executor memory, Spark can broadcast it to all executors.
Instead of shuffling both datasets, Spark sends the smaller dataset to the executors and performs the join locally.
For example:
Python
from pyspark.sql.functions import broadcast

result = large_df.join(
    broadcast(small_df),
    "customer_id"
)

Conceptually:
text
Without Broadcast

Large DF ── Shuffle ──┐
                      ├── Join
Small DF ── Shuffle ──┘

With Broadcast

                 ┌── Executor 1
Small DF ────────┼── Executor 2
                 ├── Executor 3
Large DF ────────┼── Executor 4
                 └── Executor 5

Broadcast joins can eliminate the shuffle associated with the smaller side of the join.
However, this technique is not suitable when both datasets are large. Broadcasting a dataset that is too large can put significant memory pressure on executors and potentially cause failures.
Therefore, broadcast joins should be used based on the size and characteristics of the data rather than as a universal solution.


Approach 2: Salting

When the skewed dataset is large and broadcasting is not practical, salting can be used.
The basic idea is to artificially create multiple versions of a heavily skewed key so that its records can be distributed across multiple partitions.
For example, instead of having:
text
customer_id = 9999

we can create salted keys:
text
9999_0
9999_1
9999_2
9999_3

The records are then distributed across multiple partitions rather than being concentrated in a single partition.
For a join, the smaller side generally needs to be expanded with the corresponding salt values so that the join remains logically correct.
Conceptually:
text
Before:

9999 → ████████████████████████████

After Salting:

9999_0 → ███████
9999_1 → ███████
9999_2 → ███████
9999_3 → ███████

This allows Spark to process the workload more evenly.
The main disadvantage is that salting makes the pipeline more complex. The implementation must carefully handle the salted keys, and the smaller side of the join may need to be replicated across the salt values.
Therefore, salting is most useful when there is significant skew that cannot be efficiently handled through broadcasting.


Approach 3: Adaptive Query Execution

Modern versions of Spark provide another powerful option: Adaptive Query Execution (AQE).
AQE allows Spark to make certain optimization decisions at runtime based on the actual data statistics collected during execution.
For skewed joins, Spark can detect unusually large shuffle partitions and take steps to process them more effectively.
Relevant configurations include:
Python
spark.conf.set("spark.sql.adaptive.enabled", "true")
spark.conf.set("spark.sql.adaptive.skewJoin.enabled", "true")

The important difference is that AQE does not require us to know the exact skew pattern beforehand.
Instead of relying entirely on assumptions made during query planning, Spark can use information observed during execution.
This makes AQE particularly useful for workloads where data distribution changes over time.
However, AQE should not be treated as a replacement for understanding the underlying data. Severe skew, poor partitioning, or an inappropriate join strategy can still require explicit optimization.


Comparing the Approaches

Each technique addresses skew differently.
Technique
Best suited for
Main advantage
Main consideration
Broadcast Join
One dataset is small
Can avoid expensive shuffle
Requires sufficient executor memory
Salting
Severe skew in large datasets
Distributes hot keys
Adds implementation complexity
AQE
Dynamic workloads and skewed joins
Runtime optimization
Depends on Spark configuration and workload
There is no single technique that should be applied to every skewed workload.
The correct approach depends on the size of the datasets, the distribution of the join keys, available memory, and the overall Spark execution plan.


Measuring the Improvement

Optimization should be based on measurements rather than assumptions.
A useful experiment is to run the same workload using different approaches and compare:
  • Total execution time
  • Shuffle read
  • Shuffle write
  • Number of tasks
  • Maximum task duration
  • Executor memory usage
  • Input and output data size
For example:
text
                    Execution Time
Normal Join         ████████████████████
AQE                 ████████████
Broadcast Join      ████████
Salting             ██████████

The actual results will depend heavily on the dataset and cluster configuration, so benchmarks should be collected from the specific workload being optimized.
The most important metric is not simply the average task duration. The maximum task duration and partition-size distribution can reveal whether the original bottleneck has actually been addressed.


Data Skew and AWS EMR

When Spark runs on AWS EMR, inefficient execution can also translate into increased infrastructure usage.
A skewed job may keep executors running while waiting for a small number of long-running tasks to finish.
This makes Spark optimization relevant not only to performance but also to resource efficiency.
A useful optimization workflow is therefore:
text
Identify Skew
      ↓
Inspect Spark UI
      ↓
Find Skewed Keys
      ↓
Understand the Join / Aggregation
      ↓
Choose Mitigation Strategy
      ↓
Run Benchmark
      ↓
Compare Execution Metrics
      ↓
Deploy the Better Approach

This approach is more reliable than simply increasing the number of executors or increasing cluster resources.
Adding more compute does not necessarily solve an imbalanced workload. If one partition contains a disproportionate amount of data, the underlying distribution problem can remain.


Practical Lessons

Working with distributed data processing has made one thing clear: parallelism is only useful when the workload can actually be distributed evenly.
A Spark application can have many executors and still perform poorly if a small number of partitions contain most of the data.
When investigating a slow Spark job, I would therefore avoid immediately increasing cluster resources. Instead, I would first inspect the execution plan and Spark UI, look for uneven task execution and shuffle sizes, and then determine whether data skew is contributing to the bottleneck.
Broadcast joins, salting, and AQE provide different ways of addressing the problem, but each has its own trade-offs.


Conclusion

Data skew is an important performance challenge in distributed data processing.
The problem is not necessarily the amount of data itself. The bigger issue can be how that data is distributed across partitions.
Using AWS EMR with Apache Spark, data engineers can identify skew through execution metrics and the Spark UI, then choose an appropriate mitigation strategy.
Broadcast joins can be effective when one side of a join is small. Salting can distribute heavily skewed keys across multiple partitions. AQE can allow Spark to adapt its execution strategy based on runtime information.
The key lesson is to measure the workload first, understand where the bottleneck comes from, and then apply the optimization that fits the data and execution pattern.
Good Spark performance is not just about adding more compute. It is about making sure that the compute can be used efficiently.
# aws-data-pipeline# big-data# databases# data-engineering# apache-spark
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article