The EMR Zombie Step: When Termination Isn't the End
You submit a new EMR step, but it takes longer than expected. The console shows only one step running, but is that the whole story? What’s lurking beneath the surface? Let’s investigate the zombie jobs.
Series: AWS OCTOBER MYSTERIES -2026 (3 articles)
- 3The EMR Zombie Step: When Termination Isn't the End This article
Introduction
Terminating an EMR step is expected to stop its execution and allow subsequent workloads to proceed normally. However, when a step is terminated and the next one continues running far beyond its expected execution time, the behavior raises questions about what is happening within the cluster.
Why does the current step take so long to complete after the previous step has been terminated? Could the termination have consequences that are not immediately visible in the EMR console?
This article explores what happens behind the scenes when an EMR step is terminated, investigates how YARN manages the underlying workload, and walks through the troubleshooting process to identify the cause and prevent similar behavior in the future.
The incident: An EMR Step That Wouldn't Finish
The issue occurred during an EMR workload, in which one of the steps needed to be terminated after its execution deviated from expectations. The step was stopped through the EMR console, and its status changed to TERMINATED, indicating that the termination request had been processed.
The next step was then submitted, with the expectation that the cluster would proceed with the new workload. However, the step remained in the RUNNING state far longer than expected, with no clear indication of progress. Although the previous step appeared terminated in the EMR console, the overall workflow was not progressing as anticipated.
The EMR step statuses alone were insufficient to explain the behavior. Further investigation was needed beyond the EMR console, including an examination of the application's state and resource usage in YARN, to determine why the new step was taking so long to complete.
Root Cause: The YARN Application That Survived Termination
The issue originated from a difference between EMR step termination and the lifecycle of the underlying YARN application. Terminating a step through the EMR console does not necessarily mean that every process associated with its execution has stopped.
In this case, the YARN application associated with the terminated step remained active on the cluster. As YARN continued managing that application, it retained resources that could otherwise have been available to subsequent workloads. The next step was therefore competing for cluster resources while the previous application was still running, explaining why its execution took significantly longer than expected.
The key finding was that the termination of the EMR step did not fully clear the activity associated with the previous workload. Identifying the YARN application that remained active was essential to understanding why the cluster was not processing the subsequent step as expected.
The Fix: Identify and Stop the Zombie Application
The resolution required inspecting the YARN applications running on the EMR cluster, identifying the application associated with the stalled workflow, and terminating it to release the resources it continued to consume.
Note: The commands below can be executed directly on the EMR cluster's primary node (formerly known as the master node), accessed through SSH or AWS Systems Manager (SSM) Session Manager, depending on your cluster configuration and permissions. Alternatively, you can execute them through a shell script submitted as an EMR step. For troubleshooting a stalled workflow, connecting directly to the primary node is often more practical, as it allows you to inspect and manage YARN applications without relying on another EMR step to execute the commands.
1. Inspect the running YARN applications
The first step was to list the applications managed by YARN and identify any applications that were still active unexpectedly.
1
yarn application -listThe output includes application IDs, names, states, and other details that help identify workloads still running on the cluster. The application state and associated workload should be verified before proceeding.
2. Identify the application that remained active
After identifying a potentially orphaned application, its status was inspected to determine whether it was associated with the terminated EMR step.
1
yarn application -status application_<timestamp>_<id>The application ID should be replaced with the actual ID returned by the previous command. Reviewing the application details helped confirm whether the workload was still running and whether it was safe to terminate, avoiding unnecessary disruption to unrelated applications.
3. Terminate the zombie application
After confirming that the application was no longer required, it was explicitly terminated through YARN:
1
yarn application -kill application_<timestamp>_<id>This sends a termination request to YARN for the specified application. The application state should be checked afterward to confirm that it has reached a terminal state.
4. Verify that the cluster has recovered
After terminating the application, the running application list was checked again to confirm that the lingering workload was no longer active.
1
yarn application -listFinally, the EMR console was used to monitor the cluster step and verify that processing could resume as expected. By examining both the EMR step status and the underlying YARN application state, we could confirm that the issue had been resolved rather than relying solely on the status displayed in the EMR console.
Trade-offs and Considerations
Although terminating a lingering YARN application can restore cluster resources and allow subsequent workloads to progress, the intervention requires careful consideration to avoid disrupting other processing activities.
Application termination is a corrective action, not a root-cause prevention mechanism. Killing a YARN application stops its execution, but it does not explain why the application remained active after the EMR step was terminated. If the behavior recurs, the termination workflow and application lifecycle should be investigated further.
Running applications must be identified carefully. The output of yarn application -list should be reviewed before executing yarn application -kill. Terminating the wrong application can interrupt a legitimate workload, cause processing failures, and potentially require the job to be rerun.
Termination may require additional recovery steps. Stopping an application does not guarantee that partially completed processing will be rolled back. Depending on the workload, data may have already been written or intermediate results may remain. Output consistency and retry behavior should therefore be verified before restarting the affected step.
Resource availability should be monitored after termination. Once the application has stopped, cluster resource usage and the subsequent EMR step should be monitored to confirm that the expected progress has resumed. If the delay persists, additional investigation may be necessary to identify other resource constraints or execution issues.
For production workloads, the termination procedure should be incorporated into operational troubleshooting guidance, with clear criteria for identifying orphaned or lingering applications and safeguards to prevent unrelated applications from being interrupted.
Conclusion
Terminating an EMR step does not always guarantee that the underlying workload has fully stopped. When a subsequent step runs far longer than expected, investigating the state of the YARN applications can reveal lingering processes and resource contention that are not immediately apparent from the EMR console.
The key lesson is to look beyond step statuses and verify the actual application state before taking corrective action. By identifying lingering applications, terminating them safely when necessary, and monitoring the cluster afterward, teams can restore normal processing while minimizing the risk of disrupting other workloads.
In distributed data processing, understanding the relationship between orchestration and application execution is essential for effective troubleshooting. Sometimes, resolving a long-running step starts with investigating what the previous step left behind.
Goodbye, Zombie!
Series: AWS OCTOBER MYSTERIES -2026 (3 articles)
- 3The EMR Zombie Step: When Termination Isn't the End This article
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article