The Ghost Cluster: The EMR Cluster That Never Received Jobs but Kept Costing Us Money

There was an EMR cluster in our pipeline that appeared to be available and healthy, but consistently processed no jobs while continuing to incur running costs.

The pipeline itself appeared to be functioning correctly. Files were arriving in S3, Lambda was being triggered as expected, and Spark jobs were being successfully submitted to EMR. However, when we examined the workload distribution across the available EMR clusters, we noticed that jobs were consistently being submitted to one cluster while another remained idle.

There were no obvious application failures or errors that explained this behavior. The challenge was therefore not identifying a failed component, but understanding why the workload was not being distributed as expected.

In this article, I'll walk through the original architecture, the investigation that led to the bottleneck, the trade-offs and considerations, and the final changes that made the execution flow more predictable.

The pipeline was designed around an event-driven workflow where a file arriving in Amazon S3 triggered a Lambda function responsible for orchestrating the processing.

The Lambda function identified an available EMR cluster and submitted the corresponding Spark job to it. Once submitted, the EMR cluster handled the compute workload and produced the required output.

The simplified architecture was:

The environment included two EMR clusters to provide available processing capacity for incoming workloads. The expectation was that Lambda would select an appropriate available cluster when submitting each job, allowing the available compute resources to be used efficiently.

This architecture worked as expected under normal conditions. The unexpected behavior became apparent when multiple files triggered the Lambda function within a short period of time.

The issue was reproducible when two files were uploaded to S3 at nearly the same time. The expected behavior was for the first invocation to select one cluster and the second invocation to select the other:

Instead, both invocations selected the same cluster:

From an application perspective, both executions were successful. The problem was therefore not a failed invocation or an unsuccessful EMR submission; it was the fact that both executions made the same cluster-selection decision.

The next step was to understand what information each Lambda invocation was using at the exact moment it selected the cluster.

We correlated the Lambda RequestId, the selected EMR cluster ID, the job submission timestamp, and the state of the available clusters during each invocation.

The investigation showed that the two Lambda executions were querying the cluster state within a very short period of each other. The first invocation selected Cluster A and submitted a job to it. When the second invocation performed its cluster lookup shortly afterwards, Cluster A was still being reported as WAITING.

This was the key observation because the second invocation was not aware, through the state returned by EMR, that Cluster A had just received a job.

The sequence was effectively:

The issue was therefore related to the timing between job submission and the EMR cluster state transition.

The cluster-selection logic relied on the EMR cluster STATE to determine whether a cluster was available. However, when a cluster receives a new step, its state does not transition from WAITING to RUNNING instantaneously.

There can be a short delay between the job being submitted and the cluster state reflecting that workload. During this period, the cluster can continue to appear as WAITING, even though a job has already been submitted to it.

When Lambda executions occur concurrently, both executions can therefore observe the same cluster as available:

The root cause was therefore a selection race caused by EMR state lag. The problem was not that EMR failed to update its state, but that the state update was not immediate enough for the existing concurrent selection logic to reliably distinguish a cluster that had just received a job from one that was genuinely available.

The first solution was to configure the trigger Lambda with: ReservedConcurrentExecutions: 1

This ensures that only one instance of the Lambda can execute at a time. When two S3 events arrive simultaneously, one invocation is processed first while the other is throttled and automatically retried by Lambda.

This directly fixes the concurrent selection race. Two Lambda invocations can no longer query the cluster state and make a selection at the same time.

It also means that throttled asynchronous invocations are not simply discarded. Lambda automatically retries them, allowing the events to be processed once the concurrent execution becomes available.

However, serialization alone does not define which cluster the next invocation should use. We therefore needed to make the cluster-selection logic deterministic as well.

The second solution was to introduce a module-level round-robin pointer, _rr_index, inside the Lambda execution environment.
Instead of selecting the first cluster returned by the availability check, the Lambda now establishes a stable order for the available clusters and uses the pointer to select the next cluster in sequence.
Ready clusters are preferred, with WAITING and RUNNING clusters considered before clusters that are still STARTING. The available clusters are also sorted to ensure that the selection order remains stable.
The use of in-memory state is intentional. Because the Lambda is configured with a concurrency of one, back-to-back invocations will, in normal circumstances, reuse the same warm execution environment. The module-level pointer can therefore retain its value between those invocations without requiring an external state store.

> Conceptually, with two available clusters:

The implementation also sorts the available clusters before applying the round-robin selection so that the order remains stable rather than depending on the order returned by the EMR API.

The first trade-off is reduced concurrency at the Lambda layer. Setting reserved concurrency to one means that incoming events are processed sequentially, which can increase the time required to process a large number of files. For a lightweight orchestration Lambda, this may be an acceptable trade-off when predictable resource selection is more important than parallel event processing.

The round-robin approach also relies on in-memory state within a warm Lambda execution environment. The _rr_index value does not survive a cold start, so a new execution environment can restart the sequence. This means the mechanism should not be treated as a durable scheduling state or as a global load-balancing mechanism across multiple Lambda execution environments.

Another consideration is cluster availability, the implementation still needs to account for the fact that clusters can move between states or become unavailable. The round-robin mechanism operates on the set of clusters considered eligible at the time of selection; it does not replace the availability check itself.

If the architecture eventually requires multiple concurrent trigger executions, strict global ordering, or durable scheduling state, an external coordination mechanism would be more appropriate. For the current workflow, however, serializing the orchestration layer and maintaining a lightweight in-memory pointer provide a simple way to make the behavior predictable without introducing additional infrastructure.

The issue initially looked like an EMR cluster utilization problem: two clusters were available, but consecutive jobs were repeatedly being submitted to the same cluster. The investigation showed that the underlying problem was not the availability of the clusters, but the timing of the information used to select them.

Because EMR cluster state can take several seconds to reflect a newly submitted step, two concurrent Lambda invocations could both observe the same cluster as WAITING and independently select it. The result was two successful jobs running through the same cluster while another available cluster remained unused.

The solution addressed the problem at two levels. Setting ReservedConcurrentExecutions: 1 removed the concurrent selection race by ensuring that cluster-selection decisions were made sequentially. Introducing a round-robin pointer then made those sequential decisions deterministic, allowing successive jobs to alternate across the available clusters.

The main lesson was that checking the current state of a distributed resource is not always sufficient when multiple events can make decisions concurrently. When state propagation is not instantaneous, the orchestration layer also needs to account for concurrency and maintain enough state to make resource selection predictable.