AWS Builder Center
Stateful Streaming Made Simple on Amazon EMR

Stateful Streaming Made Simple on Amazon EMR

Build stateful streaming pipelines on Amazon EMR with Spark 4.0 — detect offline IoT devices in real time, no cluster management needed.

Senior Data & AI Engineer
If you've ever tried to answer "what's happening in my data right now" instead of "what happened last night," you already know why streaming exists. Batch jobs are great until someone asks you to catch a fraud attempt in seconds, or flag a machine on the factory floor the moment it stops sending signals. That's where Amazon EMR and Apache Spark Structured Streaming come in.
In this post, I want to keep things simple: what streaming on EMR actually looks like, why the newest Spark release makes "stateful" streaming (the hard kind) much easier, and how you'd wire it up with a small, realistic example — devices that need to be watched for going silent.



The problem: remembering things across events

Most streaming tutorials show you the easy part — read a record, transform it, write it out. That's stateless processing, and it's fine for things like format conversion or basic filtering.
The harder, more useful problems need memory:
  • Has this device sent a heartbeat in the last 30 seconds?
  • Is this the third failed login attempt for this account this hour?
  • Has this shopping cart been untouched long enough to count as abandoned?
Each of these needs the engine to remember something about a specific key — a device ID, an account ID — across many events, sometimes for hours. That's stateful streaming, and until recently, it was genuinely painful to build well.

What changed: Spark 4.0 on EMR Serverless

Apache Spark 4.0 introduced a new API called  transformWithState , and it's now available on Amazon EMR Serverless. It gives you three things that used to require a lot of custom code:
  • Timers — tell Spark "call me back at this exact time for this key," instead of hand-rolling your own clock logic.
  • Automatic cleanup (TTL) — state for keys you no longer care about expires on its own, so memory doesn't grow forever.
  • Multiple state variables per key — you're not stuck cramming everything into one object.
And because it runs on EMR Serverless, you're not managing a cluster either. You submit a Spark job, it scales up and down as your event volume changes, and the driver stays alive between micro-batches so there's no restart overhead.

A simple architecture

Here's roughly what the pipeline looks like end to end:

Devices or apps send events into Kinesis or MSK. EMR reads that stream with Spark Structured Streaming, keeps track of per-key state (like "last time I heard from device X"), and writes results out — to S3 for the data lake, to OpenSearch or DynamoDB for quick lookups, and to SNS when something needs a human's attention.

The example: catching a device that goes quiet

Say you have a fleet of IoT devices, each sending a heartbeat every few seconds. You want to know within 30 seconds if one stops — and get reminded every 60 seconds until it comes back.
Here's what that looks like over time:

Every heartbeat resets a 30-second timer. If the timer ever fires without a new heartbeat showing up first, we send an alert and set another timer for 60 seconds later, just in case the device is still offline. The moment it reconnects, we cancel everything and go back to normal.

What the code roughly looks like

You don't need much code to get this behavior — three methods on a processor class:
class DeviceWatcher(StatefulProcessor):

def init(self, handle):
# remember the last time we heard from this device
self.last_seen = handle.getValueState("last_seen", schema)

def handleInputRows(self, key, rows, timer_values):
latest_ts = get_latest_timestamp(rows)
if latest_ts:
for t in self.handle.listTimers():
self.handle.deleteTimer(t) # cancel the old timer
self.last_seen.update((latest_ts,))
self.handle.registerTimer(
timer_values.getCurrentProcessingTimeInMs() + 30_000
)
yield empty_output()

def handleExpiredTimer(self, key, timer_values, info):
# no heartbeat arrived in time — raise the alarm
self.handle.registerTimer(
timer_values.getCurrentProcessingTimeInMs() + 60_000
)
yield offline_alert(key, self.last_seen.get())

Then you plug it into a streaming query like this:
alerts = (
events_df.groupBy("device_id")
.transformWithStateInPandas(
statefulProcessor=DeviceWatcher(),
outputStructType=alert_schema,
outputMode="append",
timeMode="processingTime",
)
)

alerts.writeStream \
.foreachBatch(send_to_sns) \
.option("checkpointLocation", "s3://my-bucket/checkpoints/") \
.trigger(processingTime="10 seconds") \
.start()

That's it. No manual timer queue, no custom serialization — Spark handles saving state to RocksDB and checkpointing it to S3 for you, so if the job restarts, it picks up exactly where it left off.

Getting it running on EMR Serverless

The setup is a handful of CLI commands:
# 1. Create the application, pinned to a Spark 4.0 release
aws emr-serverless create-application \
--name "device-watcher" \
--release-label "emr-spark-8.0.0" \
--type "SPARK" \
--network-configuration '{"subnetIds":["subnet-xxx"],"securityGroupIds":["sg-xxx"]}'

# 2. Submit the streaming job
aws emr-serverless start-job-run \
--application-id <APP_ID> \
--execution-role-arn arn:aws:iam::<ACCOUNT_ID>:role/EMRServerlessRole \
--job-driver file://job-driver.json \
--mode STREAMING \
--retry-policy maxFailedAttemptsPerHour=1

From there, CloudWatch and EMR Studio give you visibility into throughput, batch duration, and — this one matters a lot for stateful jobs — how big your state store is getting over time.

A few things worth remembering

  • Set a TTL on your state. Don't let it grow forever — old devices, closed accounts, and finished sessions should expire.
  • Never change your checkpoint location on a whim. It's basically the job's memory. Changing it means starting fresh.
  • Watch backlog, not just uptime. A job can be "running" and still be falling behind.

Wrapping up

Stateful streaming used to mean choosing between "simple but limited" and "powerful but painful to operate."  transformWithState  on Amazon EMR Serverless closes a lot of that gap — native timers, automatic state cleanup, and no cluster to manage. If you've been putting off a use case like this because of the operational overhead, it's worth another look.
If you want the full, production-ready version of the heartbeat monitoring example — complete code, IAM policies, and test scripts — the AWS Big Data Blog has an excellent deep dive:  Build stateful streaming applications with Apache Spark 4.0 on Amazon EMR Serverless .
Have you built something similar? I'd love to hear how you're handling state size and TTL in production.
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article