The Lambda bug that shows up six months later
Durable functions re-run your handler from line one on every resume, so anything outside a step() runs again. A walkthrough of a saga that silently skips its own rollback, why unit tests never catch it, and the checker I built after finding nothing that verifies the rule AWS documents.
I was reading the durable functions docs when I hit a line that I had to read
three times:
three times:
Any code that is not inside a durable operation must be a pure function of
the handler inputs and the results of completed operations.
That's a real constraint. No clocks, no randomness, no I/O, no touching
shared state, unless it's inside a
thing that checks it, because a constraint that strict usually comes with a
linter or a compiler error or at minimum a runtime warning.
shared state, unless it's inside a
step(). And then I went looking for the
thing that checks it, because a constraint that strict usually comes with a
linter or a compiler error or at minimum a runtime warning.
There isn't one. You just have to not do it.
Why the rule exists
Durable functions can suspend for up to 366 days. When one resumes, Lambda
doesn't pick up mid-function the way you might imagine. It runs your handler
again from line one. For every step that already finished, the SDK hands back
the checkpointed result instead of running the body a second time.
doesn't pick up mid-function the way you might imagine. It runs your handler
again from line one. For every step that already finished, the SDK hands back
the checkpointed result instead of running the body a second time.
So your handler body executes many times over the life of one logical
execution, and only the step bodies are protected. Everything else runs again
on every single resume.
execution, and only the step bodies are protected. Everything else runs again
on every single resume.
Once you see it that way the rule stops being arbitrary. It's just describing
what has to be true for replay to work.
what has to be true for replay to work.
What it looks like when you get it wrong
Here's the smallest version I can write that actually breaks:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
const handler = async (event, context) => {
const completedSteps = [];
await context.step('reserve-inventory', async () => {
await reserveInventory(event.orderId);
completedSteps.push('inventory');
});
await context.step('charge-payment', async () => {
await chargeCard(event.orderId);
completedSteps.push('payment');
});
if (somethingWentWrong) {
for (const step of completedSteps.reverse()) {
await compensate(step);
}
}
};Read it top to bottom and it's fine. It's a saga. You track what succeeded so
you can roll it back if a later step blows up.
you can roll it back if a later step blows up.
Now resume it. The handler restarts,
an empty array, and both steps are already checkpointed so their bodies never
run. Those two
completedSteps gets re-initialised to
an empty array, and both steps are already checkpointed so their bodies never
run. Those two
push calls don't happen.The compensation loop then iterates over nothing, rolls back nothing, and
returns like it worked. Your inventory stays reserved. Nothing throws,
nothing logs an error, and if you're watching a dashboard you see a
successful execution.
returns like it worked. Your inventory stays reserved. Nothing throws,
nothing logs an error, and if you're watching a dashboard you see a
successful execution.
I didn't invent that example. I found it in a public durable-functions repo,
in the file being offered as the reference saga implementation. And the thing
that makes it nasty is that it's usually invisible: three fast steps normally
finish inside one invocation, so the bug only fires when the execution
actually gets interrupted. Which is the exact situation a saga exists for.
in the file being offered as the reference saga implementation. And the thing
that makes it nasty is that it's usually invisible: three fast steps normally
finish inside one invocation, so the bug only fires when the execution
actually gets interrupted. Which is the exact situation a saga exists for.
There's a worse variant too. If the interruption lands in the middle of
compensation, you get partial rollback reported as complete.
compensation, you get partial rollback reported as complete.
Your tests won't help
This is the part that bothered me enough to build something.
Unit tests call the handler. The handler runs start to finish. Everything
passes, because the first run is always correct. That's the defining property
of the bug. To catch it you'd need a test that suspends an execution and
resumes it, and I have never once seen someone write that test.
passes, because the first run is always correct. That's the defining property
of the bug. To catch it you'd need a test that suspends an execution and
resumes it, and I have never once seen someone write that test.
So the failure lands months later, in production, on a workflow that's
holding real state. AWS's docs mention double-charging as one of the possible
outcomes, which tells you roughly what neighbourhood we're in.
holding real state. AWS's docs mention double-charging as one of the possible
outcomes, which tells you roughly what neighbourhood we're in.
So I wrote a checker
It's called replayguard and it
works two different ways, which turned out to matter more than I expected.
works two different ways, which turned out to matter more than I expected.
The static half parses your handlers and looks for what's visible in the
source: clocks outside a step, I/O outside a step, a step body writing to
something it doesn't own, branches on nondeterministic values, and step names
built from unstable inputs.
source: clocks outside a step, I/O outside a step, a step body writing to
something it doesn't own, branches on nondeterministic values, and step names
built from unstable inputs.
That last one caught me off guard when I first understood it. Checkpoints get
matched by name and order. So if you name a step with a template literal
containing
it to the existing checkpoint, and your step runs again. Side effects and
all. I found this one in an AWS sample, where a task id was being built as
matched by name and order. So if you name a step with a template literal
containing
Date.now(), the name is different on replay, the SDK can't match
it to the existing checkpoint, and your step runs again. Side effects and
all. I found this one in an AWS sample, where a task id was being built as
`task-${Date.now()}`.Doing this properly means real scope resolution, not grep. To know whether
belongs to the step body or to the enclosing function, and no amount of
pattern matching gets you there. There's also the small problem that the four
SDKs don't agree on shape:
completedSteps.push() is a violation you have to know whether that variable
belongs to the step body or to the enclosing function, and no amount of
pattern matching gets you there. There's also the small problem that the four
SDKs don't agree on shape:
| Handler | Step | |
|---|---|---|
| Python | @durable_execution | context.step(fn, name="x") |
| JS/TS | withDurableExecution(fn) | context.step("x", fn) |
| Java | extends DurableHandler<,> | ctx.step("x", Result.class, fn) |
| Rust | param typed *Context | ctx.step("x", || async {...}) |
The step body is in a different argument slot in every one of them, and Java
has a two-argument overload on top of that.
has a two-argument overload on top of that.
The dynamic half is the part I'd actually recommend if you only adopt one. It
runs your handler twice, once normally and once with the clock shoved forward
13 hours and the entropy reseeded, then diffs the sequence of operations each
run asked for.
runs your handler twice, once normally and once with the clock shoved forward
13 hours and the entropy reseeded, then diffs the sequence of operations each
run asked for.
1
2
3
4
from replayguard.dynamic import assert_deterministic
def test_handler_is_deterministic():
assert_deterministic(handler, {"orderId": "A1"})The reason this earns its place is that static analysis just can't see some
of the causes. If the nondeterminism is buried in a library you didn't write,
or the value got tainted six hops upstream, or your loop is iterating a set,
no rule is going to catch it. The dynamic harness doesn't need to know the
cause. It only needs to notice that the shape of the execution moved.
of the causes. If the nondeterminism is buried in a library you didn't write,
or the value got tainted six hops upstream, or your loop is iterating a set,
no rule is going to catch it. The dynamic harness doesn't need to know the
cause. It only needs to notice that the shape of the execution moved.
A handler whose step order comes out of
under perturbation. I can't write a static rule for that and I don't think
anyone could.
random.sample diverges immediately
under perturbation. I can't write a static rule for that and I don't think
anyone could.
What happened when I pointed it at real code
I ran it over 1,547 files of durable-function code written by about a dozen
other people. AWS's samples, all three official SDKs, eleven community repos,
and the community Rust SDK.
other people. AWS's samples, all three official SDKs, eleven community repos,
and the community Rust SDK.
The findings were interesting. The more useful outcome was that it found
thirteen bugs in my own tool, none of which my hand-written test fixtures
could have caught, for the obvious reason that I wrote both the fixtures and
the analyser and so my blind spots were in both. Step bodies passed as
keyword arguments.
that get deferred instead of running where they're written. Every one of
those came from somebody whose habits differ from mine.
thirteen bugs in my own tool, none of which my hand-written test fixtures
could have caught, for the obvious reason that I wrote both the fixtures and
the analyser and so my blind spots were in both. Step bodies passed as
keyword arguments.
n++ not registering as a mutation. Decorated step bodies
that get deferred instead of running where they're written. Every one of
those came from somebody whose habits differ from mine.
I'll also give you the unflattering result, because it's the one I'd want if
I were reading this.
I were reading this.
Three of the six rules have never found a real bug in anyone else's code. Not
once. They aren't broken, and I checked that carefully rather than assuming:
each one fires on its own violation in all four languages, and that runs as a
test on every commit. They're quiet because published example code just
doesn't contain those mistakes. I went through all 837 handlers by hand to be
sure. 85 branch conditions inside durable regions, every one of them testing
stable data. 38 computed step names, not one built from a clock.
once. They aren't broken, and I checked that carefully rather than assuming:
each one fires on its own violation in all four languages, and that runs as a
test on every commit. They're quiet because published example code just
doesn't contain those mistakes. I went through all 837 handlers by hand to be
sure. 85 branch conditions inside durable regions, every one of them testing
stable data. 38 computed step names, not one built from a clock.
So those rules work and remain unproven, which is a distinction I care about
enough to put in the README above the install instructions.
enough to put in the README above the install instructions.
If you want to try it
1
2
pip install 'replayguard[all]'
replayguard check src/It emits SARIF, so in CI the findings land as annotations on the pull request
that introduced them, which for a bug class nobody notices for six months is
really the only delivery mechanism that makes sense:
that introduced them, which for a bug class nobody notices for six months is
really the only delivery mechanism that makes sense:
1
2
3
- uses: amrutp24/replayguard@v0.1.1
with:
path: src/Two things to know before you trust a green run. It doesn't follow calls across files yet, so a handler that passes its I/O off to another module comes back clean. And the dynamic side can't prove determinism, only fail to disprove it under one particular perturbation. The report says exactly that instead of implying a clean bill of health.
Which brings me to the one thing I actually want out of writing this. Every number above came from example code, and example code is written carefully by people who are trying to demonstrate the right pattern. Nobody writes a demo where they build a step name out of
Date.now() at 6pm on a Friday.So if you're running durable functions on something real, point this at it. I want to know either way: a bug it caught, or noise it raised that it shouldn't have. One run against a codebase written under deadline is worth more to me than another thousand files of samples, and right now that data doesn't exist anywhere I can reach.
Source, and the full validation record including everything above:
github.com/amrutp24/replayguard
github.com/amrutp24/replayguard
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article