Before trusting at-least-once delivery: SIGKILLing BullMQ workers on Amazon EC2

Setup

• EC2 t3.medium (2 vCPU, 4 GB RAM), Ubuntu 24.04, US region
• Node v22.14.0, BullMQ 6.3.15, ioredis 6.0.0
• Redis 7.2.4 built from source, persistence disabled for the worker tests (I wanted worker-crash behavior isolated from persistence behavior)
• 30 jobs per variant, fixed job IDs, 5 trials per variant. Worker concurrency 5, lockDuration 3000 ms, stalledInterval 1000 ms
• Worker 1 gets SIGKILL once 5 jobs are in flight; worker 2 starts 500 ms later. Redis and the queue stay up throughout
• The "side effect" is an append of the job ID to an effect log (variant F: a push to a Redis list). Comparing the start log against the effect log tells me which jobs ran twice, which effects were written twice, and which were never written
• Metrics exported on :9101, scraped by Prometheus 3.15.0, graphed in Grafana 11.2.0, watched from my laptop over an SSH port-forward

• A, no guard: effect written at job start, then a 1.5 s wait. maxStalledCount 1.
• B, guard with no gap: a Redis SADD dedupe key checked immediately before writing the effect.
• C: same as A, but maxStalledCount 0.
• E, guard with a gap: SADD dedupe key, then a 1.5 s wait, then the effect only if the key was newly set. The kill lands inside that wait, between guard and effect.
• F, atomic guard + effect: one Redis Lua script performs the SADD and the effect push as a single atomic step at job start, then a 1.5 s wait.
• D, separate setup: 1 job, worker concurrency 1, the event loop blocked synchronously for 6 s before the effect, lockDuration 3 s, second worker started 1 s later, no kill.

Results (5 trials per variant, identical counts in every trial)

What the numbers actually say

1. The queue survives the crash. In A, B, E and F, all 30 jobs reached a terminal state, zero lost from the queue's point of view. The dead worker's locks expired, the stalled checker found the 5 in-flight jobs, and they were redelivered. BullMQ's recovery works exactly as advertised.
2. At-least-once is literal. The 5 jobs in flight at the kill started a second time, in every variant that allows re-runs, including the fully protected one. Redelivery is not an edge case; it is the design. If your side effect is not idempotent, a crash runs it again.
3. With no guard, every re-run is a duplicate effect. Variant A: 5 jobs restarted, all 5 wrote their effect twice.
4. maxStalledCount=0 converts silent duplicates into visible failures. Variant C: the 5 stalled jobs failed with "job stalled more than allowable limit", no re-runs, 25 completed. That is a real operational choice: you lose the 5 jobs loudly instead of duplicating them silently. For anything customer-facing, loud and lost beats silent and duplicated - as long as someone watches the failed set.
5. A two-step guard can be worse than no guard. Variant E's SADD check looks like the fix, and in variant B - where the check and the effect happen back to back - it is: 0 duplicates. But E puts a gap between the two steps, the crash lands in the gap, and the outcome flips: 0 duplicates, but 5 jobs completed with their effect never written. The re-run sees the dedupe key already set and skips the effect. The queue reports success. Nothing looks wrong. The work simply never happened.
6. The only variant with zero duplicates AND zero missing effects was the atomic one. F's single Lua script makes the dedupe check and the effect commit together, so there is no gap for a crash to land in. Same queue, same kill, same 5 re-runs - the second start finds the work already done and does nothing.
7. You don't even need a crash to get duplicates. Variant D blocks the event loop for 6 seconds, longer than the 3-second lock. The lock expires while the worker is alive and busy, BullMQ re-runs the job on the second worker, and the effect is written twice. The first worker then fails to finish cleanly ("Missing lock for job"). Any event-loop stall past your lockDuration - a big JSON parse, a sync crypto call, a GC pause - is a duplicate generator.

The honest limit of variant F

What I took away

• Assume every job can run twice. Design for it; the queue will not save you.
• "Completed" does not mean "done". Variant E's 5 jobs completed with their effect never written.
• Exactly-once side effects are your responsibility. BullMQ gives you redelivery; atomic dedupe-plus-effect, or a receiver that honors idempotency keys, is what makes redelivery safe.
• Decide your failure mode before it decides you: duplicate loudly, fail visibly (maxStalledCount 0), or dedupe atomically.
• Watch your event loop. A 6-second block duplicated a job with no crash at all.