AWS Builder Center
The retry fix that rejected the next order: a Lambda and SQS failure lab

The retry fix that rejected the next order: a Lambda and SQS failure lab

A real AWS failure lab exposed two bugs: duplicate fulfillment and an idempotency key that rejected a valid order. Here is the evidence, the DynamoDB integration failure, and a regression you can run locally.

A retry can fulfill an order twice. An idempotency key can prevent that—and still leave a different valid order unfulfilled.
That was the counterexample at the center of ReplayGuard, the failure-and-repair lab we built for WeMakeDevs First Commit. We sent two orders for the same product through three candidate handlers on AWS, then counted the durable fulfillment receipts independently.
Here is what the final experiment actually observed:
Key choice
ORDER-A receipts
ORDER-B receipts
Business result
No key
2
2
Duplicate fulfillment
Product SKU
1
0
A different valid order rejected
Business order ID
1
1
Both orders preserved
The SKU-key handler is the interesting failure. A test containing only one order and its retry would miss it.

Start with the business invariant

Our invariant was simple:
Every expected business order must have exactly one correct fulfillment receipt after the declared deliveries.
A worker returning successfully cannot establish that invariant. A queue becoming empty cannot establish it either: a handler can acknowledge a rejected order without fulfilling it.
We used two distinct orders,  ORDER-A  and  ORDER-B , each requesting one  FIELD-NOTES-3PACK . Their product matched; their business identities did not.
A product key answers, “Have I seen this product?” The operation needed to answer, “Have I already fulfilled this order?”
The order ID was sufficient within this lab's run and candidate namespace. A production key needs the application's actual scope: tenant, operation and stable business identity, with deliberate rules for changed payloads and retention.

Put the fault after the effect

AWS documents that  Lambda event source mappings process events at least once , so duplicate processing can occur. We made that behavior observable with a small private lab:
  1. A Lambda event-source mapping polls a standard Amazon SQS queue and invokes the worker.
  1. The worker calls a separate fulfillment Lambda.
  1. The fulfillment Lambda commits a receipt in Amazon DynamoDB.
  1. On the configured first-receive fault, it raises an exception before returning a successful response.
  1. The worker invocation fails. After visibility expires, SQS can redeliver the message.
  1. An observer reads DynamoDB independently and correlates CloudWatch message IDs, invocation IDs and fault events.
SQS triggers a worker Lambda. Fulfillment commits a DynamoDB receipt, then fails. An independent observer checks receipts and CloudWatch evidence.

The receipt is the complete simulated fulfillment effect. This experiment creates no real shipment or payment.
The experiment sends six original messages across the three candidate strategies. It waits until each A receipt and its correlated failure are observed before sending the three B messages. SQS determines the subsequent retry timing; we do not assume a total delivery order or exactly two attempts.
Our earlier experiment had crashed the worker after a successful fulfillment response. This later comparison moves the fault into the fulfillment response boundary, immediately after commit. Both expose the gap between an effect happening and the caller knowing it happened.

The receiver must own the decision

In the receiver, the chosen key determines the receipt identity. A conditional DynamoDB write creates the receipt only if that identity is absent. On a collision, the receiver reads the existing receipt consistently and compares the full order.
The essential write condition in the implementation is:
ledger().put_item(
Item=receipt,
ConditionExpression="attribute_not_exists(PK) AND attribute_not_exists(SK)",
)
 DynamoDB condition expressions  make the create decision conditional on the existing item. Because our receipt is the modeled effect, the effect and its deduplication decision live in the same write.
The resulting behavior is:
  • No key creates a fresh receipt identity on each fulfillment.
  • An order key lets a retry reuse its existing receipt.
  • A SKU key collides when B follows A. The payload comparison detects that B is a different order and returns a terminal rejection.
The SKU repair prevents a second receipt for A, but its key is too broad to preserve B.
A separate “processed” flag beside an external shipping or payment API would not automatically provide this atomicity. Those integrations need their own supported effect contract.

The first convincing-looking result was unresolved

Our first three-candidate AWS attempt produced the expected-looking receipt counts. It still did not pass.
The conflict-logging path tried to serialize an existing DynamoDB order. Boto3 had deserialized its numeric quantity as a Python  Decimal . The original local fixture had supplied an  int .
 json.dumps  raised  TypeError  before the provider could emit the payload-conflict event or return the intended rejection. B kept failing and reached the dead-letter queue.
The zero receipt count was real. The explanation we wanted to claim was not yet established.
ReplayGuard kept the SKU candidate incomplete and the experiment unresolved. We retained that report and the source that produced it.
The repair normalizes only supported, finite, integral quantities from  Decimal  to  int , validates the stored order, and uses a copy for comparison and logging. We added a local regression using boto3's actual serializer/deserializer against the archived and repaired source, with networking blocked.
A later audit also found verifier gaps around missing source hashes and contradictory terminal events. We hardened the checker and ran a fresh AWS experiment with that checker captured alongside the evidence.
The final run passed 22/22 experiment checks, including the evidence needed to identify the two faulty candidates. That means the experiment established its declared outcomes; it does not mean all three handlers passed.

Keep the failure as a regression

A recorded cloud failure becomes more useful when someone else can inspect it and test a repair against the same case.
ReplayGuard provides two separate downloads:
  • The  local repair kit  executes editable Python adapters against bounded local deliveries and a simulated receipt receiver.
After extracting the AWS archive, verify the captured evidence without AWS credentials or third-party packages:
python3 scripts/verify_key_case.py evidence.json --case case.json --json
That checks the recorded run. The archive's README separately explains deployment, cost checks and teardown for a fresh AWS experiment.
For an editable exercise, extract the local repair kit into a different directory. Use Python 3.11+ on macOS or Linux, and run these commands from the extracted kit root:
mkdir -p candidate
cp repair_lab/adapters/business_key.py candidate/adapter.py
python3 -I -S scripts/test_repair.py --adapter candidate/adapter.py --case interleaved-retries --output before.json
Expect exit 0 and one receipt for each order. In  candidate/adapter.py , change:
key=order["orderId"]
to:
key=order["sku"]
Run the identical case again:
python3 -I -S scripts/test_repair.py --adapter candidate/adapter.py --case interleaved-retries --output broken.json
Expect exit 1: A has one receipt and B has none. Restore the original source and rerun:
cp repair_lab/adapters/business_key.py candidate/adapter.py
python3 -I -S scripts/test_repair.py --adapter candidate/adapter.py --case interleaved-retries --output restored.json
Expect exit 0 and one receipt for each order again. Our  recorded clean-extraction tutorial  observed exactly that sequence: 1/1 → 1/0 → 1/1, with identical inputs and fault conditions.
Import those JSON reports into the  repair inspector . The browser recomputes outcomes from supplied observations. The terminal commands execute Python; the static website does not.

What AWS made clear

SQS supplied actual redelivery, DynamoDB supplied durable receipts and strongly consistent reads, and CloudWatch supplied the correlation trail. Those services made the gap between a failed invocation and a completed effect inspectable.
The setup also exposed practical constraints:
  • Our region's Lambda concurrency quota was 10. Reserving two slots failed because of the unreserved-pool requirement. The lab uses a preflight fallback to an SQS event-source mapping maximum concurrency of two.
  • The earliest 128 MB functions timed out during initialization before reaching the intended fault. For that original experiment, 512 MB functions and coordinated worker, provider and visibility timeouts allowed the intended failure to occur.
  • CloudFormation made deployment reproducible. Finite sends, bounded collection, short retention and explicit teardown kept the experiments small.
The final experimental stack's deletion was verified. AWS Amplify hosts the static viewer separately. Credits, processing expiry and DynamoDB TTL are useful controls, but none is a hard spending cap or a substitute for resource deletion.

The test I would keep

When reviewing an idempotency repair, add a second valid business operation that shares a tempting but insufficient key.
Then inject the fault after the effect commits and check both operations independently. Require evidence for the fault and final outcomes. Preserve the failed source and report when the repair breaks an invariant.
ReplayGuard's pass applies to declared cases and its simulated receiver contract. Missing evidence stays incomplete; unsupported effects stay unresolved. That boundary is what makes a passing result useful.
Keep the failure. Test the repair.
Built for WeMakeDevs First Commit by SudoByte. OpenAI Codex and Claude Code assisted with implementation, testing, design and article preparation.
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article