Backend Engineer Must Know Interview Concepts for Production Series - Part 1: Idempotency on AWS
Learn how to prevent duplicate orders with idempotency keys, AWS Lambda, and DynamoDB. Includes practical examples, retry handling, and crash recovery.
Series: Backend Interview Concepts to Know for Production (4 articles)
- 1Backend Engineer Must Know Interview Concepts for Production Series - Part 1: Idempotency on AWS This article
Idempotency means retrying the same logical operation without creating an additional side effect. For an order API, that means repeated attempts can return the same order instead of creating another one. On AWS, stable operation IDs, durable records in Amazon DynamoDB, and idempotent AWS Lambda handlers help make that possible.
You tap Place Order for a $22 burrito. The app spins. You tap again.
Congratulations. Two burritos. Your retry button is now a bulk-order feature.
Somewhere, a backend engineer is explaining that both requests succeeded. The customer would prefer an explanation for the second charge.
Welcome to Part 1 of Backend Engineer Interview Concepts for Production. We'll use that checkout failure to understand idempotency, build a small AWS example, and cover the crash scenario that a key table alone cannot fix.
Why can a retry create two orders?
The first request may have worked perfectly:
- Your app sends
POST /orders.
- The server creates the order.
- The response gets lost before reaching your phone.
- The app times out and retries.
- Without duplicate protection, the server creates another order.
The kitchen has your order. Your phone has a spinner. Both are reporting their own experience accurately.
A timeout means “I don't know what happened.” It does not prove that nothing happened.
The request can fail from the client's perspective after the business operation has already succeeded.
What counts as the “same operation”?
A side effect is a change your application makes: creating an order, charging a card, sending an email, or decreasing inventory.
Two HTTP attempts can represent one purchase. Two intentional purchases represent two operations, even when the customer orders the exact same burrito both times.
That distinction matters. If you deduplicate only by customer and product, you might prevent someone from ordering lunch tomorrow. A bold retention strategy.
Idempotency protects the intended change. It does not require every line of code to run only once, and it does not make every response byte identical. Returning a saved response is one useful implementation pattern.
How does an idempotency key work?
The client creates a unique key when a purchase starts and keeps it for every retry of that purchase. A UUID generated once for the purchase is a common choice:
POST /ordersIdempotency-Key: order_8f29a1Content-Type: application/json
{"sku": "burrito", "quantity": 1}The header carries an identifier. Your backend still has to implement the behavior behind it.
For a completed request, the server can remember:
{ "operation_id": "customer_42#create-order#order_8f29a1", "status": "COMPLETE", "result": {"order_id": "order_123"}}On a retry, the backend recognizes the same operation and returns
order_123 again. It doesn't ask the kitchen to start another burrito.The full decision needs more than “seen it” or “haven't seen it”:
Record state | What the backend should do |
|---|---|
No valid record | Atomically claim the operation before starting work. |
In progress | Prevent a second worker from repeating the side effect. Ask the caller to retry later or check status. |
Complete | Return the stored outcome. |
Same key, different business input | Reject the conflicting request. |
Same purchase, same key. New purchase, new key. Persist that key across the retry paths your app supports, including a page refresh if the purchase can resume afterward.
Scope the key to the authenticated customer and operation type. A client-supplied key is an identifier, not proof that the caller owns an order.
Where AWS fits: SQS, Lambda, and DynamoDB
Duplicates also arrive when nobody touches a button.
Amazon SQS standard queues provide at-least-once delivery, so a message can be received more than once. Lambda's SQS event source mapping can also process a record again. Your consumer needs to tolerate this. AWS documents this behavior in the SQS delivery guide and Lambda's SQS integration guide .
Imagine an order worker completes its work, then fails before successful processing is acknowledged. The message becomes available again. The queue has another delivery attempt; the customer still intended one order.
Here's a useful division of responsibilities:
- Amazon SQS buffers the work and delivers messages to the consumer.
- AWS Lambda runs the order-processing code.
- Amazon DynamoDB holds durable operation state that concurrent Lambda invocations can share.
- Powertools for AWS Lambda supplies an idempotency utility for the claim, result, and duplicate-handling logic.
Carry a business operation ID through the workflow. A fresh Lambda invocation ID is not a substitute.
DynamoDB makes the claim practical with conditional writes. A condition such as
attribute_not_exists(id) lets a write succeed only if that item does not already exist. The check and write happen together, so two workers cannot both successfully claim the same absent item. A separate “read, then write” sequence has a race between those steps. See DynamoDB conditional writes .For an HTTP application, Amazon API Gateway can route
POST /orders to Lambda. Neither adding the header nor choosing API Gateway automatically deduplicates your business operation.For an SQS consumer, apply idempotency per logical record, using an operation ID in the message body. Don't use the whole changing batch as the identity. Partial batch responses can reduce unnecessary retries of successful records, but they do not replace idempotent processing.
A small Lambda example that creates one order
Let's keep the demo focused: one Lambda function, two DynamoDB tables, and an order record. This example does not charge a card, reserve inventory, or send an email. Those are separate effects that need their own protection.
To keep the identifiers simple, this demo uses the operation ID as the order's primary key.
Powertools can select the key from a payload field, validate other fields, and save completed results. Its idempotency documentation covers the configuration and lifecycle.
Set up these resources:
Resource | Configuration |
|---|---|
Idempotency table | String partition key id ; TTL attribute expiration . |
Orders table | String partition key order_id ; keep these business records beyond the retry window. |
Lambda environment | IDEMPOTENCY_TABLE and ORDERS_TABLE set to the corresponding table names. |
Lambda role | GetItem , PutItem , UpdateItem , DeleteItem on the idempotency table; GetItem , PutItem on the orders table. |
Package
aws-lambda-powertools and boto3 with the function. The companion example includes an AWS SAM template and a local test suite.This is a direct Lambda test event, not an API Gateway or SQS envelope:
{ "operation_id": "customer_42#create-order#order_8f29a1", "order": {"sku": "burrito", "quantity": 1}}In an application, validate the request and derive the customer portion from trusted authentication context before invoking this handler. Don't accept someone else's customer ID from an untrusted body. Keep request metadata such as timestamps out of the business input being compared.
import os
import boto3from botocore.exceptions import ClientErrorfrom aws_lambda_powertools.utilities.idempotency import ( DynamoDBPersistenceLayer, IdempotencyConfig, idempotent,)
persistence = DynamoDBPersistenceLayer( table_name=os.environ["IDEMPOTENCY_TABLE"])orders = boto3.resource("dynamodb").Table(os.environ["ORDERS_TABLE"])config = IdempotencyConfig( event_key_jmespath="operation_id", payload_validation_jmespath="order", raise_on_no_idempotency_key=True, expires_after_seconds=86400,)
def create_order_once(operation_id, order): item = {"order_id": operation_id, "order": order} try: orders.put_item( Item=item, ConditionExpression="attribute_not_exists(order_id)", ) except ClientError as error: if error.response["Error"]["Code"] != "ConditionalCheckFailedException": raise existing = orders.get_item( Key={"order_id": operation_id}, ConsistentRead=True )["Item"] if existing["order"] != order: raise ValueError("This operation ID already belongs to a different order") return {"order_id": operation_id, "status": "CREATED"}
@idempotent(config=config, persistence_store=persistence)def lambda_handler(event, context): return create_order_once(event["operation_id"], event["order"])Two protections are doing different jobs:
- Powertools manages repeated requests. Within the configured window, it reuses a completed result and rejects a concurrent attempt while the first is still running. The
orderfield is checked for conflicting input.
- The order write protects the business effect. Its stable
order_idand conditional write prevent a second order if the handler runs again after creating the first one. If the item already exists, the code verifies that the order matches.
The code returns the original creation receipt, not a live view of the order's later shipping or payment status.
Try the event twice: you should get the same result and find one order item. Give a new purchase a new
operation_id : that should create a second item. Change the quantity while keeping the original ID: that should fail.For an HTTP integration, map validation and in-progress outcomes to documented client responses. For SQS, leave unsuccessful records eligible for retry rather than acknowledging them as completed work.
The crash after success: why a key table isn't enough
Now put an external payment call into the story:
- The function claims the key.
- The payment provider charges $22.
- The function crashes.
- The completed result never reaches the idempotency table.
The record says “in progress.” The credit card says “very much charged.”
When the claim expires, retrying the function may repeat the payment unless that payment operation has its own protection. A lock timeout tells you the worker may be gone. It does not tell you the charge never happened.
The uncertainty sits between systems. Recording a key in DynamoDB cannot roll back a charge in a payment provider.
The right solution depends on where the change happens:
- One database: a transaction can commit the business change and its idempotency record together. For a simple create, a stable business key plus a conditional insert can make the write itself idempotent, as in our example.
- An external payment provider: pass a stable, correctly scoped key to the provider's idempotency feature when supported. Follow its retention and parameter rules. If the outcome remains unclear, reconcile it with the provider before initiating a new charge.
- A database write followed by publishing an event: a transactional outbox can store the business change and the intent to publish in one transaction. The publisher can still deliver duplicates, so consumers still need idempotency. AWS explains this boundary in its transactional outbox guidance .
This is why the demo's second conditional write matters. If the order was created but Powertools never saved its result, a later attempt can recover the same order. Adding an unprotected
charge_card() call would change that guarantee.The mistakes worth catching before production
Generating a fresh key on retry. A new identity makes a duplicate look like new work. Create the key once for the purchase and carry it through retries.
Reusing a key with different input. One burrito and six burritos are different requests. Validate the fields that define the operation and reject mismatches.
Treating expiry as proof of failure. Keep records long enough for your retry and replay paths, including delayed jobs and dead-letter queue redrives. The demo's 24-hour window is an example, not a universal setting. DynamoDB TTL cleanup is asynchronous; expired items can remain for days, so application logic must evaluate expiry rather than wait for deletion. See DynamoDB TTL behavior .
Using only process memory. Another Lambda execution environment will not share your local dictionary. Correctness needs durable state across workers.
Assuming a queue protects every downstream effect. A queue cannot prevent a payment API from executing twice when your consumer calls it twice.
Use these six checks when reviewing a retryable write path.
Where else would you use idempotency?
App feature | One logical operation | Duplicate effect to prevent |
|---|---|---|
Checkout | Purchase attempt order_8f29a1 | A second order or charge. |
Welcome email | The welcome message for one signup | Welcoming the same person five times. Friendly, then alarming. |
Monthly invoice | One customer's invoice for one billing period | Two invoice documents for the same obligation. |
Inventory reservation | Reserving stock for one order | Reducing available stock twice. |
The identifier follows the business meaning. A monthly invoice needs the billing period; a checkout needs a purchase identity. There is no universally correct “deduplicate by user ID” rule.
Each row also needs protection where its effect occurs. Recording “email sent” after calling an email service still leaves a crash window if that service cannot deduplicate a retry.
How would you explain idempotency in a backend interview?
Idempotency means repeated attempts for the same logical operation do not create additional business effects. For a retryable order API, I'd use a stable key scoped to the caller and operation, an atomic claim in durable storage, payload validation, and a stored result. On AWS, Lambda with DynamoDB and Powertools is a useful implementation. I'd also explain how concurrent requests, record expiry, and a crash after an external side effect are handled.
The definition gets you started. Explaining the crash window shows that you've thought about what happens after the happy-path demo.
Questions developers ask about idempotency on AWS
Does SQS FIFO remove the need for idempotent consumers?
No. FIFO queues can deduplicate repeated sends within their five-minute deduplication interval. That guarantee does not make your consumer's external side effects atomic. Messages can be processed again if processing is not successfully acknowledged. Compare SQS FIFO deduplication with Lambda's duplicate-processing guidance .
Are POST requests idempotent?
POST is not inherently idempotent under HTTP semantics. Your application can define a retry-safe POST operation using keys, stable resource identities, and duplicate handling. PUT and DELETE have idempotent semantics, although repeated responses can differ. See HTTP's definition of idempotent methods .
Does disabling the submit button solve duplicate requests?
It helps the interface. It does not cover automatic retries, page reloads, concurrent clients, or message redelivery. Keep it as a usability improvement and protect the backend separately.
Is idempotency the same as exactly-once execution?
No. Code and message delivery may repeat. Idempotency is about preserving the intended effect when they do. Always name the operation, the storage or provider boundary, and the retention assumptions behind a “once” claim.
One experiment to try
Pick a write operation in your app and repeat the same request. Then try it concurrently. Then simulate a failure after the business change but before the response is saved.
Count the business effects, not just successful HTTP responses.
Retry the request. Not the damage.
Next in the series: database indexes and query execution. Your database shouldn't need to meet every customer just to find one email address.
Series: Backend Interview Concepts to Know for Production (4 articles)
- 1Backend Engineer Must Know Interview Concepts for Production Series - Part 1: Idempotency on AWS This article
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article