AWS Builder Center

Builder Center

Your Event Arrived Twice: Building Idempotent Serverless Pipelines on AWS

Your Event Arrived Twice: Building Idempotent Serverless Pipelines on AWS

Retries and duplicate events are normal in distributed systems, but they can corrupt business outcomes when pipelines are not designed for them. Learn how idempotency keys, durable state, retry policies, dead-letter queues, and observability help build more reliable event-driven applications on AWS.

Imagine an order-processing pipeline built with serverless components.
An order is submitted. An event is published. A Lambda function processes the event and updates the order status.
Everything looks healthy.
Then the same event arrives again.
Perhaps a message was redelivered. Perhaps a client retried a request after a timeout. Perhaps the first attempt completed its database update, but the caller never received confirmation.
If the pipeline treats every delivery as a new business operation, the result could be a duplicate payment, an incorrect inventory adjustment, or an order processed twice.
The problem is not necessarily that a component failed. The problem is that the system did not define what should happen when an operation is repeated.
This is one of the most important reliability challenges in event-driven architecture.
A reliable pipeline should not depend on every event being delivered exactly once. It should be designed to handle repeated delivery safely.

1. Why duplicate events happen

Distributed systems involve multiple components communicating over networks. A request can succeed on one side while the response is delayed or lost on the other.
Consider this sequence:
  1. A Lambda function receives an order event.
  2. It updates the order record in DynamoDB.
  3. The function encounters a timeout before returning a successful response.
  4. The event source retries the operation.
  5. The function receives the same logical event again.
From the function's perspective, the second invocation may look like a new execution.
From the business perspective, however, it is the same order operation.
Other common causes of repeated delivery include:
  • retry policies after transient failures;
  • queue visibility timeouts expiring before processing finishes;
  • clients retrying requests after uncertain responses;
  • upstream services publishing duplicate events;
  • workflow recovery after partial failures.
Retries are valuable because they allow systems to recover from temporary problems. Eliminating retries entirely would make many distributed systems less resilient.
The engineering challenge is to make retries safe.

2. Understand idempotency

An operation is idempotent when repeating it produces the same intended final effect as performing it once.
For example, setting an order's status to CONFIRMED is naturally idempotent if the operation simply assigns that status.
Incrementing a counter is not naturally idempotent:
counter = counter + 1
Running that operation twice increases the counter twice.
The distinction becomes critical when processing events.
Suppose an event represents a payment of ₹1,000. If the handler adds ₹1,000 to a balance every time it runs, duplicate delivery can produce an incorrect result.
A safer design identifies the logical operation and records whether its effect has already been applied.
The goal is not necessarily to prevent a message from being delivered twice.
The goal is to prevent the same business operation from being applied twice.

3. Design a stable idempotency key

An idempotency key identifies one logical operation across retries.
For an order event, the key might be:
order-4821-payment-capture-v1
For an inventory adjustment, it might be derived from a unique adjustment identifier assigned by the upstream system.
A good key should be:
  • stable across retries;
  • unique to the intended business operation;
  • independent of the Lambda invocation identifier;
  • available before the side effect occurs.
Using a new random identifier for every invocation defeats the purpose because each retry appears to be a different operation.
Likewise, using only a customer ID is usually insufficient: one customer can legitimately perform many distinct operations.
The key should represent the unit of work whose effect must not be duplicated.

4. A practical AWS architecture

A serverless event-processing pipeline can use several AWS services, each with a specific responsibility.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
Producer
|
v
Amazon SQS
|
v
AWS Lambda
|
+----> DynamoDB idempotency state
|
+----> Business operation
|
v
Processing result

Repeated failures
|
v
Dead-letter queue
The exact architecture depends on the application. Amazon EventBridge can be useful for event routing, while Amazon SQS can provide a durable queue between producers and consumers.
Lambda performs the processing. DynamoDB can store idempotency records when its conditional-write and consistency properties fit the workload. A dead-letter queue can isolate messages that repeatedly fail processing.
The important part is the processing contract, not the number of AWS services involved.
Before implementing this architecture, define what constitutes a successful operation, what must be recorded, and how the system behaves when it cannot determine whether a side effect occurred.

5. Use durable state to detect repeated operations

A simplified idempotency record might contain:
1
2
3
4
5
{
"idempotency_key": "order-4821-payment-capture-v1",
"status": "COMPLETED",
"result_reference": "payment-7392"
}
When an event arrives, the handler checks the record associated with its idempotency key.
The basic logic is:
  1. Determine the logical operation's idempotency key.
  2. Attempt to claim the operation using an atomic conditional write.
  3. If another completed operation already exists, return or reuse its recorded result.
  4. If this invocation successfully claims the operation, perform the work.
  5. Persist the final result and operation status.
  6. Handle failures without incorrectly marking unfinished work as complete.
The conditional write matters because two copies of the same event can arrive concurrently. A simple read followed by a write is not enough: both invocations might read that no record exists and proceed simultaneously.
An atomic conditional write allows the system to establish which invocation owns the initial claim.
The record must be stored durably and shared by all workers processing the same logical operation. An in-memory set inside a Lambda execution environment is not sufficient because separate environments do not share that state reliably.

6. The difficult part: partial failure

Consider what happens when the handler claims an operation, calls an external payment service, and then crashes before recording completion.
The payment may have succeeded even though the idempotency record still says IN_PROGRESS.
If the retry blindly repeats the payment request, the system may charge the customer twice.
If it permanently refuses to retry, the operation may remain stuck even though recovery is possible.
This is why idempotency is more than storing a Boolean flag.
A production design should define states such as:
  • IN_PROGRESS
  • COMPLETED
  • FAILED_RETRYABLE
  • FAILED_FINAL
It should also define how stale in-progress records are recovered, how concurrent claims are handled, and how the result of an uncertain external operation is reconciled.
For payment operations, for example, the downstream provider's own idempotency mechanism may be essential. The application should reuse the same logical operation key across retries where the provider supports it.
When the external outcome is uncertain, querying the provider or reconciling its transaction records may be safer than immediately repeating the side effect.
A durable idempotency record prevents duplicate work only when its state transitions and external side effects are designed together.

7. Retries and dead-letter queues solve different problems

Retries are appropriate for failures that may resolve with time.
Examples include temporary throttling, network interruptions, or transient service unavailability.
A dead-letter queue is useful when a message cannot be processed successfully after the configured retry behavior.
These mechanisms should complement one another.
A practical retry policy should consider:
  • the type of failure;
  • the maximum number of attempts or processing duration;
  • backoff between attempts;
  • the risk of repeating side effects;
  • the age of the event;
  • the cost of delayed processing.
For Amazon SQS, configure visibility timeouts and redrive policies with the processing duration and retry behavior in mind. A visibility timeout that expires while the first worker is still processing can allow another worker to receive the same message.
A dead-letter queue helps isolate repeatedly failing messages, but it does not repair them automatically. Teams need an inspection and replay process that preserves the original operation identity and avoids reapplying completed effects.
Before replaying a message, determine whether the previous attempt partially succeeded.

8. Idempotency is not the same as exactly-once execution

It is tempting to describe a robust pipeline as an exactly-once system.
That description can hide important limitations.
A message may be delivered more than once. A Lambda handler may execute more than once. An external API may complete an operation even when the caller times out.
A system can still provide effectively-once business outcomes for a defined operation if it uses the appropriate combination of idempotency, transactional boundaries, durable state, and reconciliation.
However, a DynamoDB idempotency record alone does not create an atomic transaction spanning DynamoDB, a payment provider, and every other external service.
Each boundary needs its own reliability strategy.
For example:
  • A database update and its idempotency record may need to be committed atomically when they share a database transaction boundary.
  • An external API call may require a provider-supported idempotency key.
  • A multi-step business process may need a saga or explicit compensation logic.
  • An event published after a database update may require an outbox pattern to prevent the database change and event publication from becoming inconsistent.
The right guarantee should be stated precisely: which operation is protected, which side effects are covered, and what happens during partial failure.

9. Monitor business outcomes, not just Lambda errors

A pipeline can have low invocation error rates and still produce duplicate business effects.
For that reason, infrastructure monitoring should be complemented by application-level metrics.
Useful signals include:
  • duplicate events detected;
  • successful operations;
  • operations stuck in progress;
  • retries per event;
  • messages sent to the dead-letter queue;
  • oldest message age;
  • idempotency conflicts;
  • processing latency;
  • replay success rate.
These metrics help teams distinguish normal retry activity from a real reliability problem.
For example, an increase in duplicate detections may not mean that the system is broken. It may indicate that retries are working as intended.
However, a rising number of stale IN_PROGRESS records could point to timeouts, missing recovery logic, or failures between a side effect and its completion record.
The useful question is not simply, "How many retries happened?"
It is, "Did the system preserve the intended business outcome despite those retries?"

10. A practical implementation checklist

Before deploying an event-driven pipeline, I would verify the following:
Operation identity
  • Every logical operation has a stable idempotency key.
  • The key remains unchanged across retries and replays.
Concurrency
  • Initial claims use an atomic conditional operation.
  • Concurrent deliveries cannot both independently apply the same effect.
State and recovery
  • Processing states are explicit.
  • Stale in-progress operations can be investigated and recovered.
  • Completed operations return a consistent recorded result.
External side effects
  • Downstream APIs use idempotency support where available.
  • Uncertain outcomes can be reconciled.
  • The design does not assume that a database record makes unrelated external actions atomic.
Operational readiness
  • Retries and dead-letter handling are configured intentionally.
  • Duplicate detection and stale operations are observable.
  • Replay procedures preserve operation identity.
  • Tests cover timeouts, concurrency, partial failure, and duplicate delivery.
These checks are often more valuable than adding another layer of infrastructure to a pipeline whose failure semantics are not yet clearly defined.

Conclusion

Duplicate events are not an unusual edge case in distributed systems. They are a normal condition that reliable applications must handle.
A robust serverless pipeline starts with a clear definition of the logical operation, uses a stable idempotency key, protects concurrent claims, records state durably, and plans for partial failures.
AWS services such as SQS, Lambda, DynamoDB, and EventBridge provide useful building blocks, but the reliability guarantee comes from how those components are combined and how their failure boundaries are handled.
The key lesson is simple:
Do not design your pipeline around the assumption that every event arrives once. Design it so that receiving an event again does not repeat a business effect that has already succeeded.
That is what turns a functioning event-driven workflow into a dependable production system.
# aws-lambda# amazon-sqs# amazon-dynamodb# serverless# distributed-systems
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article