AWS Builder Center

Builder Center

The Dual-Write Dilemma: Engineering Reliable Event Delivery in Distributed AWS Architectures

The Dual-Write Dilemma: Engineering Reliable Event Delivery in Distributed AWS Architectures

In distributed systems, successfully writing data to a database does not guarantee that downstream services will receive the corresponding event. This article explores the dual-write problem, transactional outbox patterns, change data capture, idempotency, and failure recovery, with practical AWS architectures for building reliable event-driven microservices.

The Dual-Write Dilemma: Engineering Reliable Event Delivery in Distributed AWS Architectures

1. Introduction: The Reliability Problem Nobody Sees

Modern distributed applications are rarely built as a single executable system. They consist of independently deployed microservices, separate databases, asynchronous communication channels, and event-driven workflows.
Consider a typical e-commerce platform.
When a customer places an order, the Order Service must:
  1. Persist the order in its database.
  1. Publish an  ORDER_CREATED  event.
  1. Notify the Inventory Service.
  1. Trigger the Payment Service.
  1. Inform the Notification Service.
At first glance, this seems like a straightforward sequence of operations.
However, there is a subtle reliability problem hidden inside this workflow.
What happens when the database successfully commits the order, but the application crashes before publishing the event?
The order exists in the database, but the downstream services never receive the notification.
Now consider the opposite situation. The application publishes the event successfully, but the database transaction fails.
The Inventory Service might reserve stock for an order that does not actually exist.
Both scenarios create inconsistencies between independently operating systems.
This problem is known as the Dual-Write Problem.
It is one of the fundamental challenges in distributed systems because a conventional database transaction cannot automatically guarantee atomicity across an independent database and an external message broker.
The objective of this article is to examine this problem, understand its failure scenarios, and design a reliable event-delivery architecture using AWS services.


2. Understanding the Dual-Write Problem

A dual-write operation occurs when an application must update two independent systems as part of one business operation.
For example:
Plain text
def create_order(order):
    database.save(order)
    event_bus.publish({
        "event_type": "ORDER_CREATED",
        "order_id": order["id"]
    })
The code appears correct, but the two operations do not share a common atomic transaction.
There are two possible execution paths.

Scenario A: Database success, event publishing failure

Plain text
Application
    |
    v
Database Write
    |
    v
Transaction Committed
    |
    v
Application Crashes
    |
    X
Event Not Published
The database contains the order, but downstream services remain unaware of it.
This creates an event-loss scenario.

Scenario B: Event publishing success, database failure

Plain text
Application
    |
    v
Event Published
    |
    v
Database Write
    |
    X
Transaction Failed
The downstream service receives an event describing a business operation that never successfully committed.
This can cause invalid inventory reservations, incorrect payment attempts, and inconsistent application state.

Why retries alone are insufficient

A common solution is to retry failed operations.
However, retries introduce another problem.
Suppose the event broker accepts a message, but the application times out before receiving the acknowledgement.
The application cannot reliably determine whether the message was published.
Retrying might publish the same event again.
Therefore, a reliable architecture must address two different concerns:
  • Preventing committed database changes from losing their corresponding events.
  • Preventing repeated event delivery from causing duplicate business operations.
This leads us to the transactional outbox pattern.


3. Transactional Outbox: Connecting Database Consistency with Event Delivery

The transactional outbox pattern introduces an intermediate storage mechanism that records events as part of the same database transaction as the original business operation.
Instead of independently writing to a database and publishing to a message broker, the application performs two database writes within one atomic transaction:
  1. Save the business entity.
  1. Save the corresponding event in an outbox record.
A separate publishing component reads the outbox and forwards events to downstream systems.
The important distinction is that the original business transaction no longer depends on the availability of the external message broker.

The revised execution flow

Plain text
              Client Request
                    |
                    v
                Order API
                    |
                    v
             Order Service
                    |
                    v
          Database Transaction
              /          \
             v            v
        Order Record   Outbox Record
             \            /
              \          /
               Transaction
                 Commit
                    |
                    v
             Event Publisher
                    |
                    v
              Message Broker
                    |
          +---------+---------+
          |         |         |
          v         v         v
       Inventory  Payment  Notification
        Service   Service    Service
If the database transaction fails, neither the order nor its outbox record is committed.
If the transaction succeeds, both records are persisted, even if the message broker becomes temporarily unavailable.
The publisher can retry delivery later.
This transforms the problem from coordinating two independent writes into coordinating one atomic database transaction followed by asynchronous event delivery.
However, the outbox pattern does not eliminate every failure. Publishers can still crash after sending an event but before recording successful delivery. Therefore, duplicate delivery must still be expected.


4. Designing the Architecture with AWS Services

There are multiple ways to implement transactional outbox architectures on AWS.
For this discussion, consider a DynamoDB-based implementation.

Proposed technology stack

AWS Service
Responsibility
Amazon API Gateway
Receives client requests
AWS Lambda
Executes business logic
Amazon DynamoDB
Stores business records and outbox records
DynamoDB Streams
Captures database changes
Amazon EventBridge
Routes domain events
Amazon SQS
Buffers downstream messages
AWS Lambda
Processes individual events
Amazon CloudWatch
Monitors system behaviour
AWS IAM
Controls service permissions

Step 1: Atomic database writes

Amazon DynamoDB provides transactional operations through  TransactWriteItems .
This API supports multiple write operations within a single all-or-nothing transaction.
The Order Service can use it to create both an order record and its corresponding outbox record.
Conceptual example:
Plain text
import boto3
import json
import uuid
from datetime import datetime, timezone

dynamodb = boto3.client("dynamodb")

def create_order(order_id, customer_id, amount):

    event_id = str(uuid.uuid4())

    order = {
        "PK": {"S": f"ORDER#{order_id}"},
        "SK": {"S": "METADATA"},
        "customer_id": {"S": customer_id},
        "amount": {"N": str(amount)},
        "status": {"S": "CREATED"}
    }

    outbox = {
        "PK": {"S": f"ORDER#{order_id}"},
        "SK": {"S": f"OUTBOX#{event_id}"},
        "event_id": {"S": event_id},
        "event_type": {"S": "ORDER_CREATED"},
        "created_at": {
            "S": datetime.now(timezone.utc).isoformat()
        },
        "payload": {
            "S": json.dumps({
                "order_id": order_id,
                "customer_id": customer_id,
                "amount": amount
            })
        }
    }

    dynamodb.transact_write_items(
        TransactItems=[
            {
                "Put": {
                    "TableName": "Orders",
                    "Item": order,
                    "ConditionExpression":
                        "attribute_not_exists(PK)"
                }
            },
            {
                "Put": {
                    "TableName": "Orders",
                    "Item": outbox,
                    "ConditionExpression":
                        "attribute_not_exists(PK)"
                }
            }
        ]
    )

    return event_id
Both writes succeed together or fail together.
The outbox record contains an immutable event identifier, event type, timestamp, and payload.
For a production implementation, the schema should also include aggregate versions, appropriate access patterns, and event retention metadata.
DynamoDB transactions have documented limits, including up to 100 actions and a 4 MB aggregate transaction size.

Step 2: Change Data Capture using DynamoDB Streams

Writing an outbox record is only the first stage.
The next challenge is detecting committed changes and forwarding them without requiring the application to publish events synchronously.
DynamoDB Streams captures item-level modifications to a DynamoDB table.
It maintains a time-ordered sequence of changes and allows downstream consumers to process those changes.
An AWS Lambda function can be configured as an event source consumer.
The processing flow becomes:
Plain text
DynamoDB Transaction
        |
        v
   Outbox Record
        |
        v
  DynamoDB Streams
        |
        v
  Lambda Publisher
        |
        v
  EventBridge Bus
The Lambda publisher should:
  • Identify records representing outbox events.
  • Extract the event identifier and payload.
  • Validate the event schema.
  • Publish the event to EventBridge.
  • Handle unsuccessful publishing attempts.
  • Preserve sufficient metadata for tracing and recovery.
An important implementation detail is that a DynamoDB transaction's stream records must not be assumed to arrive as one atomic batch or in transaction-wide order.
AWS documentation explicitly describes gradual propagation of transaction changes to streams and warns consumers not to assume transaction atomicity or ordering across those records.
This means the architecture should use explicit event identifiers and aggregate versions instead of depending on stream arrival order.

Step 3: Event routing using Amazon EventBridge

Once the event is published, Amazon EventBridge can route it to multiple independent consumers.
For example:
Plain text
{
  "source": "commerce.orders",
  "detail-type": "ORDER_CREATED",
  "detail": {
    "event_id": "evt-12345",
    "order_id": "ORD-9001",
    "customer_id": "CUST-201",
    "aggregate_version": 1
  }
}
EventBridge rules can route this event to different targets based on its source, type, and payload.
The Inventory Service does not need to know how the Order Service stores its data.
Similarly, the Payment Service does not need a direct connection to the Inventory Service.
This separation reduces service dependencies and allows individual components to evolve independently.

Step 4: Buffering with Amazon SQS

Instead of coupling event delivery directly to business processing, introduce Amazon SQS between the event-routing layer and downstream consumers.
Plain text
              EventBridge
                   |
          +--------+--------+
          |        |        |
          v        v        v
        SQS      SQS      SQS
          |        |        |
          v        v        v
      Inventory Payment Notification
       Lambda   Lambda     Lambda
This architecture provides several advantages:
Workload isolation: A sudden increase in order traffic does not require every downstream service to process events immediately.
Independent scaling: Each consumer can scale according to its own processing capacity.
Failure isolation: A temporary failure in one consumer does not necessarily prevent other consumers from receiving their events.
Asynchronous recovery: Messages can remain queued while downstream services recover.
For workloads requiring FIFO processing, SQS FIFO queues provide message ordering within message groups and deduplication features. These capabilities should not be confused with universal exactly-once business processing.


5. The Second Challenge: Idempotency and Duplicate Events

Solving the dual-write problem does not mean every event will be delivered exactly once.
Consider this scenario:
  1. The publisher sends an event successfully.
  1. The message broker accepts it.
  1. The publisher crashes before recording successful delivery.
  1. The publisher retries after recovery.
  1. The same event is published again.
This is an expected failure mode in distributed event processing.
AWS Lambda event source mappings can also process records more than once, so consumers must be designed to handle duplicate invocations.
The solution is idempotent event processing.
An operation is idempotent when executing it multiple times produces the same intended business outcome as executing it once.

Implementing idempotency

Every event should contain a unique identifier.
For example:
Plain text
{
  "event_id": "evt-12345",
  "event_type": "ORDER_CREATED",
  "order_id": "ORD-9001"
}
The consumer maintains a record of processed event identifiers.
Before performing a business operation, it checks whether the event has already been processed.
However, a simple check followed by a separate write is not sufficient because two concurrent invocations could both pass the check.
A stronger implementation uses an atomic database transaction.
Conceptually:
Plain text
Receive Event
     |
     v
Check Event ID
     |
     v
Already Processed?
    /       \
  Yes        No
   |          |
   v          v
 Ignore    Atomic Transaction
             /       \
            v         v
       Record ID   Business Update
            \         /
             Transaction
                |
                v
          Commit Success
For example, the consumer can use DynamoDB  TransactWriteItems  to:
  • Insert an event-processing record with a conditional expression that prevents duplicate insertion.
  • Update the corresponding business record in the same transaction.
If another invocation attempts to process the same event, the conditional write prevents it from applying the business operation again.
This is particularly important for operations involving financial transactions, inventory reservations, and customer account updates.
The deduplication record must be retained for an appropriate period, considering the maximum event replay and recovery window.


6. Failure Analysis: What Happens When Components Fail?

A production architecture should not only describe the successful execution path. It should also explain how the system behaves when individual components become unavailable.
Failure scenario
Potential impact
Recovery strategy
Order Service crashes before commit
No order created
Client retries safely
Database transaction fails
No order or outbox record
Retry after resolving failure
Publisher Lambda crashes
Event may remain unprocessed
Stream retry and recovery configuration
EventBridge publishing fails
Event delivery is incomplete
Retry failed entries and monitor failures
Consumer receives duplicate event
Duplicate business operation
Atomic idempotency checks
Downstream service unavailable
Queue backlog increases
SQS buffering and controlled recovery
Invalid event payload
Repeated processing failures
DLQ and controlled redrive

Dead-letter queues

A message that repeatedly fails processing should not block the entire system indefinitely.
Amazon SQS dead-letter queues provide a mechanism for isolating messages that exceed their configured receive threshold.
This makes it possible to:
  • Investigate malformed messages.
  • Identify recurring consumer failures.
  • Correct processing logic.
  • Redrive messages after the underlying issue is resolved.
However, DLQs should be monitored. Moving a message into a DLQ does not mean the business operation has succeeded.

Event replay and reconciliation

Another important recovery mechanism is event replay.
Suppose a downstream consumer was unavailable for an extended period.
The system should have a defined process for recovering missed work, validating the current business state, and safely reprocessing eligible events.
This is why retaining event identifiers, timestamps, aggregate versions, and processing status is valuable.
For critical workflows, periodic reconciliation between the source database and downstream projections can help detect discrepancies that ordinary retries might not resolve.


7. Comparing Relational Outbox and DynamoDB-Based CDC Architectures

The transactional outbox pattern is not restricted to NoSQL databases.
A relational database can implement the same principle using an outbox table within the existing business transaction.
Consideration
Relational Outbox
DynamoDB CDC
Database
Amazon RDS / Aurora
Amazon DynamoDB
Event storage
Dedicated outbox table
Outbox items or event attributes
Atomicity
Database transaction
DynamoDB transaction
Change detection
Polling or CDC
DynamoDB Streams
Publishing
Publisher service
Lambda stream consumer
Data model
Relational
Key-value/document
Best suited for
Relational business workloads
DynamoDB-native applications

Relational database approach

An application can write the business record and outbox event into separate relational tables within one SQL transaction.
A separate publisher reads committed outbox records and forwards them to Amazon SQS or EventBridge.
Change Data Capture can also be used to capture committed database changes.

DynamoDB approach

DynamoDB supports atomic multi-item writes through  TransactWriteItems .
DynamoDB Streams captures item-level modifications, allowing a Lambda consumer to forward relevant events.

Which approach should be selected?

The decision should depend on the existing application architecture, transaction requirements, data model, throughput characteristics, and operational constraints.
A relational database is often a natural choice when the business domain depends on relational transactions and complex queries.
DynamoDB is attractive when the application already uses its access patterns and requires a highly scalable managed key-value/document database.
Neither approach is universally superior.
The objective is to preserve atomicity between the business update and event creation while designing reliable asynchronous delivery.


8. Performance, Scalability and Operational Considerations

Reliability is only one dimension of a production distributed system.
The architecture must also be evaluated under realistic workloads.

Throughput and batching

Processing every stream record individually can introduce unnecessary invocation overhead.
Lambda event source mappings support batching, allowing multiple records to be processed in one invocation.
However, larger batches can increase processing latency and complicate failure handling.
Partial batch response support should be configured so successfully processed records are not unnecessarily retried when only some records fail.

Monitoring event-processing latency

Amazon CloudWatch can be used to monitor:
  • Lambda invocation errors.
  • Lambda duration.
  • DynamoDB throttling.
  • SQS queue depth.
  • SQS message age.
  • Dead-letter queue message count.
  • EventBridge publishing failures.
  • DynamoDB Streams iterator age.
A useful operational metric is event propagation latency.
It measures the time between the original business transaction and successful downstream processing.
For example:
Plain text
Event Propagation Latency =
Consumer Processing Timestamp
-
Outbox Creation Timestamp
Monitoring this metric helps identify situations where events are technically being delivered but are arriving too late for business requirements.

Backpressure and controlled concurrency

A system should not increase consumer concurrency without considering downstream database capacity.
If a sudden workload increase causes hundreds of Lambda invocations to execute simultaneously, the database may become the bottleneck.
Reserved concurrency, queue buffering, batch sizing, and controlled consumer scaling can help prevent downstream overload.
The goal is not simply to maximize the number of processed events. It is to maintain sustainable processing throughput without destabilizing dependent services.


9. Security and Governance

Event-driven systems introduce additional security considerations because business information moves across multiple services.
A production implementation should consider:
Least-privilege IAM: Each Lambda function should receive only the permissions required for its specific task.
Encryption: Enable encryption for DynamoDB, SQS, and other supported storage and messaging resources. Use AWS Key Management Service where appropriate.
Event payload minimization: Publish only the information required by downstream consumers rather than exposing complete database records.
Network isolation: Use appropriate VPC configurations and private connectivity for services that access sensitive backend resources.
Auditability: Record event identifiers, processing outcomes, failure details, and relevant correlation identifiers.
Data retention: Define suitable retention and deletion policies for outbox records, processed-event records, and dead-letter queues.
Security should be designed into the architecture rather than added after the event-processing pipeline is operational.


10. Engineering Trade-offs: What the Transactional Outbox Does Not Solve

Although the transactional outbox pattern significantly improves event-delivery reliability, it is not a universal solution to distributed consistency.
Several limitations must be considered.
1. Duplicate delivery remains possible.
The publisher can fail after sending an event but before recording its successful delivery. Consumers must remain idempotent.
2. Event ordering requires explicit design.
Multiple events can be processed concurrently or observed at different times. Aggregate versions and sequence numbers can help consumers identify ordering gaps and stale events.
3. Outbox storage introduces operational overhead.
The system needs retention policies, cleanup processes, monitoring, and potentially archival mechanisms.
4. Eventual consistency is still present.
Downstream services do not necessarily reflect the source database immediately after a transaction commits.
5. Cross-service transactions are not automatically atomic.
The outbox solves the atomicity problem between a local database transaction and event creation. If a workflow requires coordinated business operations across multiple services, a Saga pattern may be necessary.
Understanding these limitations is essential when designing production-grade distributed systems.


11. Conclusion: Reliability Is an Architectural Property

The dual-write problem demonstrates an important principle of distributed systems engineering: the successful execution of individual components does not guarantee the consistency of the overall application.
A database can commit successfully while a message broker remains unaware of the change. A message can be delivered multiple times even when the original business operation occurred only once. Independent services can remain temporarily inconsistent despite functioning correctly.
The transactional outbox pattern provides a structured solution by making event creation part of the original database transaction and separating event publication into an asynchronous process.
When combined with DynamoDB transactions, DynamoDB Streams, Lambda, EventBridge, SQS, idempotent consumers, and appropriate recovery mechanisms, it provides a practical foundation for building more reliable event-driven applications on AWS.
The most important engineering lesson is not that failures can be eliminated.
It is that distributed architectures should be designed with the expectation that failures, retries, duplicate deliveries, and temporary inconsistencies will occur.
A reliable system is one that can detect these conditions, recover safely, and preserve its business invariants even when individual components fail.
Reliability is not achieved by assuming that every operation succeeds. It is achieved by designing every operation to remain safe when it does not.


References and Further Reading

  1.  AWS Prescriptive Guidance – Transactional Outbox Pattern 
  1.  Amazon DynamoDB – Transactions: How It Works 
  1.  AWS Lambda – Using AWS Lambda with Amazon DynamoDB 
  1.  Amazon DynamoDB – Best Practices for Using DynamoDB Streams with Lambda 
  1.  AWS Samples – Transactional Outbox Pattern 
Architecture note: The implementation examples in this article illustrate the proposed design. Performance, throughput, latency, and cost should be validated through workload-specific testing before production deployment.
# architecture# serverless# aws-serverless-application-repository# distributed-systems# event-driven
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article