AWS Builder Center

Retries Are Not Enough: Building Reliable Asynchronous Systems on AWS

Explore how idempotency, retry strategies, dead-letter queues, and observability work together to make asynchronous systems more reliable, connecting backend engineering patterns with Amazon SQS and AWS cloud architecture

Backend Software Engineer | Node.js • TypeScript • APIs • Cloud Architecture
Moving work from a synchronous request to a queue is often one of the first steps toward building a more resilient and scalable backend.
But there is an important lesson that becomes clear when working with asynchronous systems:
Adding a queue does not automatically make a system reliable.
Queues help decouple components and absorb workload spikes, but they also introduce a different set of engineering challenges.
What happens when processing fails?
What happens when the same message is delivered twice?
What happens when retrying will never solve the problem?
And perhaps most importantly: how do we know when something is going wrong?
These questions become especially relevant when designing asynchronous workloads with services such as Amazon SQS.
---
The Happy Path Is the Easy Part
Consider a simple asynchronous architecture:
API → Amazon SQS → Consumer → External Service
The API receives a request and publishes a message to the queue.
A consumer retrieves the message, processes it, communicates with another service if necessary, and completes the operation.
When everything works, the architecture looks simple.
Production systems, however, are defined as much by their failure paths as by their successful ones.
The external service might be temporarily unavailable.
The consumer might crash while processing a message.
A network timeout might occur after an operation succeeds but before the consumer receives confirmation.
The same message might then be processed again.
This is where reliability becomes an architectural concern rather than simply an infrastructure choice.
---
“Just Retry” Is Not a Reliability Strategy
Retries are extremely useful for transient failures.
If an external API is temporarily unavailable, trying again after a short delay may solve the problem.
But retries need boundaries.
Imagine a consumer receives a message containing invalid data.
Processing fails.
The message becomes available again.
The consumer processes it again.
It fails again.
Without an appropriate strategy, the system can spend resources repeatedly processing something that will never succeed.
This is sometimes referred to as a poison message.
A reliable system therefore needs to distinguish between failures that may recover and failures that require investigation.
Retry policies, backoff strategies, and maximum attempt limits become part of the system design.
But another problem remains.
What happens if processing actually succeeds twice?
---
Idempotency Changes the Way We Design Consumers
Distributed systems often force us to accept an uncomfortable possibility:
The same operation may be attempted more than once.
This means consumers should ideally be designed so that processing the same message multiple times does not create unintended side effects.
This property is known as idempotency.
Consider an operation that sends an email.
Processing the same job twice could send two emails.
For a notification, that might only be annoying.
Now replace the email with a payment, inventory update, or account operation.
Duplicate processing becomes significantly more serious.
A common strategy is to associate each operation with a unique identifier and persist information about processed operations.
Conceptually:
Message arrives → Check operation ID → Process if new → Record result
If the same operation appears again, the consumer can recognize that it has already been processed and avoid repeating the side effect.
The exact implementation depends on the workload, but the principle is important:
Retries and idempotency should usually be designed together.
Retries allow the system to recover.
Idempotency makes retrying safer.
---
When Retries Stop Helping
Eventually, a system needs to decide:
This message should not be retried automatically anymore.
This is where a Dead-Letter Queue (DLQ) becomes useful.
With Amazon SQS, a queue can be configured with a redrive policy that moves messages to another queue after they have been received a specified number of times without successful deletion.
The architecture becomes:
API → SQS → Consumer
and, after repeated failures:
SQS → Dead-Letter Queue
Instead of allowing a problematic message to circulate indefinitely, the system isolates it for analysis or later recovery.
This creates an important operational boundary.
The main queue continues processing healthy workloads while failed messages become visible as a separate problem.
But simply creating a DLQ is not enough either.
A DLQ that nobody monitors is effectively a storage location for forgotten failures.
---
Reliability Requires Observability
This is where asynchronous architecture connects directly with observability.
If messages begin accumulating in a queue, how quickly will the team notice?
If messages start reaching the DLQ, who gets notified?
How long are messages waiting before being processed?
How often are consumers failing?
AWS provides services such as Amazon CloudWatch that can help monitor these signals through metrics, logs, dashboards, and alarms.
For an SQS-based workload, useful signals can include:
- Queue depth - Age of the oldest message - Messages being processed - Messages reaching the DLQ - Consumer errors - Processing latency
These metrics transform reliability from something we assume into something we can observe.
This leads to one of the most important lessons I’ve learned from backend systems:
A failure you cannot observe is a failure you cannot reliably operate.
---
Reliability Is a System Property
It is tempting to look for a single technology that makes an architecture reliable.
A queue.
A retry mechanism.
A DLQ.
Monitoring.
But reliability does not come from any one of these components.
It comes from how they work together.
A more complete asynchronous flow might look like:
API ↓ Amazon SQS ↓ Consumer / AWS Lambda ↓ Idempotency Check ↓ Business Operation
When processing fails:
Retry with appropriate policy ↓ Maximum attempts reached ↓ Dead-Letter Queue ↓ CloudWatch monitoring and alerting ↓ Investigation / recovery
Each layer addresses a different failure mode.
The queue provides decoupling.
Retries handle transient failures.
Idempotency protects against duplicate processing.
The DLQ isolates persistent failures.
Observability tells us when the system needs attention.
None of these mechanisms replaces the others.
---
From Backend Patterns to AWS Architecture
One thing I find valuable while learning AWS is recognizing that many Cloud concepts are extensions of engineering problems backend developers already encounter.
If you have worked with tools such as BullMQ or RabbitMQ, concepts like consumers, retries, failed jobs, concurrency, and message processing are already familiar.
Amazon SQS changes the operational model, but many of the fundamental questions remain:
How should failures be handled?
Can this operation safely execute twice?
When should we stop retrying?
How do we recover failed work?
How do we know the system is healthy?
Understanding those questions makes learning the AWS service itself much easier.
Instead of starting with:
“What features does Amazon SQS have?”
I prefer starting with:
“What failure modes does my system have, and which AWS capabilities help me handle them?”
That shift turns Cloud learning from memorizing services into understanding architecture.
---
Final Thoughts
Building an asynchronous system is relatively easy.
Building one that behaves predictably when things go wrong is much harder.
Queues are an important building block, but reliability comes from designing the entire failure lifecycle.
Retries help systems recover.
Idempotency makes retries safer.
Dead-letter queues isolate failures.
Observability makes those failures actionable.
And together, these patterns help transform a simple queue-based workflow into a system that can be operated with confidence.
As I continue deepening my AWS knowledge, this is one of the connections I find most useful: Cloud architecture becomes much easier to understand when we connect AWS services to engineering problems we have already encountered.
For those building asynchronous workloads on AWS: what reliability pattern has saved you the most trouble in production?
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article