From Endless Retries to Controlled Recovery: Improving an Event-Driven AWS Architecture
A Dead Letter Queue is a good start, but it doesn't solve every failure. This article explores how failure isolation, CloudWatch monitoring, SNS alerts, controlled message replay, and idempotency can make an event-driven AWS architecture easier to operate and recover when things go wrong.
From Endless Retries to Controlled Recovery:
In the previous article, I looked at an asynchronous AWS architecture where the happy path was fairly straightforward.
A request came in through API Gateway, Lambda pushed an event into SQS, and another Lambda processed it.
The interesting part started when processing failed.
Without a failure isolation mechanism, the same message could keep coming back after its visibility timeout expired. If the underlying problem wasn't temporary, retrying it wasn't going to make the situation better. It was just making the system do the same thing repeatedly.
So the next question was fairly obvious:
What should happen to a message that has already failed several times?
That's where the second version of the architecture becomes more interesting.
The main request flow didn't need to be completely redesigned. Instead, I added failure handling around the existing flow.
Architecture overview
Figure 1: The improved event-driven architecture with a Dead Letter Queue, CloudWatch monitoring, SNS notifications, and a controlled replay path.
The normal flow is still:
Users → API Gateway → Producer Lambda → SQS → Consumer Lambda
The consumer then interacts with downstream systems, including Aurora and external services such as payment and email providers.
The important change happens when the consumer can't successfully process a message.
After the message reaches the configured
maxReceiveCount, SQS moves it to the Dead Letter Queue (DLQ) instead of continuing to deliver it through the normal processing path.That sounds like a small change.
Operationally, it isn't.
Failure shouldn't keep blocking healthy work
In the earlier design, a failed message stayed in the main queue and kept coming back.
That creates an awkward situation.
Suppose there are 10,000 legitimate events and one event contains invalid data that will never succeed.
The consumer doesn't know that the message is permanently broken. As far as the consumer is concerned, it's another message that needs processing.
So it gets retried.
Then retried again.
Meanwhile, valid events are still waiting to be processed.
A DLQ gives us a way to separate those two cases.
Messages that are still worth processing remain on the main path. Messages that have repeatedly failed are moved somewhere else for investigation.
That's the part I find most useful about this pattern. It's not really about adding another queue. It's about separating normal traffic from known failures.
Once failed messages are isolated, the rest of the workload has a much better chance of continuing normally.
A DLQ is necessary, but not sufficient
This is also where I think it's easy to oversell the DLQ.
Adding one doesn't magically make the architecture resilient.
It just changes what happens after repeated failures.
The message is still broken.
Someone still has to figure out why.
If the DLQ contains five messages, that's probably manageable.
If it contains 50,000 messages and nobody is watching it, we've just moved the problem somewhere else.
So the DLQ needs to become part of the operational workflow.
That means monitoring it, alerting when it grows unexpectedly, investigating the underlying failures, and deciding what should happen to those messages afterward.
In other words, a DLQ is a failure isolation mechanism, not a failure resolution mechanism.
That distinction matters.
What should we monitor?
The updated architecture adds CloudWatch monitoring around the DLQ.
Two useful SQS metrics to start with are:
ApproximateNumberOfMessagesVisibleApproximateAgeOfOldestMessage
The number of messages tells us whether failures are accumulating.
The age of the oldest message tells us something different: how long messages have been waiting for attention.
Imagine the DLQ contains only 10 messages, but the oldest one has been sitting there for three days.
That tells a very different story from a DLQ that currently contains 100 messages but is being actively investigated and cleared.
This is why I wouldn't treat a DLQ message count of zero as the only measure of success.
The operational question is:
Are failures being detected and handled within an acceptable amount of time?
CloudWatch can provide the signal, and Amazon SNS can notify the team when that signal crosses a configured threshold.
It's a relatively small addition, but it changes the architecture from something that merely processes messages into something that can also alert us when processing is going wrong.
The replay path is where recovery gets interesting
The other addition to the architecture is the Replay Lambda.
This solves a different problem.
Let's say the reason for the failures was a temporary issue with an external dependency.
The dependency recovers.
The application has been fixed.
Now there are hundreds of messages sitting in the DLQ that may be perfectly valid.
We don't want to throw them away.
We also don't necessarily want an engineer manually copying messages around to get them processed again.
The Replay Lambda provides a controlled path to put those messages back into the main queue.
The recovery flow becomes something like:
Failure → DLQ → Investigation → Fix → Replay → Main Queue
That's a much better operational story than:
Failure → Retry → Retry → Retry → Hope
But there's an important catch.
Replay isn't automatically safe.
Replaying a message can create another problem
Consider the payment example from the first article.
A payment request reaches the external payment provider.
The payment succeeds.
But the Lambda doesn't receive the response correctly, perhaps because of a timeout or network failure.
From the application's point of view, the processing attempt failed.
The message eventually ends up in the DLQ.
Later, someone replays it.
The payment provider receives the request again.
Now we have a potential duplicate charge.
This is why I wouldn't describe the Replay Lambda as simply a retry mechanism.
It's a recovery mechanism, and recovery needs to account for what may already have happened during the original attempt.
This is where idempotency becomes important.
The consumer needs some way of recognizing that a particular business operation has already been completed, even if the original processing attempt didn't finish cleanly.
For payment-related workflows, this is especially important.
The same thinking applies to other side effects, too.
What if the email was already sent?
What if the database update succeeded but the next operation failed?
What if only half of the business workflow completed?
A replay mechanism without those considerations can turn an operational recovery process into another source of duplicate actions.
Observability needs context
Adding CloudWatch is useful, but there's another thing I'd want in a production implementation: a way to trace a business request through the system.
A message count can tell us that something is wrong.
It doesn't necessarily tell us what went wrong.
Suppose an operator gets an alert saying the DLQ has started growing.
The next questions are probably:
- Which requests are failing?
- Which customer or order do they belong to?
- How many times did each message fail?
- Which downstream dependency was involved?
- Is the same error affecting all these messages?
This is where correlation IDs become useful.
If the same identifier can be followed from the API request through Lambda, SQS, the database, and external service calls, troubleshooting becomes much less painful.
Without that context, an incident can quickly turn into someone searching through a large pile of CloudWatch logs to find the request that caused the problem.
Nobody wants that job.
The main queue can now stay focused on its actual job
There's another benefit to the updated design that's easy to overlook.
The main queue is no longer responsible for carrying repeatedly failing messages indefinitely.
That's important during an incident.
Imagine the payment provider is unavailable for 20 minutes.
A large number of payment-related events start failing.
With the DLQ pattern, messages eventually move out of the normal processing path after reaching the configured receive count.
That gives the system a way to isolate repeated failures rather than continuously processing them through the same path.
The main queue can continue handling messages that are actually processable, subject to available concurrency and downstream capacity.
The failed messages are still there.
They're not automatically lost.
They're just no longer competing with healthy traffic in the same processing path.
This is a much more controlled failure mode.
But there is still a cost
It's tempting to look at the improved diagram and conclude that the additional components are basically free insurance.
They're not.
Now we're operating:
- A DLQ
- Additional Lambda logic for replay
- CloudWatch metrics and alarms
- SNS notifications
- More operational workflows
There can also be costs associated with the original failed processing attempts, logging, database connections, and calls to external services.
The goal isn't to eliminate every one of those costs.
It's to avoid repeatedly paying for work that has no chance of succeeding while making actual failures easier to investigate and recover from.
That's an important distinction.
A resilient architecture isn't necessarily the architecture with the fewest components.
Sometimes a few extra components are what prevent a much larger operational problem later.
What I would still question
Even with this improved architecture, I wouldn't call the design finished.
I'd still want to answer a few questions before considering it production-ready.
What is the right
maxReceiveCount?How long should messages remain in the DLQ?
Who is allowed to trigger a replay?
Should every failed message be replayed, or should replay happen in controlled batches?
How do we prevent a replay from immediately filling the DLQ again?
How do we handle messages that repeatedly fail even after replay?
And most importantly, how do we make sure business operations are idempotent?
These aren't really SQS questions anymore.
They're application and operational design questions.
That's an important distinction when designing event-driven systems.
What changed?
Looking at the two architectures side by side, the AWS services aren't radically different.
The main request path is still there.
SQS is still buffering work.
Lambda is still processing it.
The database and external services are still downstream.
The difference is what happens when the happy path stops working.
In the first design, failure stayed in the main processing path.
In the improved design, failure gets isolated, surfaced to operators, investigated, and—when appropriate—replayed.
That's a much healthier way to think about asynchronous systems.
The goal isn't to pretend failures won't happen.
They're going to happen.
The goal is to make sure one bad message, one unavailable dependency, or one unexpected timeout doesn't turn into an incident that keeps feeding itself.
And that's probably the biggest lesson I took away from comparing these two architectures:
Resilience isn't about making the happy path better. It's about making the failure path predictable.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article