Your AWS Application Is Failing. Why Retrying Could Make It Worse
Retries make cloud applications more resilient, but poorly designed retry logic can turn a small failure into a system-wide outage.
Imagine you are running an e-commerce application on AWS. Everything is working normally until a sudden spike in traffic slows down your payment service.
Your application sends a request. It times out.
What should it do?
Most developers would retry the request. That sounds reasonable. The payment service might simply be experiencing a temporary issue.
But imagine that 10,000 users are doing the same thing. Each request times out, and every application immediately sends another request. The payment service, already struggling to keep up, receives another wave of traffic.
It slows down further. More requests time out. More retries follow.
A temporary slowdown has now become a much bigger incident.
This is the retry storm problem. A mechanism designed to improve reliability can actually amplify an outage when it is implemented without the right safeguards.
The interesting part is that you do not necessarily need a more powerful server or a bigger database to solve this problem. Sometimes, you need to change how your application responds when something goes wrong.
Let's explore how to do that using AWS.
1. Why retries can create a vicious cycle
Consider a service that can comfortably handle 1,000 requests per second. A sudden traffic spike pushes it close to capacity, and response times increase.
Now suppose requests start timing out, and each failed request triggers two additional attempts.
The following is a simplified illustration of what can happen when retries are applied aggressively:
| Stage | Incoming attempts |
|---|---|
| Original requests | 1,000 |
| First retry wave | 2,000 |
| Second retry wave | 4,000 |
| Third retry wave | 8,000 |
These numbers are illustrative, not a prediction of every retry system. They show how quickly traffic can multiply when retries trigger further retries.
The underlying problem is that the application is treating every failure as an isolated event. It is not accounting for the possibility that the dependency is already overloaded.
This creates three risks:
- Resource exhaustion: More requests consume CPU, memory, database connections, and network capacity.
- Increased latency: Requests spend more time waiting and retrying, tying up resources for longer.
- Cascading failures: One struggling dependency can cause other services that rely on it to become unhealthy as well.
AWS discusses these risks in its Well-Architected guidance on controlling and limiting retries.
The lesson is simple: retries should help a service recover, not force it to handle more work when it can barely handle the work it already has.
2. The first line of defence: Exponential backoff
Instead of retrying immediately, your application can wait before trying again.
A common approach is exponential backoff, where the delay increases after each failed attempt.
For example, an application might wait:
- First retry: 1 second
- Second retry: 2 seconds
- Third retry: 4 seconds
- Fourth retry: 8 seconds
The precise values depend on the application and its latency requirements. The important principle is that repeated failures should generally result in progressively longer waits.
This gives the downstream service time to recover instead of continuously sending it new requests.
What this looks like on AWS
When your application calls AWS services through a supported AWS SDK, retry behaviour is often built in. You can configure the retry mode and maximum number of attempts instead of implementing everything yourself.
For example, a shell environment can use the following configuration for AWS SDKs that support these shared settings:
1
2
export AWS_RETRY_MODE=standard
export AWS_MAX_ATTEMPTS=3Here,
standard selects the standard retry mode, while 3 specifies three total attempts, including the initial request, for SDKs using this configuration.These settings are a starting point, not a universal production prescription. Check the behaviour of your specific SDK version and test it against your workload.
Standard retry behaviour includes backoff and jitter, helping reduce the impact of transient errors and throttling.
Official reference: AWS SDK retry behaviour
3. Why exponential backoff alone is not enough
Here is a subtle problem.
Imagine 5,000 application instances all experience a timeout at approximately the same time. Every instance uses exponential backoff and waits exactly one second before retrying.
What happens one second later?
Thousands of requests arrive together again.
Even though every client waited, they all waited for the same amount of time. The system can experience another sudden traffic spike.
This is where jitter becomes important.
Jitter introduces randomness into the retry delay. Instead of every client retrying after exactly one second, some might retry after 200 milliseconds, others after 600 milliseconds, and others later within the permitted delay window.
The requests become more spread out, reducing the likelihood of synchronized traffic bursts.
Think of it this way:
- Exponential backoff reduces how aggressively clients retry.
- Jitter prevents large numbers of clients from retrying together.
- A retry limit prevents the process from continuing indefinitely.
Together, these mechanisms make retries much safer.
AWS's engineering team explains this principle in the Amazon Builders' Library article, Timeouts, retries, and backoff with jitter .
4. Not every error deserves another attempt
One of the most common mistakes in error handling is retrying every failure.
But different errors have different causes.
| Error | Typical response |
|---|---|
| Temporary network failure | Retry within configured limits |
| HTTP 503 or transient server error | Consider a retry with backoff |
| API throttling | Back off and respect applicable retry guidance |
| Invalid input | Fail fast and fix the request |
| Access denied or missing permissions | Investigate permissions instead of blindly retrying |
| Request timed out during a payment | Determine whether the operation succeeded before repeating it |
The final example is particularly important.
A timeout does not necessarily mean that the operation failed. Your application might have sent a payment request, the payment service might have processed it successfully, and the response might have been lost before reaching the application.
If the application blindly repeats the request, the customer could be charged twice.
This brings us to another critical concept.
5. Idempotency: How to retry without repeating the side effect
An operation is idempotent when repeating the same operation produces the same intended result as performing it once.
Consider an application that creates a payment.
The first request reaches the payment service and succeeds, but the response times out. The application does not know whether the payment was processed.
Instead of submitting an entirely new payment, the application retries using the same unique idempotency key.
The payment service uses that key to recognize the repeated operation and return the existing result rather than creating another payment.
A simplified flow looks like this:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
Customer clicks "Pay"
|
v
Application creates an idempotency key
|
v
Payment service processes request
|
v
Response times out
|
v
Application retries with the same key
|
v
Existing result is returnedThe idempotency key alone does not magically guarantee safety. The receiving service must implement deduplication correctly, store the operation's result reliably, and define how concurrent duplicate requests are handled.
The same principle applies to order creation, account provisioning, resource creation, and other operations where repeating a request could create unwanted side effects.
AWS explores this in detail in its Builders' Library article, Making retries safe with idempotent APIs .
6. Building a more resilient architecture with AWS
Retries are only one part of the solution. The architecture should also control how work enters the system, how failures are isolated, and how unsuccessful operations are recovered.
Here is one practical pattern for an asynchronous application:
Request → Amazon API Gateway → AWS Lambda → Amazon SQS → Worker Lambda → Database or downstream service
The components have distinct responsibilities:
- Amazon API Gateway: Provides an API entry point and can apply throttling controls.
- AWS Lambda: Handles the initial request or processes queued messages.
- Amazon SQS: Buffers work so a temporary downstream slowdown does not necessarily force every request to be processed immediately.
- Worker Lambda: Processes queued tasks with controlled concurrency and appropriate error handling.
- Dead-letter queue (DLQ): Holds messages that fail repeatedly, allowing investigation without endlessly retrying the same work.
For example, suppose an application needs to process 50,000 customer notifications. If the notification provider becomes temporarily unavailable, an SQS queue can retain messages while workers process them at a controlled rate.
However, a queue is not an unlimited safety net. Queue depth, message age, worker concurrency, visibility timeouts, and downstream rate limits all need attention. For SQS-triggered Lambda functions, configure redrive behaviour and ensure processing is safe if a message is delivered more than once.
Also remember that API Gateway throttling, Lambda concurrency controls, and SQS buffering solve different problems. They work best when configured as complementary parts of the architecture.
For further reading, see the AWS Compute Blog's guide to building resilient serverless applications .
7. A practical checklist before deploying retry logic
Before shipping a retry mechanism, ask these questions:
- Have we set timeouts? A request should not consume resources indefinitely while waiting for a response.
- Are retries limited? Define a maximum number of attempts and an overall time budget appropriate for the operation.
- Do we use exponential backoff and jitter? Avoid immediate retries and synchronized retry bursts.
- Are errors classified correctly? Retry transient failures, but fail fast on errors that will not improve through repetition.
- Are operations idempotent? Repeating a request must not unintentionally create duplicate payments, orders, or resources.
- Are retries implemented at the right layer? Multiple layers independently retrying the same request can multiply the total number of attempts.
- Can we observe the failure pattern? Monitor error rates, latency, throttling, retry counts where available, queue depth, and message age.
- Have we tested failure scenarios? Simulate throttling, timeouts, and downstream unavailability to see whether the system recovers or amplifies the problem.
Do not assume the SDK's default configuration automatically solves every application-level retry problem. Verify its behaviour and combine it with appropriate application logic.
8. The bigger lesson: Design for recovery, not just success
Developers naturally spend time optimizing the successful path. We make APIs faster, reduce database queries, improve caching, and scale compute capacity.
But production resilience is also determined by what happens when a dependency becomes slow, a network call times out, or a service temporarily rejects requests.
A robust application does not simply keep trying until something works. It knows which failures are worth retrying, how long to wait, when to stop, and how to prevent a repeated request from causing a second side effect.
That is the difference between an application that works under normal conditions and one that can survive abnormal conditions.
The next time you add a retry loop, ask yourself one question:
If every client in my system experienced this failure at the same time, would my retry logic help the system recover, or make the outage worse?
That question is worth asking before your next production deployment.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article