The Time Retries Turned Into a Retry Storm

Most backend failures are fairly easy to understand after the fact.

A database query gets slow. An API goes down. A deployment introduces a bug.

Then there are failures where the system is technically doing exactly what you told it to do, and that is the problem.

I learned that lesson while working on a backend service that depended on an external API. Like most production services, we had retry logic. If an upstream request failed because of a temporary error, retrying seemed preferable to immediately failing the user's request.

That assumption was reasonable.

Our implementation was not.

The first symptom

The problem started with what looked like a normal transient failure.

An upstream service began returning errors for a short period. There was no catastrophic outage, and under normal traffic the backend should have recovered without much trouble.

Instead, request latency started increasing.

Then the error rate increased.

Then the number of outgoing requests increased as well.

That last part was the strange one.

If an upstream dependency was struggling, I expected our successful request volume to decrease. Instead, our service was suddenly generating much more traffic toward the dependency that was already having problems.

The retry mechanism had become an amplifier.

One failed request became several.

Several failures became dozens of almost simultaneous retries.

As more requests entered the system, each one brought its own retry sequence with it.

We had created a retry storm.

What our retry logic was actually doing

The original logic was conceptually similar to this:

for attempt in range(3):
try:
return call_upstream_service()
except TemporaryError:
time.sleep(1)

At first glance, this does not look particularly dangerous.

A request fails.

Wait one second.

Try again.

The problem becomes clearer when hundreds of requests are doing it simultaneously.

Imagine 300 requests reaching the backend around the same time.

The upstream service starts failing, so most of those requests fail together.

They all wait for one second.

And then, almost exactly one second later, they all try again together.

The upstream service receives another burst.

If it is still degraded, those requests fail again.

One second later, another burst arrives.

Instead of helping the upstream system recover, our retries were repeatedly hitting it with synchronized traffic spikes.

The retry logic had accidentally synchronized the clients.

Fixed delays were the first problem

The first thing we changed was replacing the fixed retry interval with exponential backoff.

Instead of:

1 second
1 second
1 second

the delay increased after each failure:

1 second
2 seconds
4 seconds

The idea is straightforward.

If a dependency continues failing, the client becomes progressively less aggressive.

A simplified implementation looked like this:

delay = 1

for attempt in range(3):
try:
return call_upstream_service()
except TemporaryError:
time.sleep(delay)
delay *= 2

This helped, but it did not completely solve the problem.

If 300 requests failed at approximately the same time, they would still wait approximately the same amount of time.

After one second, many of them would retry together.

Then after two seconds, they could synchronize again.

We had reduced the frequency of the bursts, but not the synchronization.

Then came jitter

The next improvement was adding randomness to the retry delay.

Instead of every client retrying after exactly two seconds, one might retry after 1.4 seconds, another after 2.1 seconds, and another after 2.7 seconds.

For example:

import random
import time

base_delay = 1

for attempt in range(3):
try:
return call_upstream_service()
except TemporaryError:
delay = base_delay * (2 ** attempt)
jitter = random.uniform(0, delay)

time.sleep(jitter)

There are several jitter strategies, but the important idea is the same: do not let every failed request wake up at the same moment.

That small amount of randomness spreads retry traffic across time.

The traffic pattern became much smoother.

And that was when I started understanding that retry behavior is not really about individual requests.

It is about system behavior.

Code that looks perfectly reasonable for one request can behave very differently when thousands of requests execute it concurrently.

We also had to decide what deserved a retry

Another mistake was treating too many failures as retryable.

Not every error gets better if you repeat the request.

If an API returns a server-side transient error, retrying may make sense.

If authentication credentials are invalid, sending the exact same request three more times probably will not improve the situation.

We started separating failures into two broad categories.

Retryable failures included things such as:

- temporary upstream errors
- connection timeouts
- some rate-limiting responses
- short-lived network failures

Non-retryable failures included things such as:

- invalid input
- authentication failures
- permission errors
- requests that violated known API constraints

That distinction reduced unnecessary traffic considerably.

Retries should be selective.

A retry is not a generic response to failure. It is a response to a failure that has a reasonable probability of succeeding if attempted again later.

Observability was what made the problem obvious

One of the most useful parts of the incident was realizing that application error logs alone were not enough.

A request might eventually succeed after three attempts.

From the user's perspective, the operation succeeded.

But internally, we may have generated three times the expected traffic and added several seconds of latency.

If we only tracked final request success, we would miss most of the story.

We started paying closer attention to metrics such as:

request_count
retry_count
retry_attempt
upstream_error_rate
request_latency
upstream_latency

We also added structured logging around the retry flow.

For example:

logger.warning(
"upstream_retry",
extra={
"attempt": attempt,
"max_attempts": MAX_ATTEMPTS,
"service": "upstream_api",
"error_type": type(exc).__name__
}
)

Sending these logs and metrics into Amazon CloudWatch made the pattern much easier to see.

Instead of simply seeing:

Error rate increased

we could see something closer to:

Upstream failures increased
↓
Retry volume increased
↓
Outgoing request volume increased
↓
Latency increased
↓
More requests remained in flight

That causal chain was much more useful than the original exception message.

Retry budgets changed how I thought about the problem

One of the more important design decisions was putting a hard limit on retries.

Without limits, a degraded dependency can consume a surprising amount of application capacity.

Suppose the backend normally handles 1,000 operations.

If every operation can generate three additional attempts during an outage, the system may suddenly have to process several thousand outbound calls for roughly the same amount of user traffic.

So retries became a budget rather than an unlimited recovery mechanism.

We limited:

- the maximum number of attempts
- the maximum delay between attempts
- the total time spent retrying
- the types of exceptions eligible for retry

This also made failure behavior more predictable.

Sometimes the correct action is simply to stop retrying and return an error.

That sounds less resilient, but repeatedly attacking an unhealthy dependency is not resilience either.

Rate limiting provided another layer of protection

Retries were only one part of the traffic problem.

During sudden bursts, we also needed to prevent incoming traffic from creating an uncontrolled amount of downstream work.

Rate limiting helped place a boundary around that.

In our case, Redis was useful for tracking request limits across backend instances rather than relying only on process-local counters.

The goal was not simply to reject traffic.

The goal was to ensure that a temporary spike at the edge of the application could not multiply into a much larger spike deeper inside the system.

That distinction matters in distributed systems.

Traffic can amplify as it moves through layers.

One incoming API request might trigger:

1 incoming request
→ 2 database operations
→ 1 external API call
→ 3 retry attempts
→ several log and telemetry events

By the time you are debugging the downstream service, the original request volume may no longer explain the load you are seeing.

What the final retry flow looked like

The resulting strategy was closer to:

import asyncio
import random

MAX_RETRIES = 3
BASE_DELAY = 0.5
MAX_DELAY = 8

async def call_with_retry():
for attempt in range(MAX_RETRIES + 1):
try:
return await call_upstream_service()

except RetryableError:
if attempt == MAX_RETRIES:
raise

exponential_delay = BASE_DELAY * (2 ** attempt)
capped_delay = min(exponential_delay, MAX_DELAY)

delay = random.uniform(0, capped_delay)

await asyncio.sleep(delay)

The production implementation had more context around exception handling, logging, metrics, and timeouts, but the important ideas were fairly simple:

Retry only failures that may actually recover.

Use exponential backoff.

Add jitter.

Set strict retry limits.

Measure retries separately from final request failures.

The bigger lesson

Before this incident, I thought of retries as a reliability feature.

Afterward, I started thinking of them as controlled load generation.

Every retry creates work.

It consumes a connection, CPU time, network bandwidth, and capacity from the dependency being called.

During healthy operation, that extra work is usually negligible.

During an outage, it happens exactly when the system has the least spare capacity.

That is why retry logic deserves the same level of attention as caching, rate limiting, database connection pools, and timeout configuration.

The individual implementation can be only a few lines of code, but the system-level behavior can be significant.

What I check now

Whenever I add retries to a backend service, I ask a few questions first:

1. Which failures are actually retryable?
2. How many additional requests can one user request generate?
3. Are retries synchronized across clients?
4. What happens if the dependency remains unavailable for ten minutes rather than ten seconds?
5. Can I see retry volume independently in my monitoring?
6. Is there a hard upper bound on the amount of retry traffic the application can generate?

Those questions are much more useful than simply asking whether the application has retries.

Because almost any system can retry.

The harder part is making sure it knows when to stop.