
When Redis Goes Down It Takes Production With It
While benchmarking a personal project and following AWS guidelines, I hit a real problem: a Redis cache that was not failover-safe in front of a Multi-AZ database. It made me realize how an entire production environment can go down over one overlooked edge case. Here are the problems and simple fixes, and why System Design and a Reliability study before deploying are essential to a good product.
The short version
Redis is the obvious home for session data and rate-limit counters. It’s fast, atomic, and shared across your fleet. But the moment your app can’t function without it, Redis stops being a cache and becomes an unreplicated primary database, a single point of failure. And it fails in a particularly nasty way. If your rate limiter loses Redis and fails open, every request (including the abusive traffic the limiter existed to stop) stampedes your database and takes down production. If your sessions live only in Redis, an outage logs out every user at once. The fix isn’t “fall back to the database,” because done naïvely that is the outage. The fix is to stop treating a cache as a hard dependency: layer your defenses, make the fallback bounded, and give sessions either durability or statelessness.
The setup everyone builds
It starts innocently. You need rate limiting, so you put a counter in Redis:
INCR user:123 , set a TTL, done. You need shared sessions across your autoscaled app servers, so you put the session blob in Redis too. Both are textbook uses. Both work beautifully, until the day Redis doesn't.Here’s the thing nobody stress-tests: what does your app do when Redis is gone? Not “gone” as in a clean outage, either. The worse case is Redis getting slow. A degraded node, a failover blip, a network hiccup. Slow is worse than down, because every request now blocks waiting for a cache call, and your connection pools drain while you watch.
Whatever your app does in that moment is your real architecture. Everything else is a diagram.
This is literally the pattern that AWS SAA teaches you
Here’s the part that makes this so common: putting a cache in front of your database is the textbook AWS answer. If you’ve studied for the Solutions Architect Associate, you know the reflex. The app is slow or the database is overloaded, and the “correct” move is to add Amazon ElastiCache (Redis) in front of it. Cache-aside: check the cache, on a miss read the database and populate the cache, serve subsequent reads from memory. It genuinely works. You offload the database, you cut read latency from milliseconds to microseconds, and you scale reads without scaling the DB. It’s a good pattern. It’s the right pattern.
So this article isn’t arguing against caching. Sessions and rate-limit counters in Redis are that same instinct applied, putting the hot, frequently-accessed data in memory in front of the slow backend. The theory is sound.
But notice what the exam answer actually is. The exam doesn’t just say “add ElastiCache.” The full, marks-earning answer is “add ElastiCache in a Multi-AZ configuration with automatic failover”, or with read replicas, or cluster mode. The redundancy is not a footnote. It’s the point. The pattern is only safe because it’s assumed to be highly available. In the real world, the cache gets shipped and the “Multi-AZ, failover-safe, and what-happens-when-it’s-down” part quietly gets skipped. A single node, no replica, and an app that silently assumes it’s always there.
That’s the whole gap this article lives in: the theory is correct, but the theory includes the redundancy, and production often doesn’t. A cache in front of the DB that isn’t redundant and failover-safe isn’t the exam’s pattern, it’s the exam’s pattern with the safety removed. Everything below is what “failover-safe” has to actually mean once you take it seriously.
Failure #1: the rate limiter that becomes a DDoS on yourself
Your rate limiter has exactly two options when it can’t reach Redis, and both are traps.
Fail open. Let requests through when you can’t count them. Feels safe: “don’t block legit users over an infra hiccup.” Except now there’s no limiter at all. 100% of traffic (the good, the bursty, and the abusive) pours straight through to your backend. Your database was provisioned for the rate-limited load. It’s now getting the unlimited load. Connections exhaust, queries queue, timeouts cascade, and the app falls over. Your safety mechanism just became the outage trigger. The rate limiter existed to protect the database, and its failure is what killed it.
Fail closed. Reject everything you can’t count (503). Also feels safe: “better safe than sorry.” Except a five-second Redis blip now returns errors to every user on the platform. You converted a cache outage into a full app outage, just from the other direction.
Neither “just fail open” nor “just fail closed” is an answer. The answer is more subtle, and we’ll get there.
Failure #2: the sessions that evaporate
Now the sessions. If Redis is the only place a user’s session lives and it goes down (or even just evicts keys under memory pressure, or loses recently-written data during a node failover, which a pure cache can do), every user is instantly logged out or frozen mid-action. Your app servers are healthy. Your database is healthy. And nobody can log in, because the one volatile box holding auth state hiccuped.
This is the same root cause as the rate-limiter problem wearing a different hat: you made a best-effort cache the system of record for something that isn’t allowed to disappear.
The trap in “just fall back to the database”
The instinct is right. The database is the real source of truth, so on a Redis failure, read from the database. Sessions? Load the user from the DB. Rate limits? Count in the DB.
Naïvely, this recreates the exact catastrophe you were trying to avoid. The instant Redis drops, every request misses cache at the same time and stampedes the database simultaneously, a thundering herd. And here’s the part people miss: a circuit breaker doesn’t save you. A circuit breaker stops you from hammering the failed Redis. It does nothing to protect the database you just redirected 100,000 requests/second onto. If that DB was built for 1,000 QPS, it’s now the bottleneck, and it crashes just like Redis did. You didn’t remove the single point of failure. You moved it downstream.
A fallback is only safe if the fallback path is itself protected. That’s the whole game.
Doing it right, part 1: rate limiting that degrades instead of detonating
Three ideas, layered:
1. A local limiter that’s always on. Give every app instance an in-process token bucket. When Redis is healthy, it’s a loose backstop. When Redis is down, it becomes the limiter, not “unlimited,” just “less precise.” Size it more permissively than your global limit (so N instances don’t collectively reject legit traffic) but tightly enough to keep the database alive: roughly (DB-safe QPS ÷ number of instances). Now “fail open” means “fall open to a local limit,” not “fall open to the void.”
2. A circuit breaker for latency, not just errors. After a handful of consecutive Redis timeouts, trip the breaker and fast-fail to the local limiter in zero milliseconds instead of hanging for seconds on every call. This is what stops slow-Redis from becoming slow-everything.
3. Protect the database independently. Assume the limiter will fail one day and make the DB survive it anyway. Put a second token bucket directly in front of your database calls (the “missing piece after the circuit breaker”). Front the DB with a connection pooler. On AWS, RDS Proxy multiplexes thousands of app connections down to a handful, absorbing the connection storm a fallback surge creates. Add statement timeouts and bounded pools so a slow DB sheds load instead of queuing to death. And push a coarse limit to the edge, AWS WAF rate-based rules or API Gateway throttling, so there’s a floor that doesn’t depend on your Redis at all.
Defense in depth: edge limit → local limit → distributed (Redis) limit → database admission control. No single failure is total.
Doing it right, part 2: sessions that survive
Option A. Go stateless. Issue a short-lived signed token (a JWT, 5 to 15 minutes) plus a refresh token. The access token validates by signature, so a Redis outage doesn’t log anyone out for the token’s lifetime. The session is a credential the server verifies without touching any store. This is the strong version of “temporary credentials.” The honest catch: signed tokens are hard to revoke instantly (they’re valid until they expire). So keep a small revocation denylist (and only the denylist) in Redis, make short TTLs bound your exposure, and let the denylist fail open if it’s unreachable. You’ve inverted the dependency: Redis now holds the rare exceptions, not the entire session.
Option B. Keep server-side sessions, but make them durable. Don’t let the only copy live in a volatile cache. Put authoritative sessions in a durable store, Amazon MemoryDB (Redis speed, but writes are acknowledged only after a multi-AZ transaction log, so failover doesn’t lose data) or DynamoDB with TTL, and use Redis as a read cache in front of it. Now a Redis miss is just… a cache miss. You rehydrate from the durable store and the user stays logged in.
Either way, when you do read back from the durable store under load, coalesce: collapse concurrent lookups for the same session into a single query (singleflight), add jittered TTLs so keys don’t all expire in lockstep, and keep RDS Proxy in front. The herd becomes a trickle.
The principle underneath all of it
Every fix above is the same idea in different clothes:
A cache must never be a hard dependency. If Redis disappearing takes down your auth or your database, it was never a cache. It was an unreplicated primary you forgot to treat like one.
So: contain the blast radius with layers. Make every fallback bounded, never “unlimited to the database.” Give sessions durability or statelessness, not a volatile box as their only home. Design for slow, not just down. And decide fail-open versus fail-closed per surface: fail closed on auth and payments, fall open (to a local limit) on general APIs.
Redis isn’t the problem. Trusting it with things it was never promised to keep is. Build like it will vanish at 3 a.m., because eventually it will, and the outage becomes a shrug instead of a postmortem.
Written by Matheus Sabaudo, AWS Data Engineer.
Deep-dive with the AWS building blocks (ElastiCache vs MemoryDB, DynamoDB counters, RDS Proxy, WAF).
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article