AWS Builder Center
ElastiCache for Valkey Just Became a Durable Data Store. Here's What That Actually Means (Part 2: Benchmarks and Fault Injection Testing)

ElastiCache for Valkey Just Became a Durable Data Store. Here's What That Actually Means (Part 2: Benchmarks and Fault Injection Testing)

I benchmark ElastiCache for Valkey's durability modes against MemoryDB, then inject real AZ failures with AWS Fault Injection Service to verify acknowledged writes actually survive — with a few surprises along the way, including a client-side measurement artifact and evidence that ElastiCache's durable clusters share MemoryDB's engine internally.

Series: ElastiCache for Valkey Just Became a Durable Data Store (2 articles)

  1. 2
    ElastiCache for Valkey Just Became a Durable Data Store. Here's What That Actually Means (Part 2: Benchmarks and Fault Injection Testing) This article
Project Manager/General Manager
This is Part 2 of a two-part series. Part 1 covered the internal architecture of Durability, consistency semantics, constraints, operational design, cost, multi-Region patterns, and how it changes the way you pair Valkey with Aurora. This post puts those claims to the test by actually running them. I benchmark all three ElastiCache modes (no durability, Async, Sync) and MemoryDB with memtier_benchmark, then use AWS Fault Injection Service (FIS) to actually inject an AZ failure and check whether acknowledged writes really do survive.

Introduction

The guarantee I wrote about in Part 1, that an acknowledged Sync write never disappears, is still just a contract written in the official documentation. I don't think it's responsible to write about it without actually injecting an AZ failure and checking with my own eyes whether it behaves as documented. This post is that verification.
For the tests, I picked the claims from Part 1 that can actually be measured: write and read latency differences across durability modes on the performance side, how data behaves after an AZ failure on Sync vs. Async on the fault-tolerance side, and a comparison against MemoryDB.
I ended up re-running several of these measurements more than once, mostly because of my own misjudgments and gaps in the measurement approach that I only caught partway through. That said, these are results from my own personal environment, so treat them as one data point rather than a definitive benchmark.

Verification environment

I put together a full verification environment (CloudFormation templates and test scripts). Here's the setup:
  • ElastiCache for Valkey 9.1 (cache.r7g.large, 1 shard = primary + replica, Cluster Mode Enabled, Multi-AZ, TLS, Tokyo Region)
  • The durable cluster's primary is pinned to AZ-1 via NodeGroupConfiguration. FIS's interrupt-az-power also targets AZ-1 in the same template, so I don't have to look up where the primary is before every experiment
  • A comparison non-durable cluster, built under the same conditions (CME/TLS/same instance type)
  • An EC2 client in a public subnet, reached over SSM Session Manager
  • A MemoryDB cluster of the same family and size (db.r7g.large) in an optional stack, for the MemoryDB comparison
The durable cluster's constraints (CME and TLS both required) carry straight through into the benchmark and test script command lines. Forget --cluster-mode or --tls and you can't even run a measurement against the durable cluster.

Files used to build this

I've published the final versions of the CloudFormation templates and scripts I used for this environment — two templates and eleven scripts — in a public repository: sh-fk2/elasticache-valkey-durability-verification 

Benchmarking each mode with memtier_benchmark

I used memtier_benchmark for the ElastiCache performance tests.

Setup

  • Test targets (4 total): three ElastiCache modes (no durability, Async, Sync) plus MemoryDB
    • Durability can't be retrofitted, so no-durability and durable are built as separate clusters. Async and Sync are measured by switching the same cluster back and forth
  • I matched the node type across ElastiCache and MemoryDB to the r7g.large family. Comparing latency requires the same family and size on both sides

Running memtier_benchmark

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
# needs a TLS-capable build (build from source if your package is too old)
# AL2023 is on the pcre2 family, so use pcre2-devel (pcre-devel may not be in the repo)
sudo dnf install -y autoconf automake make gcc-c++ pcre2-devel zlib-devel openssl-devel libevent-devel
git clone https://github.com/RedisLabs/memtier_benchmark.git
cd memtier_benchmark && autoreconf -ivf && ./configure && make && sudo make install

# measurement (50% writes, 50% reads, 256 bytes, 10 threads x 10 clients, fixed at a 30,000 ops/sec target)
memtier_benchmark \
--host=<configuration-endpoint> \
--port=6379 \
--protocol=redis \
--tls \
--cacert=/etc/pki/tls/certs/ca-bundle.crt \
--cluster-mode \
--threads=10 \
--clients=10 \
--test-time=120 \
--rate-limiting=300 \
--data-size=256 \
--key-maximum=100000 \
--ratio=1:1
--cacert is required. I got stuck on this one too: run it with just --tls, and every single connection fails with:
1
TLS connection error: certificate verify failed
What's confusing is that there's nothing wrong with the certificate itself. openssl s_client against the same instance comes back Verification: OK, and valkey-py (ssl=True) connects without issue. ElastiCache's certificate is signed by a public CA, so any check against the OS's CA bundle passes fine. It's specifically memtier's own behavior: it doesn't automatically consult the system's CA store. The path above is for Amazon Linux 2023; adjust it for other distributions.
Setting --ratio to 1:1 (50% writes) was deliberate. The durability-mode differences show up on the write path, so pushing the write ratio up makes the gap I want to observe easier to see. That choice does have a side effect, though. The AWS official numbers I quoted earlier in this series used an 80:20 read:write ratio, so given the mismatched conditions, this article's table can't be lined up directly against the official numbers.
One more thing: I originally ran this command with no --rate-limiting at all, at full throttle (closed-loop, send as fast as it'll go). That approach had a blind spot. Each mode's write speed is completely different, so at full throttle, no-durability runs at around 138,000 ops/sec while Sync runs at around 59,000 ops/sec, each mode landing on its own throughput. That's not "the same load condition" for comparison purposes, and it specifically breaks any comparison of read latency (more on this below). So I switched to --rate-limiting (a per-connection request-rate cap) and --test-time (a fixed duration), pinning all four modes to the same target throughput (30,000 ops/sec combined) and re-measuring. With 10 threads x 10 clients = 100 connections, --rate-limiting=300 gives a combined target of 300 x 100 = 30,000 ops/sec. (memtier's own runtime log reports Total connections (10 x 10 x 2) = 200, since it opens connections to both the primary and the replica. A smoke test confirmed GET traffic never actually uses the replica connections, though, so only the 100 primary-side connections carry any traffic — that's the effective count the rate math above is based on.)
The idea of comparing at a fixed TPS is itself borrowed from AWS's own benchmark methodology (50K/100K TPS). The tool AWS uses, though (valkey-benchmark), doesn't appear to have a native rate-limiting feature, at least as of the publicly proposed GitHub issue I found. The official post states it distributed traffic across "ten separate EC2 instances," so I'd guess the fixed TPS isn't an explicit software throttle so much as a calibration achieved by tuning connection counts to hit the natural throughput target.
I also record DurabilityLag (during Async) and TrafficManagementActive in parallel throughout, to confirm no log-side throttling snuck into the measurement.
I benchmarked MemoryDB with the same command (CME + TLS + RESP, connecting the same way as durable Valkey). Just swap the endpoint for MemoryDB's. MemoryDB is always synchronously durable, so there's only one mode to measure there.

Results

The measurement conditions are exactly the command above (--ratio=1:1, 256 bytes, 10 threads x 10 clients, fixed at a 30,000 ops/sec target, 120 seconds, a 100,000-key space). All four modes were measured under identical load conditions.
ModeWrite P50Write P99Throughput (ops/sec)
ElastiCache, no durability0.255 ms0.775 ms29,983
ElastiCache Async0.247 ms0.775 ms29,994
ElastiCache Sync3.535 ms5.567 ms29,873
MemoryDB (always synchronous)3.295 ms5.343 ms29,894
Median of 3 runs per mode. Misses/MOVED/ASK were all zero across every run. Misses are cache misses (GETs that hit a key that didn't exist); MOVED/ASK are redirects caused by a change in the cluster's slot assignment (MOVED is a normal rebalance, ASK is the transient kind you see mid-resharding). All three staying at zero confirms the key space was hit 100% of the time and the cluster's topology never shifted during the measurement.
Writes came out exactly as expected. Sync and MemoryDB are about 13-14x slower than no-durability/Async, a reasonable cost for waiting on 2-AZ persistence (this also lines up with AWS's own note that "Sync runs 2-3ms").
ElastiCache Sync and MemoryDB are essentially tied in this table too. That isn't a coincidence. While investigating this, I compared the os field from the INFO command and found that ElastiCache's durable cluster (in either Sync or Async mode) reports os as Amazon MemoryDB. The non-durable cluster stays Amazon ElastiCache. That's a measured confirmation that ElastiCache's durability feature runs on the same engine stack as MemoryDB internally.

The read-latency mystery

I've deliberately left read (GET) latency out of that table. There's a reason.
At first, when I measured with --ratio=1:1 issuing writes and reads alternately over the same connection, GET P50 came out looking faster specifically on Sync and MemoryDB (about 0.25ms for no-durability/Async, versus about 0.16ms for Sync/MemoryDB). A GET command never touches the durability-confirmation path at all, so a gap tied to durability mode shouldn't exist in the first place.
Suspecting this gap came from the client-side measurement method itself, I went as far as fully separating the SET-only and GET-only connections, then issuing them simultaneously from two separate EC2 instances (a design closer to how AWS's own benchmark uses ten instances), and re-measured.
ModeRead P50Read P99
ElastiCache, no durability0.151 ms0.391 ms
ElastiCache Async0.159 ms0.463 ms
ElastiCache Sync0.159 ms0.447 ms
MemoryDB0.167 ms0.479 ms
Once GET was fully isolated, all four modes landed in a tight 0.15-0.17ms band. The gap I'd seen at first was a client-side measurement artifact: --rate-limiting controls the interval between issued commands on a single connection without distinguishing between types, so a fast SET on no-durability/Async needs an artificial wait afterward, while a slow SET on Sync/MemoryDB eats up nearly all of that wait time on its own, and the following GET inherits none of it — a fight over the same connection's timing budget. Read performance doesn't depend on durability mode. The only real difference is on the write side.

Fault-tolerance testing with FIS

Beyond performance, I want to confirm that data really doesn't disappear when an AZ failure actually happens. I used FIS (Fault Injection Service) to trigger the failure.

The FIS action I used

I built an experiment template with exactly one action, aws:elasticache:replicationgroup-interrupt-az-power. It force-stops the ElastiCache nodes in a given AZ. If the primary is the target, a promotion runs to whichever replica has the least replication lag. Since replacement-node startup in the affected AZ is also blocked during the interruption, you get to observe degraded-state behavior at the same time.
I got stuck twice writing this template, so let me save you the trouble. The rule is: tags and the AZ go on the target, duration alone goes on the action.
First, parameter placement. The duration for how long the interruption lasts is an action parameter, but availabilityZoneIdentifier, which specifies the target AZ, turned out to be a required target parameter. Put it on the action and template creation fails outright.
Second, how you specify the target. aws:elasticache:replicationgroup doesn't support ARN-based targeting (resourceArns) at all, only tags. Write it with an ARN and CreateExperimentTemplate returns a 400 with:
1
TargetResourceType aws:elasticache:replicationgroup does not support resourceArns.
I couldn't find this documented anywhere. The generic target syntax page lists resourceArns, and the list of supported resource types includes aws:elasticache:replicationgroup, but nowhere does it say this specific combination is unsupported. I only found out when my CloudFormation stack hit CREATE_FAILED.
One thing to watch for with tag-based targeting: since a tag matches every resource that has it, you need a tag key that's applied only to the durable cluster, so you don't accidentally pull in the non-durable comparison cluster too. I used a dedicated tag, FisAzPowerTarget. availabilityZoneIdentifier doesn't help here as a safety net, either, since the non-durable cluster also pins its primary to the same AZ-1; once the tag matches, the AZ won't exclude it.
https://docs.aws.amazon.com/fis/latest/userguide/fis-actions-reference.html#elasticache-actions-reference
https://docs.aws.amazon.com/fis/latest/userguide/targets.html
One tip for building this environment: ElastiCache's primary placement isn't deterministic by default, so pinning the primary's AZ with NodeGroupConfiguration's PrimaryAvailabilityZone, and matching FIS's availabilityZoneIdentifier to that same AZ in the template, means you never have to look up where the primary landed before every experiment. (I'd hit this exact problem with RDS clusters in past FIS testing more than once, so I made sure to get it right this time.)
Now let's go through each test and what came out of it.

Test 1: Does Sync Durability keep acknowledged writes from disappearing?

The durable cluster requires CME + TLS, so the client writes cluster-aware and over TLS too. I record every acknowledged key, then after the failover, check that every single one of them still exists.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
from valkey.cluster import ValkeyCluster

client = ValkeyCluster(
host='<configuration-endpoint>',
port=6379,
ssl=True,
socket_timeout=5,
)

acked_keys = []
fail_count = 0

for i in range(50000):
key = f"key:{i}"
try:
if client.set(key, f"value:{i}"):
acked_keys.append(key)
except Exception:
# a connection drop or CLUSTERDOWN during the failover lands here.
# writes without an ACK aren't guaranteed, so count separately
fail_count += 1

print(f"acked: {len(acked_keys)}, failed: {fail_count}")

# --- after the failover completes ---
missing = [k for k in acked_keys if not client.exists(k)]
print(f"acked keys that disappeared: {len(missing)}") # should be 0 on Sync
The only claim this test can back up is that acknowledged writes survive. A write that hit the exception path might still have landed and survived, or might not have, depending on the specifics, and that's true on Sync too. Application-side design should assume failed writes need a retry.
The sample uses range(50000), but that count has to be tuned to your environment. This is a single-threaded sequential SET (one round trip per operation), so you can't set a target throughput; it's whatever the per-operation round trip gives you. And since Sync (ACK waits on 2-AZ persistence) and Async (ACK before persistence) have wildly different effective throughput, the same count takes very different amounts of time to run. Leave it at 50,000 and the loop might finish in a few tens of seconds, meaning there's no write in flight the instant the fault gets injected. I ran 10,000 writes first to measure the real ops/sec, then worked backward to the production count from "measured ops/sec x (wait time before FIS starts + the FIS action's duration)."
As a secondary check, I also confirm the total key count across all shards.
1
2
# returns the sum across all primaries (omit target_nodes and you only get one node's count)
print(client.dbsize(target_nodes=ValkeyCluster.PRIMARIES))
I also record DurabilityLag (should stay 0 on Sync) and the failover event log throughout the experiment.
Result: Zero acknowledged keys disappeared (confirmed across all 6 FIS injection runs). Exactly as expected, not a single acknowledged write vanished.

Test 2: Measuring Async's RPO

Run the same write loop under Async while injecting the AZ failure, then check the number of acknowledged keys that disappeared against the DurabilityLag value just before the failure. The spec's ceiling is 10 seconds, but I want to see how the actual loss lines up against the observed buffer value. While I'm at it, I also watch whether write rejections on buffer overrun (DurabilityBufferExceededErrorCount) happen, and whether the client's backoff retry can absorb them.
Run the write loop long enough to cover the FIS action's duration (2 minutes by default), so the fault-injection window is guaranteed to overlap the write window. The count is decided the same way as Test 1.
Result: Here's what disappeared, across 3 runs. Well under the write volume the spec's 10-second ceiling would allow, and the difference from Sync came through clearly in the measurement.
RunAcknowledged keys lost
11,346
21,141
3967

Test 3: Switching dynamically from Async to Sync

Change the durability mode from Async to Sync while writes are in flight, and check whether that causes any downtime.
1
2
3
4
aws elasticache modify-replication-group \
--replication-group-id <cluster> \
--durability sync \
--apply-immediately
Record how long it takes EffectiveDurability to transition, and the write success rate during that window.
This command needs a CLI version that supports --durability. My local 2.34.41 didn't support it; 2.36.8 did. What makes an old CLI annoying isn't just that --durability fails with Unknown options, it's that describe-replication-groups's EffectiveDurability comes back as null instead of an error. You can't tell whether the switch hasn't finished, or the CLI is just dropping the response field.
Result: No downtime. The switch took about 14.6 seconds, with zero errors or write stalls throughout (a single switch, measured once — unlike the other tests, I didn't repeat this one).

Test 4: Comparing failover against MemoryDB

The performance comparison turned out to be a clean apples-to-apples swap: point memtier at the MemoryDB endpoint instead, and it's the same conditions. I figured the failover comparison would be just as simple if I reused FIS as-is, but it turned out not to be.
I stood up the same db.r7g.large, 1-shard-plus-1-replica cluster on MemoryDB too, to compare failover behavior. What matters here is making sure the failover-injection method matches.
There are two fundamentally different ways to trigger a failover:
  • Planned (graceful) failover: promote a replica through an API call, while the primary stays alive. test-failover on ElastiCache, failover-shard on MemoryDB
  • Unplanned (abrupt) failure: kill the primary outright. FIS's interrupt-az-power on ElastiCache (= Test 1). MemoryDB has no FIS action that hits a single cluster with an AZ power cut the same way (the only one available is aws:memorydb:multi-region-cluster-pause-replication)
My original plan was to trigger MemoryDB the same way too, with FIS (unplanned), to match ElastiCache's FIS (unplanned). It turned out MemoryDB doesn't have an unplanned-failure action equivalent to ElastiCache's FIS power cut. So I lined both services up on planned failover instead.
1
2
3
4
5
6
# ElastiCache (planned failover)
aws elasticache test-failover \
--replication-group-id <cluster> --node-group-id 0001

# MemoryDB (planned failover). Check the shard name with describe-clusters first
aws memorydb failover-shard --cluster-name <cluster> --shard-name <shard>
What this comparison mainly captures is promotion time (both are planned, so data loss is near zero either way and won't show a difference there). ElastiCache durable and MemoryDB use the same Multi-AZ transactional log design, and the official docs describe the same design philosophy on both sides: promote whichever replica is "most up to date" (MemoryDB) or has "the least replication lag" (ElastiCache). So the underlying mechanism itself isn't expected to differ much; what's actually worth measuring is how fast each service's implementation of that same mechanism runs.
Result: Here's what MemoryDB's failover-shard measured, across 4 runs.
RunDuration
122.264s
218.349s
322.190s
421.133s
That's somewhat longer than ElastiCache and with more variance too. Given the shared engine noted above, though, I think a gap this size more likely comes from the small run count than a real difference, and it needs re-verification.
Measuring it turned up an interesting side note. Comparing the FIS test from Test 1 (unplanned) against this planned failover (test-failover) under the same load condition and the same measurement method, ElastiCache itself came out to almost the same duration either way. Result: here's the breakdown, and both land in the same range.
Planned failover (test-failover, 3 runs each):
RunSync durationAsync duration
117.356s18.505s
217.419s17.233s
317.502s16.180s
FIS alone, unplanned (6 runs each):
RunSync durationAsync duration
117.204s17.144s
217.311s17.372s
317.343s17.388s
417.362s15.263s
518.960s22.376s
622.233s22.441s
(Sync and Async are independent measurements — a matching run number doesn't mean they're paired.)
Whether the failure was triggered by an API-driven planned promotion or an unplanned AZ power cut, I couldn't find a clear difference. Taken together with the os-field discovery above (that ElastiCache's durable cluster runs on MemoryDB's engine stack internally), that supports the view that both the planned and unplanned paths ultimately exercise the same underlying failover mechanism.
Where ElastiCache durable and MemoryDB can't be compared is the unplanned-failure path (AZ power cut) and the dynamic Async-to-Sync switch. I'd like to test the unplanned-failure side once FIS supports it.
One more thing worth keeping in mind when planning to run both side by side: the two services count their test-failover API quotas differently. ElastiCache's test-failover is documented as "up to 15 shards in any rolling 24-hour period," counted by shard; MemoryDB's failover-shard is documented as "up to five times in any 24-hour period," counted by call.
MemoryDB's limit bites first in practice. Five is a hard, explicit number, so getting a step wrong and having to redo the test can burn through your daily quota. ElastiCache counts by shard instead, so how many times you can repeatedly fail over the same shard on a single-shard cluster isn't something the documentation answers. If you're planning on repeated testing, treat this as an unknown constraint. This round of testing prioritized comparing failover duration, so I didn't actually test how many repeats are possible.

Wrapping up the series

Let me pull together what both parts of this series added up to.

Picking by use case

Use caseChoice
Cache only (RDB is the source of truth)ElastiCache (no durability)
Cache, but I want data to survive a restartElastiCache Async Durability
Sessions, real-time balances, payment tokens (can't lose data, needs strong consistency)ElastiCache Sync Durability
DR setup, offload reads to a secondary Region (no need for durability within a Region)ElastiCache Global Datastore
Multi-Region active-active, or durability within a Region combined with multi-RegionMemoryDB Multi-Region

What this series covered

  • Internal architecture: the Multi-AZ transactional log does double duty as both persistence and replication, and that's exactly why Sync's "an acknowledged write never disappears" guarantee holds, with the primary's strong consistency preserved across a failover (a replica is eventually consistent, plus sequentially consistent). Async trades that guarantee for lower latency, at the cost of up to 10 seconds of data loss on failure
  • Constraints: durability can only be enabled at creation (no retrofitting, no disabling, only sync/async is switchable later). The service requires CME, Multi-AZ plus a replica, TLS, and a supported instance family. It can't be combined with Global Datastore, Serverless, or data tiering, and the CME requirement means you can't share it through PrivateLink either
  • Application impact: commands don't change, but CME forces your client to become cluster-aware (hash tags for multi-key operations), Async needs retry logic for its write rejections, and Sync needs a reworked latency budget. DurabilityLag is your effective RPO to watch, with backpressure kicking in at 10 seconds
  • Cost: on Tokyo r7g.large with 2 nodes, that's Async at $307/mo (no surcharge), Sync at $362/mo (+18%), and MemoryDB Valkey at $379/mo, and ElastiCache's edge grows the longer your commitment (3-year Reserved Nodes or Database Savings Plans)
  • Test results: Sync lost zero acknowledged keys to an FIS power cut (6 runs); Async lost 967-1,346 acknowledged keys across 3 runs, well under what the spec's 10-second ceiling would allow. The Async-to-Sync switch took about 14.6 seconds with no downtime (measured once). Planned failover (test-failover) landed in essentially the same range as unplanned FIS, with no clear difference. MemoryDB's planned failover (failover-shard) was somewhat longer, at 18.3-22.3 seconds, but given the shared underlying engine, that gap needs re-verification rather than a conclusion. On performance, writes on Sync/MemoryDB are about 13-14x slower than no-durability/Async, while read performance turned out not to depend on durability mode at all, once measured properly
Part 1 worked out what Durability actually promises, straight from the official documentation. Part 2 put that promise to the test, hands-on, and looked at where it differs from MemoryDB.
I worked out, in Part 1, that the internal structure behind Sync's "an acknowledged write never disappears" behavior is the Multi-AZ transactional log. In Part 2, I verified that contract's real-world validity by actually injecting an AZ failure and counting how many acknowledged keys disappeared.
The MemoryDB comparison went cleanly on the performance side, measured under the same conditions. On the failover side, though, I noticed a difference: only ElastiCache lets you inject an unplanned failure. That's not a claim that one service is better; it's a finding that the two services simply differ in what's testable.
Now that ElastiCache has durability, you can no longer default to the assumption that "it's fine if cache data disappears." That doesn't mean RDS or Aurora become unnecessary, though. As Part 1 covered, what Durability changes is the guarantee against immediate loss; it isn't a substitute for query capability, long-term retention, or analytics use cases. One more reason to revisit how you divide responsibilities with your existing RDB — that's my conclusion across this two-part series.

Series: ElastiCache for Valkey Just Became a Durable Data Store (2 articles)

  1. 2
    ElastiCache for Valkey Just Became a Durable Data Store. Here's What That Actually Means (Part 2: Benchmarks and Fault Injection Testing) This article
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article