How many AZs you need to deploy to in Prod Environment?
Two AZs look resilient—until they're not. A single AZ failure can split votes evenly, halting writes cluster-wide. Three AZs prevent this, cut costs 25%, and limit blast radius. Here's the math that proves it.
A database cluster runs across two Availability Zones. One AZ experiences a network partition. The cluster needs a majority vote to accept writes, but with nodes split evenly across two AZs, the surviving AZ holds exactly half the nodes -- not a majority. The cluster refuses writes. Both AZs are technically available, but the system is down. Deploying to two AZs created the illusion of redundancy while actually introducing a single point of failure that did not exist with one AZ.
This failure mode is not theoretical. It applies to every quorum-based system deployed on AWS: etcd in EKS clusters, Aurora storage nodes, DynamoDB partitions, OpenSearch master election, and Kafka controller quorum in MSK. The fix is straightforward -- deploy across three Availability Zones -- and every AWS region provides a minimum of three AZs, so this option is always available. Understanding why three is the minimum requires examining how quorum works, what the probability math looks like, and where AWS services implicitly depend on three-AZ deployments.
The Two-AZ Trap
Consider a production deployment spanning two Availability Zones, AZ-A and AZ-B. An application runs behind an Application Load Balancer with an Auto Scaling Group distributing EC2 instances evenly: three instances in AZ-A, three in AZ-B. The database is an RDS Multi-AZ instance with the primary in AZ-A and the standby in AZ-B. On the surface, this looks resilient -- losing either AZ leaves the other to handle traffic.
The problems surface during an actual AZ failure.
Quorum loss in self-managed services. The customer's own quorum-based systems -- self-managed databases, service meshes, coordination services -- lose quorum when one of two AZs fails. Meanwhile, AWS managed services like Aurora, DynamoDB, and EKS run quorum internally across three AZs and survive the same failure. The result is an architecture where the managed infrastructure stays healthy but the customer's workload on top of it goes down -- a mismatch that three-AZ deployment eliminates.
Capacity cliff. When AZ-A fails in a two-AZ deployment, AZ-B absorbs 100% of the traffic increase -- a doubling of load. Auto Scaling Groups will attempt to launch replacement instances, but this depends on available capacity in AZ-B, instance launch rate limits, and health check warm-up time. During this window, the surviving AZ operates at or beyond its capacity. With three AZs, losing one means each surviving AZ absorbs only a 50% increase -- a fundamentally different failure trajectory.
Deployment blast radius. AWS often performs phased rollouts aligned to AZ boundaries. A faulty deployment in a two-AZ setup exposes 50% of infrastructure before detection. With three AZs, the blast radius drops to 33%, providing more time to detect and roll back before customer impact spreads.
Static stability cost. The AWS Builders' Library defines static stability as the ability of a system to continue operating correctly even when a dependency fails. To achieve static stability across two AZs, each AZ must independently carry the full production load -- requiring 200% of baseline capacity (100% provisioned in each AZ). With three AZs, the requirement drops to 150% (50% in each AZ), because losing any one AZ still leaves two AZs holding the full load between them.
The most dangerous aspect of the two-AZ architecture is that it appears to provide redundancy during normal operation. The failure only manifests during the exact scenario the redundancy was meant to protect against.
Quorum Fundamentals and the Odd-Number Rule
A quorum is the minimum number of nodes in a distributed system that must agree on an operation before it is committed. In majority-based consensus protocols (Raft, Paxos, ZAB), the quorum size is:
Plain text
Q = floor(N/2) + 1
where N is the total number of voting nodes. This guarantees that any two quorums overlap by at least one node, preventing conflicting decisions (split-brain).
The formula produces an asymmetry that is critical for AZ planning:
Total Nodes (N) | Quorum (Q) | Failures Tolerated (N - Q) |
|---|---|---|
2 | 2 | 0 |
3 | 2 | 1 |
4 | 3 | 1 |
5 | 3 | 2 |
Two properties stand out. First, N=2 tolerates zero failures -- both nodes must be available, making the system strictly less available than a single node (which has only one possible point of failure instead of two). Second, N=4 provides no fault-tolerance improvement over N=3 -- both tolerate exactly one failure, but N=4 requires an additional node. This is why quorum-based systems universally prefer odd node counts: for a system that must tolerate F failures, the minimum node count is:
Plain text
N = 2F + 1
Adding a node beyond 2F+1 increases cost without increasing fault tolerance until the next odd number is reached.
When nodes are distributed evenly across Availability Zones, the AZ count determines whether an AZ failure can be survived:
- 2 AZs, 2 nodes (1 per AZ): Losing one AZ means losing one node. Quorum requires 2. System is down.
- 2 AZs, 4 nodes (2 per AZ): Losing one AZ means losing 2 nodes. Quorum requires 3. Only 2 remain. System is down.
- 3 AZs, 3 nodes (1 per AZ): Losing one AZ means losing 1 node. Quorum requires 2. Two nodes remain. System operates normally.
No matter how many nodes are placed across two AZs with an even split, losing one AZ always leaves exactly half the nodes -- which is never a majority. With an uneven split (e.g., 3 nodes across 2 AZs: 2 in AZ-A, 1 in AZ-B), the system survives losing AZ-B but not AZ-A. This creates an asymmetric failure domain where one AZ is critical and the other is not -- defeating the purpose of multi-AZ deployment.
Three AZs with one node per AZ is the minimum configuration that symmetrically tolerates the loss of any single AZ.
The Availability Math
Availability of a system is the fraction of time it operates correctly. For individual components, availability is typically expressed as a percentage -- "three nines" (99.9%) means approximately 8.76 hours of downtime per year.
Availability | Common Name | Annual Downtime |
|---|---|---|
99% | Two nines | 3.65 days |
99.9% | Three nines | 8.76 hours |
99.99% | Four nines | 52.6 minutes |
99.999% | Five nines | 5.26 minutes |
99.9999% | Six nines | 31.5 seconds |
When multiple components are arranged in series (each must work for the system to function), their availabilities multiply:
Plain text
A_series = A₁ × A₂ × ... × Aₙ
Three services in series, each at 99.9%, yield a composite availability of 0.999³ = 99.7% -- roughly 26 hours of downtime per year.
When components are arranged in parallel (any one can serve the request), unavailabilities multiply:
Plain text
A_parallel = 1 - (1 - A₁) × (1 - A₂) × ... × (1 - Aₙ)
Two parallel services at 99.9% each yield 1 - (0.001)² = 99.9999% -- about 31.5 seconds of downtime per year.
Where Quorum Changes Everything
Quorum-based systems are neither pure series nor pure parallel. They require a specific minimum number of nodes to be available -- a k-of-N system. The availability of a k-of-N system follows the binomial distribution:
Plain text
A(k-of-N) = Σ C(N, i) × pⁱ × (1-p)^(N-i) for i = k to N
where p is the per-node availability, C(N,i) is the binomial coefficient, and the sum runs over all states where at least k nodes are operational.
The closed-form expressions for the relevant configurations:
2-of-2 (two AZs, quorum of two):
Plain text
A = p²
2-of-3 (three AZs, quorum of two):
Plain text
A = 3p² - 2p³
3-of-5 (five AZs, quorum of three):
Plain text
A = 10p³ - 15p⁴ + 6p⁵
Substituting concrete values reveals the magnitude of the difference:
Configuration | Formula | p = 99% | p = 99.9% | p = 99.99% |
|---|---|---|---|---|
Single node | p | 99% | 99.9% | 99.99% |
2-of-2 | p² | 98.01% | 99.8% | 99.98% |
2-of-3 | 3p² - 2p³ | 99.97% | 99.9997% | 99.999997% |
3-of-5 | 10p³ - 15p⁴ + 6p⁵ | 99.999% | ~99.9999999% | ~100% |
The 2-of-2 configuration is worse than a single node. With nodes at 99.9% individual availability, a single node delivers 8.76 hours of annual downtime. A 2-of-2 quorum delivers 17.5 hours -- double the downtime, because both nodes must be up simultaneously. This is the mathematical reality of a two-AZ quorum deployment.
The 2-of-3 configuration, by contrast, transforms 99.9% per-node availability into 99.9997% system availability: from 8.76 hours of annual downtime to approximately 95 seconds. Adding one node and one AZ produces a ~660x improvement in annual downtime.
Applying This to Availability Zones
The same formulas apply when treating each Availability Zone as a single unit with its own availability. If each AZ has an independent availability of 99.99% (roughly the EC2 Multi-AZ SLA), and a system requires a quorum across AZs:
2-of-2 AZs (two AZs, both must be healthy for quorum):
Plain text
A = (0.9999)² = 0.99980001 = 99.98%
Annual downtime: ~1.75 hours.
2-of-3 AZs (three AZs, any two must be healthy for quorum):
Plain text
A = 3(0.9999)² - 2(0.9999)³ = 0.99999997 ≈ 99.999997%
Annual downtime: ~0.95 seconds.
The probability of two AZs failing simultaneously -- assuming independent failures -- is on the order of 10⁻⁸, which is approximately ten thousand times less likely than a single AZ failure. AWS designs AZs with physical separation (up to 100 km apart), independent power infrastructure, separate flood plains, and isolated network paths specifically to maximize this independence.
Correlated failures (a shared control-plane bug, a regional DNS issue, or a widespread software defect) do occur and reduce the effective independence between AZs. The December 2021 us-east-1 outage, where an internal network congestion event affected all AZs in the region, is a concrete example. Even accounting for correlation, the probability of a multi-AZ simultaneous failure remains orders of magnitude lower than a single-AZ failure, and the quorum math still heavily favors three AZs.
AWS Services That Depend on Quorum
The quorum requirement is not limited to customer-managed clusters. Several foundational AWS managed services use quorum-based replication internally, and their behavior during AZ failures directly impacts any workload that depends on them.
Amazon Aurora
Aurora provides the most transparent quorum model of any AWS managed database. Each Aurora cluster stores six copies of its data across three Availability Zones -- two copies per AZ, organized in Protection Groups of 10 GB segments.
- Write quorum: 4 of 6 copies must acknowledge a write.
- Read quorum: 3 of 6 copies must acknowledge a read.
If an entire AZ fails, two copies are lost but four remain -- meeting the write quorum exactly. Aurora can therefore sustain a full AZ failure without losing write availability. It can further tolerate one additional node failure (AZ loss + 1) without losing read access, since three copies remain.
When an AZ loss is sustained, Aurora automatically repairs by re-replicating the lost copies to the two surviving AZs, restoring to four copies total. During this degraded state, Aurora operates with a reduced quorum: writes require 3 of the 4 remaining copies, and reads require 2 of 4. This quorum reduction is a deliberate design choice -- described in the AWS "Aurora Under the Hood" blog series -- that maintains availability while the cluster heals. Deploying Aurora in a region where only two AZs are used for the application does not change Aurora's internal replication -- it still places copies across three AZs -- but it does mean the application tier may fail to connect to Aurora even when the database itself is healthy.
Amazon DynamoDB
DynamoDB replicates each partition as three replicas across three Availability Zones: one leader and two followers. The replication group uses Multi-Paxos for leader election and write consensus. Writes are acknowledged after a quorum of replicas confirms the write. Strong consistency reads are served from the leader.
If one AZ fails, the leader (if located in the failed AZ) is replaced via Multi-Paxos election among the two surviving replicas. Because two of three replicas remain, quorum is maintained and the table continues to serve reads and writes. This design is baked into DynamoDB's architecture and is the foundation of its 99.999% SLA for Global Tables and 99.99% SLA for standard tables.
Amazon EKS Control Plane
The EKS managed control plane runs at least two API server instances and three etcd instances distributed across three Availability Zones. etcd uses the Raft consensus protocol, which requires a majority quorum (2 of 3) for writes.
When one AZ fails, the two surviving etcd nodes maintain quorum and the Kubernetes API server continues to process requests. If the control plane were limited to two AZs (which is not possible since AWS manages this), losing one AZ would drop etcd to a single node -- below quorum -- and the entire cluster would become read-only or unresponsive.
This is why AWS unconditionally provisions three etcd nodes across three AZs, regardless of how many AZs the customer chooses for their worker nodes. The control plane silently depends on three-AZ availability. Deploying worker nodes across only two AZs creates a mismatch: the control plane can survive any single-AZ failure, but the data plane cannot.
Amazon OpenSearch Service
OpenSearch uses dedicated master nodes for cluster coordination and master election. AWS recommends three dedicated master nodes distributed across three Availability Zones. The quorum formula for master election is:
Plain text
Q = floor(master_nodes / 2) + 1
With three master nodes, quorum is 2. If one AZ fails, the two surviving master nodes can elect a new master. With only two master nodes across two AZs, quorum is 2 -- both nodes must be available, and any AZ failure causes quorum loss and cluster lockup.
OpenSearch's Multi-AZ with Standby deployment option -- the recommended production configuration, offering a 99.99% SLA -- requires three Availability Zones. It is not available for two-AZ deployments.
Amazon MSK (Managed Streaming for Apache Kafka)
MSK distributes Kafka brokers across Availability Zones. Apache Kafka's controller quorum (whether using ZooKeeper or the newer KRaft protocol) is a majority-vote system. MSK Express brokers require three Availability Zones -- this is a hard constraint, not a recommendation. Standard brokers default to three AZs and support a minimum of two, but the three-AZ default exists precisely because Kafka's replication and controller election assume an odd number of failure domains.
Amazon RDS Multi-AZ DB Cluster
The newer RDS Multi-AZ DB Cluster deployment (distinct from the classic Multi-AZ instance) places one writer and two readers across three separate Availability Zones. Transaction commits use a quorum mechanism: a write is acknowledged when at least one of the two standby instances confirms the transaction log write to local storage. This provides RPO=0 (zero data loss) and typical failover in under 35 seconds. The three-AZ distribution is integral to this quorum guarantee -- it cannot function across only two AZs.
Other Common Services That Depend on Quorum
The quorum problem extends well beyond AWS managed services. Any distributed system deployed on EC2, ECS, or EKS that uses consensus-based replication or leader election is subject to the same two-AZ failure mode. Common examples:
- etcd -- Raft consensus; standalone clusters for service discovery or Kubernetes state
- HashiCorp Consul -- Raft consensus for service discovery and health checking
- HashiCorp Vault -- Raft-based HA storage backend
- MongoDB -- Replica set elections require a majority of voting members
- PostgreSQL with Patroni -- Leader election depends on an external consensus store (etcd, Consul, or ZooKeeper)
- Apache ZooKeeper -- Consensus backbone for Kafka (pre-KRaft), HBase, Solr, ClickHouse, and NiFi
- CockroachDB / TiDB -- Raft-based range replication with configurable replication factor (default 3)
- Redis Sentinel -- Quorum vote for failover; losing the AZ with the majority of Sentinels blocks promotion of a healthy replica
- Elasticsearch (self-managed) -- Master-eligible node quorum for cluster coordination
In every case, deploying across two AZs with an even split of quorum participants provides zero AZ fault tolerance. Three AZs with one participant per AZ is the minimum that survives a single AZ failure.
Capacity, Cost, and Operational Tradeoffs
Static Stability: 150% vs 200%
The strongest cost argument for three AZs is counterintuitive: three AZs require less total compute capacity than two AZs for the same level of resilience.
Static stability means pre-provisioning enough capacity so that losing one AZ does not require any scaling action -- the remaining AZs already have enough capacity to handle the full load. With two AZs, each AZ must independently carry 100% of the load, requiring 200% total capacity. With three AZs, each AZ carries 50% of the load (since two surviving AZs together handle 100%), requiring only 150% total capacity.
For a workload needing 100 compute units at peak:
Configuration | Units per AZ | Total Provisioned | Overhead |
|---|---|---|---|
2 AZs | 100 | 200 | 100% |
3 AZs | 50 | 150 | 50% |
Three AZs save 25% on compute costs compared to two AZs when static stability is required. For large workloads -- hundreds or thousands of instances -- this difference is substantial.
Per-AZ Fixed Costs
Some infrastructure costs scale linearly with the number of AZs. NAT Gateways are the most common example: each AZ typically requires its own NAT Gateway for high availability. At approximately $32.85/month per gateway (us-east-1 pricing) plus $0.045/GB data processing, adding a third AZ adds roughly $33/month in fixed NAT Gateway costs. For most production workloads, this is dwarfed by the compute savings from the 150% vs 200% provisioning difference.
Cross-AZ Data Transfer
Traffic between AZs within the same region incurs a $0.01/GB charge in each direction ($0.02/GB round-trip). With three AZs, a larger fraction of service-to-service traffic crosses AZ boundaries compared to a two-AZ setup. Application Load Balancers do not incur cross-AZ data transfer charges for traffic within the same VPC. Network Load Balancers with cross-zone load balancing enabled do incur $0.01/GB per direction.
For data-transfer-intensive workloads, this cost can be meaningful. Mitigation strategies include AZ-affinity routing (preferring same-AZ communication for read-heavy traffic) and keeping cross-AZ traffic limited to replication and failover paths.
Auto Scaling Group Behavior
ASGs maintain balanced instance counts across enabled AZs. During an AZ failure:
- 2 AZs: The surviving AZ starts at 50% capacity. The ASG launches replacements, but this takes time (instance launch, health check warm-up). During the gap, the system runs at half capacity.
- 3 AZs: The two surviving AZs start at 67% capacity combined. The load increase per AZ is 50% rather than 100%, and the system is far more likely to remain stable while replacements launch.
ASG rebalancing can temporarily exceed max capacity by 10% (or one instance, whichever is greater). For a two-AZ deployment that loses half its capacity, a 10% overshoot margin is insufficient to compensate. For a three-AZ deployment losing a third of its capacity, the math works better.
As of November 2024, EC2 Auto Scaling supports AWS Application Recovery Controller (ARC) zonal shift and zonal autoshift. When a zonal shift is active, the ASG stops scale-in to prevent capacity removal and launches new instances only in healthy AZs. This feature further improves three-AZ recovery behavior by preventing the ASG from prematurely terminating instances during transient AZ issues.
Load Balancer Behavior
Application Load Balancers have cross-zone load balancing enabled by default at the load balancer level (it can be disabled per target group since November 2022) and redistribute traffic automatically when an AZ becomes unhealthy. No cross-AZ data transfer charges apply for ALB traffic within the same VPC. The impact of an AZ failure is proportional to the traffic share of the failed AZ -- 50% for two AZs, 33% for three.
Network Load Balancers have cross-zone load balancing disabled by default. Without cross-zone enabled, each NLB node routes traffic only to targets in its own AZ. If an AZ fails, the NLB node in that AZ has no healthy targets and returns errors until DNS removes the unhealthy AZ's IP address (TTL is 60 seconds). During this convergence window, up to 50% of traffic fails in a two-AZ deployment versus 33% in a three-AZ deployment.
Practical Recommendations
Existing two-AZ deployments should be expanded to three AZs as a priority reliability improvement. The migration path is incremental:
- Extend VPC subnets to a third AZ.
- Update Auto Scaling Groups and target groups to include the third AZ.
- Rebalancing happens automatically for ECS, EKS, and ASG-based workloads.
- For services that already use three AZs internally (Aurora, DynamoDB, EKS control plane), no data-layer changes are needed.
For services with explicit AZ configuration (OpenSearch, ElastiCache, MSK), verify that the cluster spans three AZs with node counts that are multiples of three. OpenSearch domains should be migrated to Multi-AZ with Standby for the 99.99% SLA.
Monitor per-AZ metrics separately. Aggregate metrics hide AZ-specific degradation -- a failure pattern AWS calls "gray failures." Use AZ-aware health checks and consider enabling ARC zonal autoshift for automated AZ evacuation during detected impairments.
References
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article