AWS Aurora Blue/Green Deployment: Understanding Switchover Downtime
Introduction
Amazon Aurora Blue/Green Deployments provide a safer way to deploy database changes by maintaining two separate environments: a blue environment that serves production traffic and a green environment where changes can be tested before production switchover.
At first glance, Blue/Green Deployment may appear to provide a zero-downtime database deployment. However, the switchover itself can still introduce a short period of unavailability.
In this article, we will look at how Aurora Blue/Green Deployment works, what happens during switchover, why downtime can occur, and which factors can affect the duration.
What Is Aurora Blue/Green Deployment?
An Aurora Blue/Green Deployment creates a separate green environment based on the existing blue environment.
The architecture can be visualized as:
Application | v ECS / ALB | v Aurora Writer Endpoint | +----------+----------+ | | v | Blue Environment | Production DB | | | | Replication | v | Green Environment <---------+ New DB VersionThe blue environment continues serving production traffic while changes are prepared and tested in green.
A typical deployment process is:
- Create the green environment.
- Aurora replicates changes from blue to green.
- Deploy the new application/database version to green.
- Run tests and database migrations where supported by the deployment strategy.
- Confirm replication is healthy.
- Perform the switchover.
- Production traffic starts using the green environment.
The important point is that the green environment is not simply another replica inside the same production cluster. Blue/Green Deployment provides a separate environment intended to allow changes to be validated before making them production.
Why Is There Downtime During Switchover?
The most important thing to understand is that Blue/Green Deployment does not mean that the database can switch between environments with absolutely no interruption.
During switchover, Aurora needs to coordinate the transition between the old and new environments.
Conceptually, the process looks like:
Before switchover:
Application | v BLUEProduction traffic | vDatabase
During switchover:
Application | X | [Transition] | v GREEN
After switchover:
Application | v GREENProduction trafficExisting database connections may be terminated or become invalid during this transition.
Applications then need to establish new connections to the new writer.
Therefore, even when the database is highly available, clients can observe a short interruption.
Multi-AZ Does Not Mean Zero-Downtime Deployment
A common misconception is:
"My Aurora database is Multi-AZ, so switching to another database should have zero downtime."
Multi-AZ and Blue/Green solve different problems.
Multi-AZ is primarily an availability mechanism. It allows Aurora to maintain database availability in the event of infrastructure or instance failures.
Blue/Green Deployment is a deployment mechanism. It allows you to prepare a new database environment and then switch production from blue to green.
For example:
Multi-AZ
Aurora Cluster | +---------+---------+ | | AZ-1 AZ-2 Writer ReplicaIf the writer fails, Aurora can perform a failover.
Blue/Green is different:
Blue Environment Green Environment---------------- -----------------Production New version | | +------ Replication -----+ Switchover | v
Green becomes productionHaving Multi-AZ does not eliminate the transition required between two different environments.
What Happens to Existing Connections?
One of the most important sources of perceived downtime is application connection handling.
Suppose an ECS application currently has:
Application | +-- DB Connection 1 --> Blue +-- DB Connection 2 --> Blue +-- DB Connection 3 --> BlueAfter switchover, those connections may no longer be usable.
The application needs to create connections to the new production database.
This can result in errors such as:
connection resetconnection refusedconnection closedtransaction aborteddatabase connection lostThe exact behavior depends on the application, connection pool, driver, retry logic, and how the database endpoint changes are handled.
Why Can RDS Proxy Help?
RDS Proxy can be useful in architectures where database connection management is important.
Instead of applications connecting directly to Aurora:
ECS | +----> Aurorathe architecture becomes:
ECS | vRDS Proxy | vAuroraThe proxy manages database connections and can reduce the impact of database failovers and connection changes.
This does not mean RDS Proxy magically makes Aurora Blue/Green switchover zero-downtime.
However, it can help reduce application-side connection disruption because applications maintain connections to the proxy rather than directly managing connections to the database instances.
Factors That Affect Switchover Downtime
The duration of the interruption is not necessarily constant.
Several factors can influence what users observe.
1. Replication Lag
Before switchover, Aurora needs the green environment to be sufficiently synchronized with blue.
A healthy deployment with very small replication lag is preferable.
For example:
Blue | | replication vGreen
Replication lag: 100 msis generally a much better situation than:
Replication lag: 30 secondsA large replication lag can prevent or delay a safe switchover.
2. Number of Database Connections
An application with many active connections can experience more complicated connection behavior during the transition.
For example:
ECS Task 1 ├── 20 DB connections └── connection pool
ECS Task 2 ├── 20 DB connections └── connection pool
ECS Task 3 ├── 20 DB connections └── connection poolWhen the database changes, these connections need to recover or be recreated.
3. Application Connection Pool Configuration
Connection pool settings can have a significant effect on recovery time.
For example:
pool_size = 20connection_timeout = 10sidle_timeout = ...If the application waits too long before detecting a broken connection, users may experience a longer outage.
4. Retry Logic
Applications without database retry logic may immediately return an error when an existing connection becomes invalid.
With appropriate retry behavior:
Request | vDB connection fails | vRetry | vNew connection | vRequest succeedsthe same database transition can be much less visible to users.
Retries should be implemented carefully, especially around transactions, to avoid accidentally executing non-idempotent operations twice.
5. DNS and Endpoint Resolution
Database endpoints are another important component.
Applications commonly connect using an Aurora endpoint rather than a hardcoded database IP address.
For example:
my-cluster.cluster-xxxx.ap-southeast-1.rds.amazonaws.comDuring a switchover, endpoint resolution and connection establishment need to converge on the new environment.
Applications should therefore avoid caching database IP addresses for unnecessarily long periods.
6. Application Startup and Connection Establishment
After the switchover, the application may need to establish new database connections.
If connection creation is slow, the database may already be ready while the application still appears unavailable.
This is why measuring only the database operation is not enough.
A better measurement is an end-to-end request:
Client | vALB | vECS | vDatabaseFor example:
POST /loginGET /users/meIf both requests succeed before the switchover and fail during the transition, the observed downtime represents what an actual user experiences.
Measuring the Actual Downtime
A simple monitoring script can repeatedly send requests to the application.
For example:
while true; do timestamp=$(date '+%Y-%m-%d %H:%M:%S.%3N')
response=$(curl -sk \ -o /dev/null \ -w "%{http_code}" \ https://api.example.com/health)
echo "$timestamp $response"
sleep 0.1doneYou can then trigger the Aurora switchover and observe the period where requests stop succeeding.
For example:
18:00:00.100 20018:00:00.200 20018:00:00.300 20018:00:00.400 50318:00:00.500 50318:00:00.600 50318:00:00.700 20018:00:00.800 200In this example, the observed application interruption is approximately:
600 msThis measurement is more meaningful than simply looking at the RDS event timestamps because it measures the actual client-visible impact.
A Better Production Test
For production-like testing, monitor several layers simultaneously:
Test Client | v ALB | v ECS | v RDS Proxy | v AuroraMonitor:
- HTTP status code
- request latency
- database connection errors
- ECS application logs
- RDS events
- Aurora replication lag
- RDS Proxy metrics, if used
- CloudWatch metrics
This makes it possible to distinguish between:
Database switchover timeand:
Actual application downtimeThey are not necessarily the same thing.
Database Migrations Require Special Attention
Blue/Green deployments become more complicated when the new application version includes database schema changes.
For example, suppose version 1 contains:
users----idnameemailand version 2 removes:
emailA migration that immediately drops
email can create compatibility problems if the blue application is still running and expects that column.A safer approach is often an expand-and-contract migration.
Step 1 — Expand
Add the new schema without removing the old one.
users----idnameemailphoneStep 2 — Deploy
Deploy application code that can work with both schemas.
Step 3 — Migrate Data
Backfill or transform data as required.
Step 4 — Switchover
Switch production traffic to the new application/database environment.
Step 5 — Contract
Only after the old application is no longer required should obsolete columns or tables be removed.
This approach reduces compatibility problems during deployment.
Recommended Architecture
For an ECS application using Aurora Blue/Green, a robust architecture can look like:
Internet | v ALB | v ECS Service | v RDS Proxy | v Aurora Blue/Green / \ BLUE GREEN Production New VersionThe deployment process becomes:
1. Create Green | v2. Replication | v3. Deploy application | v4. Run compatibility tests | v5. Verify replication | v6. Prepare application for reconnect | v7. Switchover | v8. Verify application | v9. MonitorKey Takeaways
Aurora Blue/Green Deployment significantly reduces the risk of database deployments, but it should not be interpreted as a guarantee of zero client-visible downtime.
The most important points are:
- Blue/Green provides a separate environment for testing and deployment.
- Switchover is still a transition between database environments.
- Existing database connections can be interrupted.
- Multi-AZ improves database availability but does not eliminate Blue/Green switchover interruption.
- Replication lag can affect the readiness of the green environment.
- Connection pools and retry logic strongly influence application recovery time.
- RDS Proxy can help with connection management but does not guarantee zero downtime.
- End-to-end monitoring is the best way to measure real user-visible downtime.
- Expand-and-contract migrations are safer for applications that need backward-compatible schema changes.
In practice, the goal should not simply be to ask whether Aurora Blue/Green has "zero downtime."
A better question is:
How quickly can the application recover from the database connection transition, and how much of that transition is visible to users?
With proper connection management, retry handling, monitoring, and a carefully designed deployment strategy, the impact of the switchover can be reduced significantly.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article