AWS Builder Center

RollbackShield

Keeping the Path Back Alive During Production Deployments

A deployment platform can move your code back to an older release.
That does not automatically mean the system around that code can safely go back with it.
A newer release may already have written data the previous release does not understand, queued work that will execute later, changed infrastructure assumptions, or lost the exact artifact needed for recovery.
That is the problem I built RollbackShield to explore.
RollbackShield is a deployment reversibility control plane built on AWS. It observes a deployment, identifies the candidate release and the previous release, checks whether the previous release is still a viable rollback target, protects that rollback window, coordinates rollback, verifies recovery, and records the evidence behind the decision.
The question it tries to answer is simple:
Can this system still safely return to the previous release right now?



Where the idea came from

The idea started while I was building and operating LoreVia, my SaaS platform for fiction authors.
As the product grew, it stopped being just a frontend and an API. It accumulated persistent data, authentication, background processing, deployments, queues, and application behavior that changed between releases.
That made me think about what “rollback” really means.
My first mental model was:
new deployment has a problem
↓
deploy the old container
↓
done
But that model ignores everything the newer release may have already changed.
The deployment platform might successfully restore the old container while the application still cannot operate correctly against the current production state.
The container rolled back.
The system did not.



A concrete example

To avoid confusing “version 1” and “version 2” with product versions, I use two terms throughout RollbackShield:
  • Previous release — the release we want to be able to return to.
  • Candidate release — the new release currently being rolled out.
Imagine the previous release understands these order states:
CREATED
PAID
CANCELLED
REFUNDED
The candidate release adds a new state:
PARTIALLY_REFUNDED
During rollout, the candidate tries to persist:
Order.status = PARTIALLY_REFUNDED
The candidate understands that value.
The previous release does not.
If production is rolled back after this write, the previous release may encounter state it was never designed to read.
So RollbackShield evaluates the mutation against the active rollback contract:
candidate understands value YES
previous release understands value NO
rollback protection active YES
The result is:
BLOCK
CROSSES_ROLLBACK_HORIZON
The incompatible value never reaches persistence.
A compatible mutation such as:
Order.status = PAID
continues normally.
That is the central idea behind RollbackShield: while rollback protection is active, the candidate release should not be allowed to destroy the ability of the previous release to operate.



Rollback is a compatibility window

Most deployment tooling focuses on the new release:
  • Is it healthy?
  • Did the containers start?
  • Are error rates acceptable?
  • Is the rollout progressing?
RollbackShield adds another question:
Is the previous release still usable?
A deployment is treated as a temporary compatibility window between two generations of the system.
RollbackShield tracks evidence across several dimensions, including:
runtime revision
rollback artifact
application-state compatibility
asynchronous work
rollback policy
previous-version health
supporting infrastructure
It deliberately does not create an arbitrary numerical “rollback safety score.”
A condition is:
supported by evidence
blocked by evidence
unknown
If RollbackShield cannot prove something, it does not silently treat that thing as safe.
That became one of the design principles of the project:
Unknown is information.



The control plane and enforcement plane

One of the most important architectural decisions was keeping RollbackShield out of the latency-sensitive path of normal application writes.
A naive design would look like this:
Application
↓
RollbackShield API
↓
Network request
↓
Compatibility decision
↓
Database
That would make every protected mutation dependent on control-plane latency and availability.
Instead, RollbackShield separates responsibilities into two planes.

Control plane

The AWS-hosted control plane manages:
organizations
integrations
services
deployment observations
releases
rollback contracts
release epochs
reversibility evidence
audit history
rollback orchestration

Enforcement plane

A Java SDK runs inside the protected application.
Conceptually:
Application
↓
RollbackShield Java SDK
↓
Locally cached rollback policy
↓
Deterministic evaluator
↓
Database
The SDK fetches the rollback policy from the control plane, caches it locally, and evaluates protected mutations in-process.
There is no network round-trip to RollbackShield for every business write.
The cloud manages policy lifecycle, infrastructure observation, evidence, and rollback orchestration.
The application makes the latency-sensitive compatibility decision locally.
RollbackShield separates centralized rollback orchestration from latency-sensitive local mutation enforcement.



Fencing asynchronous work

Database state is only one part of rollback safety.
A candidate release can also create work that executes later.
For example:
jobId = job-183
release = candidate-release
releaseEpoch = 42
The work may sit in a queue while the deployment is still running.
If the candidate release is later abandoned, simply restoring the previous container is not enough. The queued job still exists.
RollbackShield uses release epochs to solve this.
When rollback begins:
epoch 42 → INVALID
A worker later receives the old job and asks whether it may execute.
If the epoch is no longer valid:
CANCEL
The stale job may still be physically delivered, but it cannot perform its protected side effect.
This gives rollback a useful property:
An abandoned release should lose the authority to continue changing the restored system.



How AWS fits into RollbackShield


The deployed hackathon version runs almost entirely on AWS.
The request path is:
Browser
↓
AWS Amplify Hosting
↓
Amazon API Gateway
↓
Application Load Balancer
↓
Spring Boot on Amazon ECS Fargate
Authentication uses Amazon Cognito with PKCE.
The control plane uses:
  • Amazon DynamoDB for control-plane state
  • Amazon ECS for runtime discovery, deployment observation, health checks, and rollback execution
  • Amazon ECR for immutable container artifact identity
  • Amazon SQS for asynchronous work and release-epoch fencing
  • Amazon EventBridge for domain-event fan-out
  • Amazon CloudWatch for logs, alarms, health, and operational evidence
  • AWS Secrets Manager for service credentials
  • AWS IAM and STS for temporary role-based access
  • AWS CDK for infrastructure as code
  • AWS SDK for Java v2 for AWS service integrations
  • AWS Amplify Hosting for the production Next.js frontend
  • Amazon API Gateway as the HTTPS API edge
The Spring Boot control plane runs as a long-running ECS Fargate service.
Internally, the backend is a modular monolith rather than a collection of premature microservices. Domains such as releases, contracts, integrations, work fencing, reversibility, audit, identity, and organizations have explicit boundaries while remaining operationally simple.
The deployed RollbackShield architecture: Amplify and Cognito at the frontend, API Gateway and ALB at the API edge, Spring Boot on ECS Fargate, and AWS services for state, messaging, artifacts, observability, and rollback execution.



Connecting to real infrastructure

RollbackShield is not intended to work by asking an operator to manually type two version numbers into a dashboard.
The connector layer discovers actual infrastructure.
In the live AWS environment, RollbackShield discovered:
17 AWS resources
0 discovery errors
Those resources included compute, artifacts, queues, event buses, and log groups.
For the demo workload, the important distinction was:
previous release:
ECS task definition :1

candidate release:
ECS task definition :2
Here,  :1  and  :2  are ECS task-definition revisions for the demo service, not “RollbackShield version 1” and “RollbackShield version 2.”
RollbackShield observed those revisions directly from ECS.
It also resolved the corresponding container artifacts in ECR to immutable SHA-256 digests instead of relying on mutable image tags.
That gives the control plane an exact answer to another important rollback question:
Does the artifact required by the previous release still exist?



Reversibility preflight

Before treating rollback as viable, RollbackShield evaluates the available evidence.
The checks include areas such as:
compute restore path
rollback artifact availability
database compatibility
queued-work fencing
policy freshness
runtime-health observability
rollback execution capability
The result is not a score.
Instead, the control plane can report that rollback is supported, blocked, or unknown based on the available evidence.
One useful example from the live demo is database compatibility.
The demo does not have a migration source connected.
RollbackShield therefore reports that dimension as unknown.
It does not infer “safe” simply because migration evidence is missing.



The live rollback

The most important part of the project for me was verifying the full flow against real AWS infrastructure.
The demo workload was an ECS service called  payments .
RollbackShield observed:
candidate task definition :2
previous task definition :1
It then performed the following flow:
observe deployment
↓
identify previous release
↓
resolve immutable ECR artifacts
↓
run reversibility preflight
↓
activate rollback protection
↓
enqueue work with release epoch
↓
request rollback
↓
invalidate candidate epoch
↓
verify previous artifact
↓
update ECS service to task definition :1
↓
monitor deployment
↓
verify service health
↓
mark release ROLLED_BACK
↓
reject stale candidate work
AWS independently confirmed the restored service state:
task definition :1
running tasks 1
desired tasks 1
health HEALTHY
The stale job created by the abandoned candidate release was later delivered.
Its release epoch had already been invalidated.
The redemption result was:
CANCEL
That was the outcome I wanted from the beginning.
Not merely:
rollback endpoint returned success
but:
previous revision restored
artifact verified
service healthy
candidate work invalidated
stale work cancelled
evidence recorded
Live rollback evidence from the deployed RollbackShield control plane after restoring the ECS service to task definition :1.



Audit evidence

Rollback actions are recorded as an append-only sequence of evidence.
The live flow included events such as:
RELEASE_OBSERVED
RELEASE_PREFLIGHT_EVALUATED
CONTRACT_ACTIVATED
WORK_ENQUEUED
ROLLBACK_REQUESTED
EPOCH_INVALIDATED
ROLLBACK_EXECUTION_STEP
ROLLBACK_COMPLETED
WORK_CANCELLED
That lets RollbackShield explain much more than:
Did a rollback happen?
It can answer:
What deployment was observed?
What was the rollback target?
What artifact represented that target?
Was protection active?
Which epoch was invalidated?
Did ECS restore the expected revision?
Did the restored service become healthy?
What happened to work created by the abandoned release?
The evidence trail is part of the product rather than something assembled afterward from screenshots and logs.



The bugs that only appeared on real AWS

One of the most useful parts of the hackathon was discovering how much a real deployment exposes that local testing does not.

DynamoDB access-pattern bugs

Some entity types shared a GSI partition by organization or service.
Initial repository queries could return multiple item types from the same partition and then attempt to deserialize them as the wrong domain object.
The fix was to make sort-key prefixes part of the access pattern instead of assuming every item in the partition had the same type.

ECS health behavior

A container could report healthy internally while the Application Load Balancer still considered the target unhealthy.
This forced me to debug the actual ALB-to-ECS path rather than assuming container health and load-balancer health were equivalent.

 iam:PassRole 

Real ECS rollback attempts exposed missing  iam:PassRole  permissions for the execution and task roles.
That was not theoretical IAM work: the rollback failed until the permissions were corrected.

Cognito token handling

The backend needs the tenant identity stored in  custom:organization_id .
That claim is carried in the Cognito ID token, so the deployed frontend had to use the correct token type rather than assuming Cognito access and ID tokens were interchangeable.

Synchronous rollback duration

A real rollback can take long enough to exceed an HTTP edge timeout even while the server-side orchestration continues successfully.
The current implementation therefore treats the final release state as authoritative and polls it when necessary.
A future version should move rollback execution fully behind an asynchronous  202 Accepted  workflow.



What I am most proud of

The most important accomplishment is that the core scenario is not just represented in UI state.
It actually runs.
Without rollback protection:
deploy candidate release
↓
candidate writes state the previous release cannot understand
↓
candidate queues delayed work
↓
production problem occurs
↓
restore previous container
↓
previous release encounters incompatible state
↓
candidate work can still execute
With RollbackShield:
observe candidate release
↓
identify previous release
↓
run reversibility checks
↓
activate protection
↓
compatible mutation → ALLOW
↓
incompatible mutation → BLOCK
↓
queue work with release epoch
↓
rollback
↓
invalidate candidate epoch
↓
restore previous ECS revision
↓
verify HEALTHY 1/1
↓
stale candidate work → CANCEL
The live AWS environment proved the second flow end to end.



What I learned

The biggest thing I learned is that rollback is not really a deployment operation.
It is a compatibility window between two generations of a system.
Keeping an older application binary is relatively easy.
Keeping the environment around that binary understandable is much harder.
That environment includes:
database state
queued work
events
configuration
container artifacts
runtime infrastructure
other consumers
So the important question is not:
Do we still have the previous release?
It is:
Does the current system still make sense to the previous release?
That changed how I think about deployment safety.



What comes next

The hackathon version focuses on one complete vertical slice rather than pretending to support every infrastructure combination.

Database evolution protection

RollbackShield already has migration-analysis foundations.
A future release could reason about destructive changes such as:
ALTER TABLE customers
DROP COLUMN billing_address;
If the active rollback target still depends on that column, the deployment could be blocked with an explanation rather than a generic warning.

Historical replay certification

For event-driven systems, a new consumer could be tested against historical events:
7-day history PASS
30-day history PASS
90-day history FAIL

Consumer dependency tracking

Services could advertise the event versions and fields they still depend on.
RollbackShield could answer questions such as:
Can billing_address be removed?

NO

Still required by:
invoice-worker
tax-export

More deployment integrations

RollbackShield should complement deployment platforms rather than replace them.
Future integrations could include:
GitHub Actions
AWS CodePipeline / CodeDeploy
GitLab CI
Jenkins
Argo CD

Reversibility Graph

Long term, I want RollbackShield to model the dependency chain between software generations and the state they produce.
For example:
payments candidate release
↓ emits
OrderUpdated
↓ consumed by
invoice-worker
↓ writes
Invoice
↓ incompatible with
payments previous release
Instead of only reporting:
ROLLBACK UNSAFE
RollbackShield could explain exactly which dependency made rollback unsafe.



Final thought

Deployment tooling is very good at answering:
How do I move this release forward?
and:
How do I redeploy the previous artifact?
RollbackShield is an attempt to answer the harder question in between:
Have we preserved a system that the previous release can still operate?
That is the difference between moving code backwards and making a system genuinely reversible.
Deployment tools can move your code backwards. RollbackShield makes sure your system can actually go back.
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article