AWS Builder Center

Builder Center

The Terraform Plan Said "No Changes." Then It Deleted the Database.

A clean CI run, a green plan, and one workspace mismatch that let terraform apply delete a production RDS instance. How our pipeline reviewed one environment and executed against another — and the guardrail that now stops it.

The plan was clean.
I remember because I checked it twice — once out of habit, once because I didn't trust it. No changes. Your infrastructure matches your configuration. Green tick in CI, merged, laptop shut. I went to make chai.
Twenty minutes later my phone was buzzing against the kitchen counter.
The alert was blunt: RDS instance prod-db-01 — deleting.
Not failing. Not unreachable. Deleting.
I opened the AWS console with one hand and called my teammate with the other. The instance was already gone from the list. In its place, a state transition I'd only ever seen in test accounts, where nothing matters and nobody cries.
Here's what had actually happened.
Two weeks earlier, someone on the team cleaned up our Terraform config. They removed an RDS resource block they believed was unused — a leftover from an old consolidation. Reasonable call. The commit got reviewed, approved, merged.
But it never got applied to production. The config said the database shouldn't exist. Prod's state file said it did. Nothing had run in between to close that gap.
So production was sitting there with a loaded gun and no one pulling the trigger.
The trigger was our pipeline.
Our plan stage had staging hardcoded in the workflow. Our apply stage read the workspace from a variable that had been switched to prod a few weeks back, during a branch-mapping change nobody thought twice about.
Which meant: every merge to main ran terraform plan against staging — where nothing had changed, because staging was untouched — and then ran terraform apply against production.
The plan was honest. It just wasn't describing the place we were about to change.
For two weeks, every deploy showed a green tick for one environment and silently applied to another. We got lucky for fourteen days. Then we didn't.
We restored from a snapshot in about three hours. The data was fine — we had point-in-time recovery on, which is the only reason this is a blog post and not a very short career.
But the part that kept me up wasn't the restore. It was that nothing in our pipeline ever compared the plan to the apply. We reviewed one thing and executed another, and we'd built the whole thing so confidently that no one had noticed.
The fixes were small:
  • prevent_destroy = true on every stateful resource. Terraform now refuses, loudly, and I have to mean it.
  • deletion_protection = true on RDS in production, set at the AWS level too — not just in config. Config can be wrong. The API can't be talked into it.
  • terraform plan -out=tfplan in CI, and terraform apply tfplan in the apply stage. Same file, same workspace, no re-planning between review and execution.
  • A two-line check at the top of the apply job that fails the build if TF_WORKSPACE doesn't match the workspace the plan was generated in.
That last one is the actual fix. The rest are seatbelts. That one stops the car.
I still run terraform plan twice before I merge. Not because it helps — it doesn't, I've proven that — but because it makes me feel like I'm doing something.
The green tick doesn't mean your infrastructure is fine. It means your infrastructure matches your configuration. Those are two different sentences, and I spent a long night learning the difference.
# terraform# devops# amazon-rds
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article