It was 11:47 AM on a Tuesday. The kind of boring, quiet morning where you feel invincible.
A senior dev asked me to clean up a bloated legacy table that was dragging down our read replicas. "Just DROP and re-create," he said. "We have automated snapshots. We're safe."
I connected to the production RDS instance. SSH tunnel. Bastion host. The whole security theater. I typed the command. My finger hovered over Enter for half a second.
Then I pressed it.
My phone exploded. 500 errors lighting up like a Christmas tree. Microservices screaming into the void. The table was gone.
No sweat, right? That's what RDS automated backups are for.
I clicked over to the AWS console. Navigated to "Automated backups." Felt my soul physically leave my body.
The list was empty.
Completely blank. Not a single snapshot. Just a cold, gray screen staring back at me.
"Backup Retention: Disabled."
I stared at the screen for ten seconds. Mouth open. Waiting for the UI to load. It wasn't loading. The backups were never there.
My brain started racing. I remembered setting the retention to 35 days. I remembered checking the box when we launched. I wasn't crazy.
Turns out, I had copy-pasted the Terraform configuration from a dev instance. That dev instance had backup_retention_period = 0 to save a few bucks. I'd migrated that exact config to production without changing a single number. We were running production for a year on a bare-metal database with zero safety net.
A year. Zero backups.
The next 14 hours were the most disgusting thing I've ever done as an engineer.
I wrote a spaghetti Python script that scraped ELB access logs. I was trying to reverse-engineer the missing rows from API Gateway request logs. I was begging CloudWatch to remember what my users had done. It was a data recovery heist, and I was the worst thief in the world.
We recovered about 60% of the data. The rest? A year of customer analytics. Gone. Poof. Just because I was too lazy to triple-check a single integer in a config file.
The habit that saved my sanity after that:
I don't just "set" backups anymore. I test them.
Every Sunday night, a tiny Lambda function spins up a new RDS instance from the latest snapshot. It runs a test query, verifies the data exists, and shuts it down. If it fails, it blows up my Slack with a "YOUR BACKUP IS BROKEN" message. No excuses. No "I'll check it later."
Also, I have a strict rule now: No DROP TABLE on production. Ever. All schema changes go through a migration tool with a 24-hour waiting period. If I want to drop something, I have to request it, approve it, and then sit and stare at the query for five minutes before executing it.
Here's what I wish someone had told me:
A backup isn't real if you've never restored it. That checkbox in the AWS console? It's just a promise. Promises don't save your job when the CTO asks why the customer dashboard is empty.
Check the retention period. Check the storage. Check the actual snapshot exists.
And for the love of all that is holy, don't run destructive commands on a Tuesday morning when you haven't had your coffee yet.
Loading article