AWS Builder Center

I Ran a Database Migration on Production Without a Backup (and Lived to Regret It)

A rushed schema change on a live RDS instance turned a five-minute “quick fix” into a weekend of data recovery. Here’s the exact mistake I made — and the backup habit that now saves me every time.

It was Friday night. I’d just pushed a new feature for my side project and realized the database schema was missing a column. The fix seemed simple: add a  last_login  timestamp to the users table. I could do it in five minutes with a single SQL command.
So I opened my MySQL client, connected directly to the production RDS instance, and typed:
ALTER TABLE users ADD COLUMN last_login DATETIME;
I hit Enter. The command started running. And then… nothing.
Thirty seconds passed. Then a minute. The table was locked. Every query against the  users  table started piling up behind my  ALTER TABLE . My app’s API calls began timing out. Users couldn’t log in. I had brought down my own production database with a one-line command — and I had no backup.
The panic was instant. I tried to kill the query with  KILL QUERY <id> , but MySQL was already too deep into the table copy. The connection dropped, and when I reconnected, the table was corrupted. The  users  table — the core of my entire app — was now unreadable.
I had two options: restore from a snapshot, or start crying. I did both.
The only snapshot I had was from a week ago, because I had disabled automated backups to save a few dollars on RDS. That meant losing seven days of user signups, password resets, and activity logs. I ended up restoring the old snapshot, manually re-importing the missing rows from a CSV I’d exported two days earlier (thank God for that), and rebuilding the rest by hand. It took the entire weekend.
The worst part? The  ALTER TABLE  would have taken ten seconds if I’d done it during a maintenance window with a proper backup. Or if I’d used  pt-online-schema-change  to avoid locking the table. Or if I’d tested it on a staging database first. But I was in a hurry, and I thought “it’s just adding a column, how bad could it be?”
Now, before I touch production data, I follow three non-negotiable rules:
  1. Take a manual RDS snapshot. Even if automated backups are enabled, a fresh snapshot takes two minutes and gives me a rollback point with zero data loss.
  1. Test the exact command on a cloned database. RDS makes it easy to spin up a temporary instance from a snapshot. I run the migration there first — every time.
  1. Use online schema change tools for big tables. For MySQL, that means  pt-online-schema-change  or  gh-ost . They copy the table in chunks without locking the original, so your app stays up.
That Friday night cost me two days of sleep and a bruised ego. But it taught me a lesson I’ll never forget: a backup isn’t just a safety net — it’s the difference between a five-minute fix and a weekend of firefighting.
Your future self will thank you for taking the snapshot. Take it before you need it.
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article