It was Friday night. I’d just pushed a new feature for my side project and realized the database schema was missing a column. The fix seemed simple: add a last_login timestamp to the users table. I could do it in five minutes with a single SQL command.
So I opened my MySQL client, connected directly to the production RDS instance, and typed:
ALTER TABLE users ADD COLUMN last_login DATETIME;
I hit Enter. The command started running. And then… nothing.
Thirty seconds passed. Then a minute. The table was locked. Every query against the users table started piling up behind my ALTER TABLE . My app’s API calls began timing out. Users couldn’t log in. I had brought down my own production database with a one-line command — and I had no backup.
The panic was instant. I tried to kill the query with KILL QUERY <id> , but MySQL was already too deep into the table copy. The connection dropped, and when I reconnected, the table was corrupted. The users table — the core of my entire app — was now unreadable.
I had two options: restore from a snapshot, or start crying. I did both.
The only snapshot I had was from a week ago, because I had disabled automated backups to save a few dollars on RDS. That meant losing seven days of user signups, password resets, and activity logs. I ended up restoring the old snapshot, manually re-importing the missing rows from a CSV I’d exported two days earlier (thank God for that), and rebuilding the rest by hand. It took the entire weekend.
The worst part? The ALTER TABLE would have taken ten seconds if I’d done it during a maintenance window with a proper backup. Or if I’d used pt-online-schema-change to avoid locking the table. Or if I’d tested it on a staging database first. But I was in a hurry, and I thought “it’s just adding a column, how bad could it be?”
Now, before I touch production data, I follow three non-negotiable rules:
- Take a manual RDS snapshot. Even if automated backups are enabled, a fresh snapshot takes two minutes and gives me a rollback point with zero data loss.
- Test the exact command on a cloned database. RDS makes it easy to spin up a temporary instance from a snapshot. I run the migration there first — every time.
- Use online schema change tools for big tables. For MySQL, that means
pt-online-schema-change or gh-ost . They copy the table in chunks without locking the original, so your app stays up.
That Friday night cost me two days of sleep and a bruised ego. But it taught me a lesson I’ll never forget: a backup isn’t just a safety net — it’s the difference between a five-minute fix and a weekend of firefighting.
Your future self will thank you for taking the snapshot. Take it before you need it.
Loading article