It was 11:47 PM. I was three cups of chai deep, ready to call it a night. All I had to do was clean up a few hundred test images from a dev S3 bucket. I'd done it a hundred times. A quick aws s3 rm , then bed.
The plan: set an environment variable to club-photos-dev , run the delete, and sleep like a responsible engineer. My shell history already had the command waiting. I typed:
export BUCKET=club-photos-prod
aws s3 rm s3://$BUCKET --recursive
I hit Enter without looking. The terminal began to scroll — fast. Key after key flew past. At first I thought, "Wow, that's a lot of test photos." Then I saw it: uploads/2026/summit/group_photo_final_final.jpg. A real photo. An actual user memory.
My stomach hit the floor. I bashed Ctrl+C , but the damage was done. 4,000+ images — every photo our college club's members had ever uploaded — were gone. The bucket was empty. No recycle bin. No version history. Just a blinking cursor that seemed to mock me.
I opened the S3 console, hoping for a miracle. Versioning: Not enabled. I had never turned it on because "it's just a small project." What were the odds of a catastrophic typo at midnight? The universe had just answered.
No backup. No cross-region replication. No soft delete. Just a hollow /uploads/ prefix returning 404 on every image link. I messaged the club president with the understatement of the year: "Small issue with the gallery. Working on it."
I didn't sleep that night. The next morning, I had to tell a dozen angry users their photos were gone forever. We posted an apology. I rebuilt the gallery with placeholder graphics. And I spent the entire weekend implementing guardrails I should have had from day one.
What that night taught me (and what I now do religiously)
- Turn on S3 Versioning everywhere. Even on dev. It costs pocket change and would have let me restore everything in three clicks.
- Use MFA Delete on production buckets. That extra five-second token check would have saved 4,000 files and my dignity.
- Never run destructive CLI commands without a dry-run first. S3 doesn't have a native
--dryrun , but a simple aws s3 ls --recursive would have shown me I was about to nuke real data.
- Isolate permissions. My cleanup script now runs under a role that can only touch a specific prefix inside the dev bucket. A typo in the bucket name now gets me an "Access Denied," not an extinction event.
- No risky commands after 10 PM. Tired brains make irreversible mistakes. I now enforce a personal rule: destructive actions get a second pair of eyes or wait until morning.
The aws s3 rm command doesn't ask "Are you sure?" It trusts you completely. And if you hand it a production bucket by mistake, it will faithfully, silently, and instantly delete everything you point at it. I learned that the hard way.
Now every bucket I touch has versioning, MFA delete, and an IAM boundary that would make a security auditor proud. And every time I see s3:// in a terminal, I pause just long enough to remember that night — and triple-check the name. Sometimes the cheapest lessons in the cloud are the most expensive ones.
Loading article