How a Wishlist Button Melted Our Database and Why I Moved to AWS WAF
Bots abused a harmless button, filled our database with 470,000 junk rows, and pinned CPU at 99%. Here’s how AWS WAF at the edge fixed it
Hello guys
Quick war story from an e-commerce site I help run. One weekend it started falling over, and the trail led to bots abusing a little “wishlist” button to generate data non-stop. This is how I found it and moved our protection to AWS WAF.
The alert
Saturday morning, a stream of emails:
Health check changed to Unhealthy — HTTP timeout occurred
Site was up, but painfully slow - homepage taking 8–13 seconds. The emails were a symptom, not the disease.
Not the web server
First instinct is always the web server. But it was bored:
1
2
3
load average: 0.93 # on 4 CPUs — idle
memory: 4.9G / 7.6G # fine
swap: 0BRequests were waiting on something, not computing. That points downstream - the database.
The database was on fire
Database CPU:
1
CPUUtilization: 99.7% (sustained 30+ min)
CloudWatch CPUUtilization climbing to 99.7% and staying pinned.
No single slow query - instead, a flood of the same one, dozens running at once. They were all coming from the shop’s “compare/wishlist” feature. Every product page asks: “has this visitor flagged this product?”
The 470,000-row surprise
I looked at the table behind that feature:
1
2
3
Total rows: 470,280
Anonymous: 470,275
Logged-in (legit): 5Five real rows. 470,000 junk rows, all anonymous. Who clicks “add to wishlist” 470k times? Bots. The button was open to anonymous visitors, and every click wrote a row.
Requests to the wishlist/compare endpoint pouring in from IPs all over the world - AR, BO, US, RO, BR, CO… clearly automated.
Worse, the lookup wasn’t indexed for that pattern - it scanned ~34,000 rows on every product view. × hundreds of concurrent views = 99% CPU.
The fix: stop it at the edge with AWS WAF
The insight: bots are anonymous - they don’t carry a login session cookie. Real users do. Our edge is CloudFront → AWS WAF, so a blocked request never reaches the origin or the database. The rule:
Block “wishlist” requests that carry no session cookie.
Verified:
1
2
3
wishlist endpoint, no cookie → 403 (bot blocked)
wishlist endpoint, with cookie → 301 (user allowed)
homepage → 200 (normal)
AWS WAF blocking the bot requests at the edge - 403 before they ever reach the origin
Then I cleaned the table - not with a plain
DELETE (a previous one ran 38 minutes). Backup the good rows → truncate → restore:1
2
3
4
CREATE TABLE backup AS SELECT * FROM the_table WHERE <legit rows>;
TRUNCATE TABLE the_table;
INSERT INTO the_table SELECT * FROM backup;
DROP TABLE backup;TRUNCATE is near-instant regardless of size, because it drops the whole table instead of deleting row by row.The payoff
| Metric | Before | After |
|---|---|---|
| DB CPU | 99.7% | 12.7% |
| Rows scanned / query | 34,582 | 1 |
| Homepage load | 8–13s | ~1.3s |
CPU dropped off a cliff in two minutes. The emails stopped.
The moment the bots were cut off: CPU falls from 99.7% straight down.
Database connections collapsing back to baseline right after.
The full WAF (in Terraform)
Once the fire was out, I built a proper Web ACL - version-controlled, with layered AWS managed rule groups:
| Priority | Rule |
|---|---|
| 1 | AntiDDoS |
| 2 | Amazon IP Reputation |
| 3 | Common Rule Set |
| 4 | Known Bad Inputs |
| 5 | SQLi |
The one I lean on most for bots is the Amazon IP Reputation List (
AWSManagedRulesAmazonIpReputationList). AWS maintains it from their own threat intelligence - IPs tied to bots, scanners, and active malicious activity across the whole AWS network. I don't have to build or update a blocklist myself; if an IP already has a bad reputation, it gets blocked before it touches my origin. Given my bots were coming from IPs all over the world (you can see that in the screenshot above), letting AWS identify the known-bad ones automatically saved me a ton of manual IP-set wrangling.
Over a few days: 1.63M requests seen, 360.5K blocked at the edge. That's traffic that never touched my origin.
Honest thoughts
The wishlist WAF rule was temporary. It stopped the bleeding fast, but the real fix was in the app: stop letting anonymous users write those rows at all. Once that shipped, I removed the WAF rule. WAF bought me time; the app fix made it permanent. Fix the symptom and the cause.
What’s next
- CloudWatch alarms on WAF blocked-request spikes
- A general rate-based rule as a backstop
- Rotating the secret origin header so only edge traffic reaches the origin
Takeaways
- Anonymous + database writes = a trap. Wishlists, votes, likes - bots will abuse them. Use a cookie, not a table.
- 99% CPU isn’t “scale up.” Find what is eating it first.
- WAF is strongest at the edge - blocked traffic never touches your origin.
The health-check emails were just the smoke detector. The fire was a friendly “add to wishlist” button with half a million bot clicks.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article