AWS Builder Center

I Forgot to Set a TTL on DynamoDB and Woke Up to a 50 GB Bill — And a Self-Replicating Monster

A midnight Lambda trigger meant to clean up expired sessions backfired. A single >= instead of > turned a 100 MB dev table into a 50 GB nightmare — and taught me why serverless loops need hard stops.

Last Tuesday I went to bed feeling like a cloud architect. I’d wired up a DynamoDB Stream to a Lambda that would automatically delete expired user sessions. No more manual scripts, no more bloated storage. I even added a tiny safety net that would re-insert any item if it looked like it was deleted by mistake. I closed my laptop and slept like someone who believes in their code.
At 6:43 AM my phone buzzed. AWS Budget Alert: you’ve exceeded 100% of your monthly spend. Cost Explorer showed a vertical line - a 100 MB dev table had swollen to 50 GB overnight, and the charges were already past $200 and climbing.
I opened the DynamoDB console. The table was stuffed with millions of session records, all expired, all identical, all multiplying. My Lambda was running hot, deleting and re-inserting items faster than it could stop. The root cause was a single character in my code: "greater than or equal to" instead of just "greater than".
Here's how the loop happened. The stream triggered on INSERT, MODIFY, and REMOVE events. When an item's TTL timestamp was in the past, the Lambda would delete it. That delete action fired a REMOVE event. My safety net caught that event and checked the old item's TTL. The idea was: if the item was deleted while its TTL was still in the future (meaning some other process accidentally nuked an active session), I'd re-insert it with a fresh ID. The condition should have been "old TTL greater than current time". Instead, I typed "greater than or equal to".
That tiny mistake meant every expired item (whose TTL was definitely in the past) was treated as a wrongful deletion. The Lambda deleted it, caught the REMOVE, saw "old TTL is greater than or equal to current time" as true, and re-inserted the same data with a brand-new ID and a TTL one hour into the future. An hour later that new item expired, and the loop repeated. Every cycle added one permanent zombie to the table. A geometric explosion, running silently while I slept.
By 8 AM I'd traced the loop, disabled the safety net handler, and deployed a fix that stopped processing REMOVE events entirely. Then I spent the afternoon deleting 12 million zombie records. I also finally enabled DynamoDB TTL - the built-in feature I'd completely ignored because I thought my Lambda was smarter. Turns out a free, background deletion process beats a hand-rolled feedback loop every time.

Three lessons I won't forget

  1. Enable DynamoDB TTL for any time-bound data. It's free, efficient, and doesn't rewrite your logic at 2 AM.
  1. Filter stream events ruthlessly. I only needed INSERT events. Listening to REMOVE turned a one-way street into a roundabout of doom.
  1. Test safety nets with deliberately broken data. If a single off-by-one comparison can fill a 50 GB table, your guardrail needs a harder stop - a max retry count, a dead-letter queue, or simply not re-inserting inside the same stream handler.
Now, before I deploy anything that touches DynamoDB Streams, I ask myself: "If this loop runs 10,000 times, what stops it?" If I can't answer in one sentence, the code isn't ready.
The $200 sting faded, but the image of that billing graph climbing while my coffee got cold is burned into my brain. Serverless still demands operational thinking - and sometimes the best solution is the boring one you forgot to turn on.
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article