Our hackathon leaderboard used one DynamoDB key for every write

At a campus hackathon last spring, my team built a little browser game with a global leaderboard in DynamoDB. I did the database part, and I was proud of how simple it was: one item per score, partition key GAME#1, sort key the score plus the player ID so ties wouldn't overwrite each other.

It worked perfectly with the four of us testing it.

During the demo, a few dozen people were playing, and to make the leaderboard look busy, a teammate also ran a bot script that posted thousands of fake scores per second. Score writes started failing with throttling errors, even though the table's total capacity looked fine. The problem was that every single write went to the same partition key.

DynamoDB spreads data across partitions by partition key, and each partition has its own throughput limits, roughly 1,000 write units and 3,000 read units per second. Adaptive capacity  helps a lot with uneven traffic, but one key value still has a ceiling. Our whole game was one key value.

After the hackathon I rebuilt it with write sharding, which I'd read about but never thought I'd need:

To show the top 10, the backend queries all ten shards in parallel and merges the results. Reads got a little more complicated. Writes spread across ten keys instead of piling onto one, and when I load tested it with a script, the throttling was gone.

The question I ask now before picking a partition key is: at the busiest moment, how many requests will hit the same value? For a leaderboard the honest answer is "all of them." That's a quick question to ask during design and a pretty stressful thing to find out with judges watching.