Stop Fetching Everything to Find the Top N
[Series - Do Less Work in Your Application | 1 of 3] -- Each post in this series covers a different variant of the same root problem: fetching data into the application to do work Amazon DocumentDB can do closer to the source.
Series: Do Less Work in Your Application (3 articles)
- 1Stop Fetching Everything to Find the Top N This article
You walk in Monday morning with coffee in hand, ready for standup in twenty. There's a new ticket waiting in the queue from product:
need widget asap on category page...show top 5 best selling products for current week. sholud update daily. thx
You read it twice (once as a developer, the other as a PM translator) and think "not bad - I've seen ranking logic before, probably wrote something similar a few sprints back". The shape of the solution is already forming in your head - group by category, sort by sales, slice the top five. It all seems pretty straightforward.
Your documents keep one product's total sales for a given category and week. You've got
category and product_id (unique within each category and week), and units_sold is numeric.Here's how you wire it up:
sales_database = client["sales_analytics"]weekly_product_sales = sales_database["weekly_product_sales"]
docs = list(weekly_product_sales.find( {"week": current_week}, {"_id": 0, "category": 1, "product_id": 1, "units_sold": 1}))
products_by_category = {}for product in docs: category = product.pop("category") products_by_category.setdefault(category, []).append(product)
top_5_by_category = { category: sorted( products, key=lambda p: (-p["units_sold"], p["product_id"]) )[:5] for category, products in products_by_category.items()}And great news - this works. Each category gets a list containing up to five products. We're all done here, thanks for reading!
Well, not quite. That code fetches every matching document, groups and sorts them in the application, and keeps at most five per category. With a small dev collection the extra work is hard to notice, but what you've written is a fetch-sort-discard pattern:
- DocumentDB returns one projected result for every qualifying product
- Your application groups and sorts those results, then keeps the top five from each category
- Everything else gets discarded (much like my complete set of 1992 Donruss baseball cards, left at my parents' house after I moved out)
You're not asking DocumentDB for the top five products per category. You're asking for every product that matches the filter, then building the top-five lists yourself. None of that grouping or sorting happens until the full result set has already crossed the network.
Why this becomes a problem
You're paying for work you throw away. Every product that passes your filter travels across the network, gets held in memory, gets sorted...and then most of them get dropped on the floor.
Let's say your filter matches 50,000 products across 20 categories. Your application is going to receive all 50,000 results to keep at most 100 (five per category). The other 49,900 were basically tourist - they showed up, stood in line and took up space, then left without doing anything.
And it gets worse as your data grows. The extra application work scales with the number of matching documents, not the number you actually keep. More products means a bigger network payload, more memory, and more time spent grouping and sorting results you're about to throw away.
This pattern survives because the overhead is invisible with small dev datasets. It shows up later when the collection grows or the query runs more often, and usually long after anyone remembers writing it.
Pick the right tool for the job
DocumentDB can figure out the top results per
category for you with the $topN accumulator . Instead of fetching everything and doing the math yourself:pipeline = [ {"$match": {"week": current_week}}, {"$group": { "_id": "$category", "top5": { "$topN": { "output": {"product_id": "$product_id", "units_sold": "$units_sold"}, "sortBy": {"units_sold": -1, "product_id": 1}, "n": 5 } } }}]
results = list(weekly_product_sales.aggregate(pipeline))top_5_by_category = { result["_id"]: result["top5"] for result in results}DocumentDB still processes all the products that match the week (it's not magic), but
$topN changes what comes back: one grouped document per category , containing at most five products. Instead of 50,000 results crossing the wire, you get 20.>>Side note - the secondary sort on
product_id keeps things deterministic when two products tie on units_sold .What this changes
When categories have more than five matching products (which is basically all of the time in production),
$topN means a lot less data crossing the network and a lot less memory usage in your application. Your code is better than that now. It's in a new tax bracket and that work is below them. DocumentDB gets to take over.This doesn't mean DocumentDB only looks at five products per category - it still examines the full matching set. And it's not a universal "make everything faster" button. But it does mean less data is transferred, less client resources are used, and your grammar-challenged PM gets a widget that loads in milliseconds instead of holding the page hostage. The PM may not know the difference between those two implementations, but your users absolutely do.
A test both approaches against the same environment - 100,000 matching products across 20 categories (5,000 per category). Both produced identical top-five results.
Approach | Median latency | Peak Python allocation |
|---|---|---|
Fetch, group, and sort in the application | 37,982 ms | 45.30 MB |
Group and select with $topN | 135.4 ms | 0.05 MB |
From 38 seconds to 135 milliseconds. From 45 MB to basically nothing. The aggregation returned 20 category documents containing 100 products total instead of materializing all 100,000 in the application.
>>Results are medians from five runs against the same collection and index. Actual results depend on data shape, indexes, cluster configuration, client placement, network conditions, current El Niño forecast, and GTA 6 release date.
Recognizing the pattern
For a small collection or a one-off script (not against production during Cyber Monday on your busiest collection), fetch-sort-discard might be fine. But start to reconsider when:
- The number of matching documents is large relative to the grouped result you actually need
- The query runs repeatedly - on a timer, on every page load, triggered by users
- You're starting to notice the memory, network transfer, or processing time
The signal is simple: your application fetches a big result set, organizes it into groups, and keeps only a few from each one. When you spot that, return the grouped answer from DocumentDB instead of the raw material needed to calculate it.
Check out part 2 of this series where we talk about pulling statistics from your data without pulling all your data.
Join the Community
Also, if you are working with DocumentDB or other NoSQL databases, then come join the AWS NoSQL Launchpad ! This is where developers help each other figure out NoSQL design decisions, and it's filled with AWS NoSQL experts.
People share real scenarios, discuss when to use which service, and work through access pattern design together. Practical discussions about the decisions that matter.
Series: Do Less Work in Your Application (3 articles)
- 1Stop Fetching Everything to Find the Top N This article
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article