Random Sampling Doesn't Need a Full Fetch
[Series - Do Less Work in Your Application | 3 of 3] - Each post in this series covers a different variant of the same root problem: fetching data into the application to do work Amazon DocumentDB can do closer to the source.
Series: Do Less Work in Your Application (3 articles)
- 3Random Sampling Doesn't Need a Full Fetch This article
Two weeks after the top-5 widget ships, a new Slack message comes in and your heart skips a beat - it's from the fraud team. Fraud or security teams reaching out directly can only mean 1 of 2 things - they either need something or you need to start getting your resume ready.
This time, thankfully, it's the former:
hey can we get a random 1% sample of completed orders piped into our review queue? doesn't need to be exact, just roughly 1 in 100. daily is fine. thx
You've done this before. Pull the completed orders, flip a coin on each one, and keep the winners:
import random
order_database = client["order_management"]orders = order_database["orders"]
completed_orders = list(orders.find({"status": "complete"}))sampled_orders = [ order for order in completed_orders if random.random() < 0.01]And just like your top-5 code, this works. Each order gets its own random shot at selection, and you end up with roughly 1% of the total. Ship it.
Except...you just pulled every single completed order to keep one out of a hundred. The other 99% crossed the network, sat in memory, lost their coin flip, and got thrown away. Sound familiar? Same pattern as article 1 - fetch everything, do work, discard most of it.
Why this becomes a problem
You're paying for data you throw away. Every completed order comes across the network and sits in memory just long enough to lose a coin flip and get discarded. At least when you were doing
$sample you actually had use for the document before you threw it away.With 500,000 completed orders, your application pulls all of them to keep about 5,000. The random selection doesn't start until the entire result set is in your application. And as the collection grows, so does the payload, the memory footprint, and the time your application spends receiving data it's about to drop.
Don't substitute order for randomness
Two tempting shortcuts options you may consider:
- Take every hundredth result
- Pick a random starting point and grab one continuous block.
Both feel random...but neither is. Every hundredth order isn't random. It's a pattern, and patterns interact with other patterns in your data.
Think about what the fraud team is actually trying to do. They can't review every order, so they want a random cross-section, one that's just as likely to surface a suspicious order from Tuesday's batch as from Friday's, from the US as from Europe, from a new customer as from a repeat buyer. Randomness is what makes the sample trustworthy. It's what lets them say "we didn't find anything" with confidence, because every order had a fair shot at being reviewed.
Now imagine your results are ordered by time and you take every hundredth one. You might consistently land on orders placed at 2 AM, or skip an entire region's batch because it falls between your intervals. The fraud team thinks they're reviewing a representative slice but they're actually reviewing a pattern that could systematically miss entire categories of fraud. They'd never know what they're not seeing.
To show this isn't theoretical, I built a rigged dataset with 50,000 orders designed to punish pattern-based sampling:
Group type | Count | Region A share |
|---|---|---|
8 standard groups | 2,500 each | ~10% |
2 oversized groups | 10,000 each | ~70% |
The orders are arranged in time order, so the two oversized groups (heavy on
Region A ) sit in specific positions in the sequence. Then I added one more trap by forcing every 20th order across the entire dataset to Region A . So any method that picks "every Nth" lands on that repeating value and over-samples Region A without realizing it.I ran each sampling method 10 times at a 5% rate, then checked whether the sample's region breakdown matched the overall population. A "mismatch" means the sample was statistically skewed, and it over- or under-represented a region compared to what a truly random draw would produce.
Method | Runs where the sample was skewed |
|---|---|
Random selection in the application | 0 of 10 |
Every twentieth order | 10 of 10 |
One contiguous window | 10 of 10 |
$sampleRate | 1 of 10 |
>>Results are medians from five runs against the same collection. Actual results depend on data shape, indexes, cluster configuration, client placement, network conditions, local availability of blinker fluid, and whether the dress is blue and black or white and gold.
Every-20th and contiguous-window sampling produced a skewed sample every. single. time. They landed on the patterns baked into the data and over-represented
Region A . True random selection and $sampleRate didn't because they give each order its own independent chance, regardless of position.>>Note - The one $sampleRate mismatch in 10 runs is within the range of what the statistical test itself flags as noise. No sampling method is immune to occasional variance, but there's a difference between "occasionally unlucky" and "structurally broken."
Let DocumentDB select the sample
sampled_orders = list(orders.find({ "status": "complete", "$sampleRate": 0.01}))Each matching order gets a 1% chance of being returned. DocumentDB performs the selection and your application receives only the winners (not the entire population). The result count varies from run to run and you'll get around 5,000 back from 500,000 matches (not exactly 5,000). But it will be a fresh set every run, which is exactly what the fraud team wants.
What's the impact
So how much does this actually save? Both methods were tested with 100,000 matching orders and a 1% sample rate. With
$sampleRate , the application now receives ~1,000 orders instead of the entire 100,000. Less network, less memory, and faster response. DocumentDB still scans the matching set (the work has to go somewhere, it's not free) but you're no longer paying the transfer and memory tax on 99% of results you were going to throw away.Approach | Orders returned | Latency | Data transferred |
|---|---|---|---|
Fetch every match, then sample in the application | ~983 | 36,878 ms | 10.6 MB |
Select with $sampleRate | ~986 | 366 ms | 110 KB |
From 37 seconds to 366 milliseconds, and 10.6 MB to 110 KB. Both returned roughly the same number of sampled orders but one downloaded the entire collection to get there while the other didn't.
And that "random 1% for the fraud queue" request? It now runs in under a second instead of holding up resources for half a minute. The fraud team doesn't know the difference between those two implementations, but your infrastructure bill does.
Recognizing the pattern
For a small dataset or a one-time script, application-side sampling is fine. But start to reconsider when:
- The number of matching documents dwarfs the sample you actually need
- The query runs repeatedly, such as daily jobs, scheduled pipelines, or user-triggered actions
- Client memory, transfer size, or processing time is becoming noticeable
- You need a random percentage rather than a stable cohort
The signal is the same one we've seen all series - your application fetches a big result set only to discard most of it. When you spot that, return the subset from DocumentDB instead.
Wrapping the series
Three articles with three variations of the same mistake - hauling data across the network to do work DocumentDB was already willing to do for you. Ranking, aggregation, and random sampling may be different problems, but they all follow the same principle: Ask for the answer you need, not the raw material to calculate it.
The patterns are easy to miss because they work (always the first requirement), pass tests, and survive code review - but they just don't scale. And pushing a fix in production is always harder than getting it right up front. By the time you notice, the code's been running for months, other things depend on it, and now it's a project instead of a one-line change.
Next time you see a query that fetches everything and then filters, sorts, or samples in the application... you'll know what to do.
Join the Community
If you're building on DocumentDB (or any NoSQL), come hang out in the AWS NoSQL Launchpad . Developers helping each other figure out NoSQL design decisions, with AWS NoSQL experts in the mix.
People share real scenarios, discuss when to use which service, and work through access pattern design together. Practical conversations about the decisions that actually matter.
Series: Do Less Work in Your Application (3 articles)
- 3Random Sampling Doesn't Need a Full Fetch This article
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article