Stop Fetching Everything to Compute a Percentile
[Series - Do Less Work in Your Application | 2 of 3] -- Each post in this series covers a different variant of the same root problem: fetching data into the application to do work Amazon DocumentDB can do closer to the source.
Series: Do Less Work in Your Application (3 articles)
- 2Stop Fetching Everything to Compute a Percentile This article
The top-5 widget has been live for three days and nothing has caught fire, which in this industry counts as a celebration. Then the same PM who asked for it comes back with a follow-up:
top 5 widget is a hit!! next - product page needs a delivery estimate per shipping zone. dont promise the average, it makes us look too good - promise the 95th percentile. still want the typical time shown next to it, plus something that tells me if a zone is all over the place. thx
Setting aside the stray HTML entity, this is a reasonable ask, and the PM is right about the average. Delivery times are right-skewed: most orders ship quickly and a small tail takes much longer. On the data below, nearly 40% of orders are slower than the average. An average hides that tail. A p95 promise ("95% of orders arrive within X hours") doesn't.
Your
orders collection has status , completed_at , shipping_zone , and fulfillment_hours . Three numbers per zone: median, p95, standard deviation. You've written this before:import statisticsimport numpy
order_database = client["order_management"]orders = order_database["orders"]
docs = list(orders.find( {"status": "complete", "completed_at": {"$gte": thirty_days_ago}}, {"_id": 0, "shipping_zone": 1, "fulfillment_hours": 1}))
hours_by_zone = {}for order in docs: hours_by_zone.setdefault(order["shipping_zone"], []).append(order["fulfillment_hours"])
delivery_estimates = { zone: { "typical": statistics.median(hours), "p95": numpy.percentile(hours, 95), "spread": statistics.stdev(hours), } for zone, hours in hours_by_zone.items()}And great news - this works. Every zone gets its three numbers, the widget renders, the PM is happy.
Except look at what just crossed the network. Seven zones, three statistics each. Twenty-one numbers. To produce them, your application downloaded every completed order from the last thirty days. Same shape as the top-5 widget, one step worse: this time you don't even keep a subset. You keep nothing.
Call it fetch-compute-discard.
Why this becomes a problem
Twenty-one numbers is twenty-one numbers whether you have 10,000 completed orders or 10 million. That's the whole point of a summary statistic. But the naive version's cost scales with the input: 100,000 orders means 100,000 documents across the wire, held in a Python list in your application, sorted by
numpy.percentile , then dropped.The widget also refreshes. It's on a product page, so it either runs on every page load or on a timer feeding a cache. Either way you're paying that transfer repeatedly to recompute numbers that barely move between runs.
And the growth is the wrong shape. Every new order makes the calculation more expensive and the answer more stable. You're doing more work over time to produce a number that changes less over time.
Your p95 is three different numbers
Here's the part that has nothing to do with performance.
There's more than one way to compute "the 95th percentile," and Python ships two that disagree by default. Most codebases contain a third one somebody hand-rolled:
-
numpy.percentile(hours, 95)
-
statistics.quantiles(hours, n=100)[94]
-
sorted(hours)[int(0.95 * len(hours))]
I built a fulfillment-time distribution per zone, sized to look like a real fleet, and asked all three for p95 on the identical data:
Zone size | Orders | Distinct p95 answers | Spread |
|---|---|---|---|
2 busiest zones | 7,900 - 8,400 | 1 | 0.0% |
3 mid-size zones | 410 - 3,100 | 1 - 2 | 0.0 - 0.3% |
2 smallest zones | 95 - 180 | 2 - 3 | up to 8.3% |
Your busiest zones agree exactly. The smallest one spans 4.6 hours, an 8.3% spread on the promise you're printing on the product page. And
apac-south , at 180 orders, came back with 54.2, 56.3 and 56.4 - three different numbers for the same data.That pattern is why this survives. The biggest zones agree and the smallest ones don't, which means the zones you check when you spot-check are the zones that agree. The zones that disagree are the ones nobody opens.
Standard deviation has the same problem.
statistics.stdev and numpy.std compute two different things by default, and neither is wrong - they answer different questions, and both pick for you silently. If your dashboard imports one and your alerting service imports the other, they'll disagree forever, and not because either has a bug.Now multiply that by every service in your stack that needs these numbers. The product page, the internal ops dashboard, the weekly report, the SLA alert. Each one imports its own library, each one picks its own default, and each one is confident. Nobody is wrong, and none of them match.
Let DocumentDB compute the statistics
DocumentDB can produce all three numbers per zone in a single
$group , using $median , $percentile , and $stdDevSamp :pipeline = [ {"$match": {"status": "complete", "completed_at": {"$gte": thirty_days_ago}}}, {"$group": { "_id": "$shipping_zone", "typical": {"$median": {"input": "$fulfillment_hours", "method": "approximate"}}, "p95": {"$percentile": {"input": "$fulfillment_hours", "p": [0.95], "method": "approximate"}}, "spread": {"$stdDevSamp": "$fulfillment_hours"} }}]
results = list(orders.aggregate(pipeline))delivery_estimates = { result["_id"]: { "typical": result["typical"], "p95": result["p95"][0], "spread": result["spread"], } for result in results}Twenty-one numbers come back instead of 100,000 documents. One shape note, because it bites immediately:
$percentile takes p as an array and returns an array, which is what the ["p95"][0] is doing. Ask for p: [0.5, 0.95, 0.99] and all three come back in the same pass. The Developer Guide pages linked above cover the remaining argument shapes and the empty-group behavior.Now, about that
method argument. "approximate" isn't a default you can override, it's the only value these operators accept - exact , continuous and discrete are all rejected. The percentile is estimated rather than computed exactly.Sounds alarming, so I measured what it actually costs you. Against the same seven zones, DocumentDB's p95 matched a number at least one of the three libraries also produced on six of them, and matched all three on three of them.
apac-south was the only zone where it came back with a fourth number nobody else had: 54.1, against their 54.2, 56.3 and 56.4. The approximation doesn't pull in a direction of its own - it lands inside the range your libraries were already spread across. And standard deviation isn't approximate at all: $stdDevSamp matched Python's statistics.stdev to four decimal places on every zone.What you gain isn't approximate either. One definition, evaluated in one place, handed to every consumer.
Note - below a few hundred documents per group, none of this saves you. A 95-order zone doesn't have a p95 worth defending to one decimal place no matter who computes it, and moving the calculation into the database won't turn a thin sample into a solid one.
When you genuinely need an exact percentile, pull the data. Invoice reconciliation, a compliance report, a data scientist poking at a hypothesis - those are all fine reasons to move rows into pandas. This is about the numbers that get computed on a schedule, on every page load, or inside an alert.
What's the impact
I tested both approaches against the same collection and index, 100,000 matching orders across 7 shipping zones, five runs each.
Approach | Documents returned | Median latency | Data transferred | Peak Python allocation |
|---|---|---|---|---|
Fetch every match, compute stats in the application | 100,000 | 2,281 ms | 5.95 MB | 41.4 MB |
Compute with $median / $percentile / $stdDevSamp | 7 | 204 ms | 542 bytes | 10.1 KB |
Eleven times faster, and the transfer falls by four orders of magnitude, from 5.95 MB to 542 bytes, because the answer genuinely is only twenty-one numbers. Peak memory in your application goes from 41 MB to about ten kilobytes.
Results are medians from five runs against the same collection and index. Actual results depend on data shape, indexes, cluster configuration, client placement, network conditions, whether Pluto is a planet this week, and how many people in the room can agree on what "medium rare" means.
Recognizing the pattern
For a small collection or a notebook you're going to close in ten minutes, computing statistics in the application is fine. But start to reconsider when:
- The result is a handful of summary numbers and the input is most of a collection
- The calculation runs repeatedly, on a timer, on page load, or inside an alert
- More than one service needs the same statistic, and they don't share a library
- Anyone has ever asked why two dashboards disagree
The signal is the one from the whole series: your application fetches a large result set and keeps almost none of it. Statistics are the version where it keeps literally none of it. When the answer is smaller than the input by four orders of magnitude, compute the answer where the data already is.
Check out part 3 of this series, where the fraud team asks for a random sample and we look at why "every hundredth document" isn't random.
Join the Community
If you are working with DocumentDB or other NoSQL databases, come join the AWS NoSQL Launchpad ! This is where developers help each other figure out NoSQL design decisions, and it's filled with AWS NoSQL experts.
People share real scenarios, discuss when to use which service, and work through access pattern design together. Practical discussions about the decisions that matter.
Series: Do Less Work in Your Application (3 articles)
- 2Stop Fetching Everything to Compute a Percentile This article
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article