Fresh Data to a Dashboard, and What It Costs
What a user sees is ingest lag plus read latency. I ramp four time-series stores to ~63k msg/s, measure both hops on one clock, and price each.
Fresh Data to a Dashboard, and What It Costs
A dashboard's job is to show what's happening now. But "fresh" isn't one number: the staleness a user actually sees is how old the newest reading is by the time it lands on screen — the ingest lag plus the read latency. I drove a realistic well-fracturing telemetry load from ~100 to ~63,000 msg/s into four data stores, measured both hops on one clock, and — reading the kernel's cgroup counters — measured what each store burns in CPU and memory to keep its read fast.
Every dashboard is a bet that the number on screen is close enough to reality to act on. Two things break that bet, and they're easy to conflate:
- Ingest lag — how long it takes a reading to travel from the device to being queryable in the store.
- Read latency — how long the query itself takes to run once you ask.
The user sees their sum. A store can land data in milliseconds and still paint a stale panel if the read is slow, because the answer is computed against data that was current when the query started — and then takes
read latency to come back. This post measures both hops for four stores under one load ramp, shows how they add up to the freshness a user experiences, and prices the CPU and memory it takes to keep the read cheap.The pipeline under test
Every reading a device publishes lands on an Amazon MSK topic (
sensors.raw.sim), and three stores consume that topic independently. Devices publish over MQTT to AWS IoT Core , whose rules engine forwards to MSK — if you want the device-registration side of that story, I wrote it up in Four Ways to Register a Device with AWS IoT Core . A load generator produces straight into the same topic, so all three stores see the identical stream as I ramp from ~100 to ~63,000 msg/s:
The only thing added for the experiment is the load generator producing into the one topic all three stores read, over SASL/SCRAM authentication .
The four reads, and when to reach for each
Three stores, but four reads — because TimescaleDB is measured two ways, and the difference between them turns out to matter for freshness:
- TimescaleDB-raw — a plain
SELECT avg(...) WHERE ts > now() − windowover the raw hypertable . TimescaleDB is Postgres with time-partitioned tables, so this is just SQL over the full-resolution data. Choose it when you need ad-hoc queries over raw history and your write rate is modest — it's the least special-cased, most flexible option, and it's still Postgres. - TimescaleDB-CAGG — the same store read through a continuous aggregate : pre-materialized buckets with the un-materialized hot tail folded in live. Choose it when you're already on Postgres/TimescaleDB and want fast rollups without standing up a second system — you get most of the pre-aggregation win with a one-line
CREATE MATERIALIZED VIEW. - RisingWave — a streaming database whose materialized view is kept continuously current as each row arrives, so a fleet aggregate is always pre-computed. Choose it when reads are frequent and write volume is high: it trades a steady, bounded CPU/memory cost for read latency that stays flat as load grows.
- InfluxDB (here, Amazon Timestream for InfluxDB ) — a purpose-built time-series database read over a downsampling task that pre-reduces the series on a fixed schedule (here,
every: 10s). The read is the cheapest of the four, but because the rollup only refreshes on its timer, the value it serves carries a cadence-bound staleness gap (~5.3 s here) that doesn't shrink under load. Choose it when your workload is natively time-series (metrics, retention policies, downsampling), you want a managed store that indexes by time out of the box, and an aggregate that's a rollup-interval behind is acceptable — or read the raw measurement directly if you need sub-second freshness and can afford the heavier read.
Every read also returns the row-count it aggregated, so "same answer" is backed by "same rows in scope." Under load each store folds the same millions of points into the fleet mean, and the four means agree to rounding.
The freshness a user actually experiences
Here is the whole point of the post in one idea: the freshness a user sees is not the freshness of the store. It's the age of the newest reading by the time the aggregate lands in the browser — and that's the store's ingest lag plus the time its read takes to return. So the number that matters is this one:
| Offered | TimescaleDB-raw | TimescaleDB-CAGG | RisingWave | InfluxDB |
|---|---|---|---|---|
| 114/s | 3547 | 2714 | 4528 | 5139 |
| ~9,000/s | 1328 | 296 | 1543 | 5394 |
| ~13,000/s | 818 | 273 | 1752 | 5071 |
| ~27,000/s | 977 | 448 | 1497 | 5427 |
| ~35,000/s | 1865 | 610 | 1558 | 5404 |
| ~63,000/s | 5326 | 1210 | 1941 | 5426 |
Effective age of the displayed aggregate ≈ ingest lag + read latency (ms). Transport and render add a little more on top, equally for all four. The InfluxDB column is the path a real dashboard reads — the 10 s downsampling rollup — whose freshness is a cadence-bound ~0.1–11 s sawtooth (median ~5.4 s), flat regardless of load. (Read raw instead of the rollup and InfluxDB's ingest is sub-second; see Hop 1.)
The ranking is probably not the one you'd guess. At full load the two stalest panels are TimescaleDB-raw (~5.3 s) and the InfluxDB rollup (~5.4 s) — neck and neck, but for opposite reasons:
- TimescaleDB-raw has the freshest raw ingest of any store (~72 ms at 63k, as we'll see) and still shows a ~5.3 s panel, because its slow read swamps everything — it fails on the read.
- InfluxDB is never the freshest, despite also having a freshest-of-all raw ingest. The dashboard doesn't read raw; it reads a rollup that only refreshes every 10 s, so the displayed value carries a flat ~5.4 s staleness gap that no amount of load closes — it fails on ingest.
Under load the real winner is TimescaleDB-CAGG at ~1.2 s — fresh ingest and a cheap read — with RisingWave (which looks worst on ingest alone) close behind at ~1.9 s. To see why the order comes out this way, pull the number apart into its two independent hops: ingest lag (how stale the newest row is) and query performance (how long the read takes to hand it back).
Hop 1 — ingest lag: how stale the newest row is
This is
now − MAX(ts_ms) for the stalest actively-reporting site, probed with a cheap MAX lookup so it isolates one thing: the time from a reading being generated to it being queryable. It's a property of the ingest path, independent of whatever query you run afterwards.Median ingest lag, one column per store, one row per rung (ms, lower is fresher). The InfluxDB column is its rollup freshness — the age of the newest 10 s-mean point the dashboard actually reads — not its raw ingest:
| Offered | TimescaleDB-raw | TimescaleDB-CAGG | RisingWave | InfluxDB (rollup) |
|---|---|---|---|---|
| 114/s | 2716 | 2702 | 4417 | 5123 |
| ~9,000/s | 211 | 171 | 1431 | 5378 |
| ~13,000/s | 127 | 102 | 1659 | 5056 |
| ~27,000/s | 116 | 90 | 1350 | 5412 |
| ~35,000/s | 75 | 52 | 1462 | 5384 |
| ~63,000/s | 72 | 56 | 1780 | 5411 |
The InfluxDB-rollup column is a median over ~27 dense samples per rung (the sawtooth spans ~0.1–11 s); the others are medians over the sampling window.
Everything is stale at the 114/s trickle — even TimescaleDB-raw sits at ~2.7 s — because the newest reading that exists is bounded by the emit cadence: spread ~114 msg/s over 40 series and any one site may have gone a second or two without publishing. As load climbs, TimescaleDB-raw and its CAGG get fresher — down to ~55–75 ms — because new rows now arrive continuously and, once inserted, are queryable immediately.
InfluxDB is the interesting case here, and it's why the effective ranking flipped. Read raw, InfluxDB's ingest is the freshest of all — sub-second at every rung, down to single-digit ms at 63k (1 s Telegraf flush over event-time timestamps). But a dashboard doesn't read the raw measurement; it reads a downsampling rollup — a Flux task that recomputes a 10 s mean
every: 10s into a separate bucket. That rollup's freshness is not bounded by ingest at all; it's bounded by the task cadence. Between task runs the newest rollup point ages, so its freshness is a ~0.1–11 s sawtooth (median ~5.4 s) — and because the cadence is fixed, that number doesn't move with load: its per-rung median holds ~5.1–5.4 s from the 114/s trickle all the way to 63k (I pin this with ~27 dense samples per rung, since a single reading catches one random phase of the sawtooth). Pre-aggregating on a schedule buys a cheap read (Hop 2) at the cost of a fixed staleness gap on Hop 1. RisingWave's MV avoids exactly this by recomputing per row rather than on a timer — which is why it's continuously current under load where the scheduled rollup is not.RisingWave behaves differently, and it's worst exactly where you'd least expect — at the lowest rate (4417 ms), not the highest. From ~9k/s up it settles onto a flat ~1.3–1.8 s floor set by its streaming emit path (source → MV → checkpoint → queryable), independent of volume. But a reading only becomes visible in the MV after the engine commits the next barrier/checkpoint and advances its watermark — and both of those are driven by the arrival of newer events. Starve the stream at 114/s and there's rarely a fresh event to push the watermark and trigger the next checkpoint promptly, so the newest visible aggregate lags an extra ~2 s on top of the sparse-emit baseline the other stores also pay. Feed it a steady flow and the mechanism keeps moving, dropping it back to its floor. So RW's low-load freshness is an artifact of a continuous engine being run intermittently — the opposite of the raw stores, where a row is queryable the instant it lands. But note the contrast with the InfluxDB rollup: RW's per-row recomputation keeps it at ~1.3–1.8 s under load, while the scheduled rollup sits at ~5.4 s regardless — so above a trickle, RW's continuously-current MV is the fresher pre-aggregate of the two. (RisingWave's own
now() is barrier-bound and lags ~744 ms under load, so its freshness is computed against the TimescaleDB wall clock, not RW's.)Hop 2 — query performance: the cost of asking
The read has to run before the browser can paint, and a read whose cost grows with load makes the answer stale on arrival. This is where the store's data model shows up: a store that pre-pays the aggregate on write answers from an already-computed result, while a raw relational scan re-reads the underlying rows every time you ask.
Median read latency for the dashboard aggregate, one column per store, one row per rung (ms). The InfluxDB column is the rollup read — a
mean() over the pre-reduced 10 s-mean series:| Offered | TimescaleDB-raw | TimescaleDB-CAGG | RisingWave | InfluxDB (rollup) |
|---|---|---|---|---|
| 114/s | 831 | 12 | 111 | 16 |
| ~9,000/s | 1117 | 125 | 112 | 16 |
| ~13,000/s | 691 | 171 | 93 | 15 |
| ~27,000/s | 861 | 358 | 147 | 15 |
| ~35,000/s | 1790 | 558 | 96 | 20 |
| ~63,000/s | 5254 | 1154 | 162 | 15 |
TimescaleDB-raw re-reads the underlying rows every time, so its latency tracks the amount of data in the window — well under a second at low-to-mid load, climbing to 5.25 s at 63k as the 15-minute scan swells to ~3 M rows. The pre-aggregating stores barely move regardless of load, because the aggregate is already computed and the read just selects it: RisingWave's MV holds ~100–160 ms, the InfluxDB rollup ~15 ms, and even the real-time CAGG climbs only to ~1.2 s (it still folds the un-materialized hot tail live). At 63k the raw scan is ~32× slower than the RisingWave MV (5254 ms vs 162 ms). The busier the pipeline, the more a pre-computed read pays back — and the InfluxDB rollup is the cheapest read of all. That's the trade the rollup makes: it wins Hop 2 outright by paying for it with the fixed staleness gap on Hop 1.
Putting the two hops back together
Now the effective-age table at the top explains itself, and it splits into two distinct failure modes that land at nearly the same number for opposite reasons. TimescaleDB-raw had the freshest ingest of all (72 ms) yet shows a ~5.3 s panel, because its slow read swamps everything — it fails on Hop 2. InfluxDB is the mirror image: the freshest raw ingest and the cheapest read of any store, yet its dashboard sits at a flat ~5.4 s — because the value it serves comes off a rollup that only refreshes every 10 s, so it fails on Hop 1. Between those two, the winners are the stores that keep both hops small at once: TimescaleDB-CAGG at ~1.2 s (fresh ingest, cheap-ish read) and RisingWave at ~1.9 s (a modest ingest floor plus a ~160 ms read). RisingWave — which had the stalest ingest number under load (1.8 s) — still comes out fresher end-to-end than either the raw scan or the rollup, because it's the only pre-aggregate kept continuously current and read cheaply. The lesson: effective freshness is the sum of two hops, and a store can blow either one. A slow read makes fresh data stale on arrival; a scheduled pre-aggregate makes a cheap read serve stale data. Pick the store that keeps both small — which, for a fresh dashboard under load, means a continuously-maintained aggregate, not one rebuilt on a timer and not a raw scan.
What it costs to keep the read cheap
A fast read under load isn't free — a store either pre-pays the aggregate on write (steady CPU and memory) or re-pays it on every read (latency that grows with load). This ramp reads each store's kernel cgroup counters directly — CPU-time delta ÷ wall-clock = average cores in use, and working-set memory (charged pages minus inactive file cache) — which is the ground-truth accounting, not
kubectl top's smoothed signal.
Average cores in use (per store; the committer sums its 3 Redpanda Connect replicas):
| Offered | TimescaleDB | RisingWave (compute) | Committer (ingest) |
|---|---|---|---|
| 114/s | 0.24 | 0.41 | 0.02 |
| ~9,000/s | 0.45 | 0.69 | 0.69 |
| ~13,000/s | 0.58 | 0.80 | 0.99 |
| ~27,000/s | 0.97 | 1.00 | 1.88 |
| ~35,000/s | 1.49 | 1.24 | 2.71 |
| ~63,000/s | 2.52 | 1.82 | 4.76 |
Working-set memory (MB):
| Offered | TimescaleDB | RisingWave (compute) | Committer (ingest) |
|---|---|---|---|
| 114/s | 9263 | 11558 | 241 |
| ~9,000/s | 9567 | 11484 | 247 |
| ~13,000/s | 9802 | 11389 | 250 |
| ~27,000/s | 10215 | 11260 | 248 |
| ~35,000/s | 10849 | 11143 | 294 |
| ~63,000/s | 11886 | 11032 | 266 |
Three things stand out:
- The ingest committer is the compute workhorse, not the database — 0.02 → 4.8 cores as it consumes and transforms the stream. The write path stresses the pipeline harder than any read does.
- RisingWave pays a steady, bounded price for its flat read. Its memory is flat (~11 GB across the whole ramp — if anything it drifts down slightly under load) and its compute grows only modestly (0.4 → 1.8 cores). Bounded streaming state is the point: the MV keeps the current aggregate, not the raw history — so throughput costs CPU, not a growing footprint. That flat ~160 ms read is bought with a fixed ~11 GB you can size for up front.
- TimescaleDB grows on both axes — 0.24 → 2.5 cores and 9.3 → 11.9 GB working set — because buffer cache and hot chunks scale with the hypertable it writes and scans. The raw scan's multi-second read is the read-time tail of that same growth.
(InfluxDB is a managed instance with no accessible cgroup, so it's omitted from the resource tables; its read latency is in the table above.)
The takeaway
If you want fresh data on a dashboard, optimize the number the user actually sees: ingest lag + read latency, not either one alone. Two ways to get it wrong sit at opposite ends of the same equation, and in this ramp they land within ~100 ms of each other at full load for completely different reasons. A raw store can have the freshest ingest (72 ms) and still show a ~5.3 s panel, because under load the read is most of the staleness. And a store with the freshest raw ingest and the cheapest read — InfluxDB here — can still show a ~5.4 s panel, because the value it serves comes off a rollup rebuilt only every 10 s. The first fails on the read; the second fails on ingest. The user pays for whichever you neglect.
The reliable way to deliver a fresh panel is to make the read cheap and keep the pre-aggregate continuously current as load climbs. That means pre-paying the aggregate on write, per row — a continuous aggregate or a streaming materialized view — so the query answers from an already-computed result that reflects the newest data. A scheduled rollup (a task that recomputes every N seconds) wins the read but reintroduces staleness on the ingest hop: it's the right call only when an N-second-old answer is acceptable. The continuous options come at a steady, bounded CPU and memory cost you can size for up front — the trade worth making when your reads are frequent and your write rate is high. Read latency and freshness aren't separate scorecards; they add, and the store that continuously pre-pays the aggregate wins where the user actually looks.
Numbers are medians per rung from a single ramp on equal, realistically-sized hardware.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article