AWS Builder Center

Builder Center

Running TimescaleDB on EKS with CloudNativePG

Running TimescaleDB on EKS with CloudNativePG

A production pattern for self-hosting TimescaleDB on Amazon EKS with the CloudNativePG operator — the image-tag gotcha, HA, and initdb-vs-upgrade DDL.

Walt

TimescaleDB  is one of the nicest ways to keep time-series data: it's a PostgreSQL extension, so you get hypertables, continuous aggregates, and compression without leaving SQL or giving up joins, indexes, and the rest of the Postgres toolbox. The catch is where you run it. TimescaleDB is not on the list of extensions that Amazon RDS for PostgreSQL supports , so if you want it inside your own VPC — next to the services that write to it, on your own network — you self-manage it. (The hosted alternative is Timescale's own Tiger Cloud , which runs outside your account.)
If your workloads already live on Kubernetes, self-managing Postgres has gotten dramatically less scary thanks to operators. This post walks through the pattern I landed on for a real-time IoT pipeline: TimescaleDB on Amazon EKS , managed by the CloudNativePG  operator. It's a clean, declarative setup — but there are three or four non-obvious traps that will cost you an afternoon each if you hit them cold. I'll call every one of them out.
TimescaleDB on EKS managed by CloudNativePG — operator reconciles a Cluster CR with a primary and replica pod on separate nodes, each backed by an EBS volume

Why an operator, and why CloudNativePG

The hard part of running a database yourself was never "start Postgres." It's failover, replication, backups, failover again when the first one goes sideways, and rolling upgrades. CloudNativePG  (CNPG) is a CNCF Sandbox operator that encodes all of that as a Kubernetes custom resource. You declare a Cluster, and the operator runs the primary, provisions streaming replicas, promotes a replica on primary failure, and manages WAL archiving and backups — the whole PostgreSQL lifecycle .
Here's the key insight that makes TimescaleDB work with it: CNPG provides the HA machinery; the container image provides the Postgres binaries. TimescaleDB is just an extension, so to run it under CNPG you don't need a special "TimescaleDB operator" — you point CNPG at an operand image that is the normal CNPG PostgreSQL base with the TimescaleDB extension binaries baked in. Everything else — failover, replicas, backups — is unchanged.
That one swap is where the first trap lives.

Trap #1: CNPG rejects the official TimescaleDB image tag

The instinct is to reach for Timescale's official timescale/timescaledb-ha image. It won't deploy. CNPG runs an image-tag validator  that expects the tag to encode a PostgreSQL version in a semver-ish form so it can reason about major-version upgrades. The official tags look like pg16-ts2.16 — not a leading semver — and CNPG's admission webhook rejects them.
The fix is to use a community image whose tag is shaped the way CNPG expects, <pg>.<minor>-ts<ext>:
1
imageName: ghcr.io/clevyr/cloudnativepg-timescale:16.4-ts2.16
The clevyr/cloudnativepg-timescale  image is the upstream CNPG PostgreSQL base with the TimescaleDB extension layered on, tagged so the validator passes. Same binaries, CNPG-shaped tag. Pin the exact version — don't float the tag — so a replica rebuild during a failover can never pull a different Postgres or extension minor than the primary is running.

The Cluster manifest, annotated

With the image sorted, the Cluster resource is mostly declarative. Here's the shape of it, with the parts that matter for TimescaleDB specifically:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
name: timescaledb-cloud
spec:
instances: 2 # 1 primary + 1 replica
imageName: ghcr.io/clevyr/cloudnativepg-timescale:16.4-ts2.16

postgresql:
# timescaledb is a preload library: it must load at server start,
# before any connection, or CREATE EXTENSION fails.
shared_preload_libraries:
- timescaledb
parameters:
max_connections: "200"
shared_buffers: "512MB"
effective_cache_size: "1536MB"
maintenance_work_mem: "128MB"
work_mem: "8MB"
wal_level: logical

storage:
size: 50Gi # becomes an EBS volume via the CSI driver

resources:
requests: { cpu: 500m, memory: 1Gi }
limits: { cpu: 2000m, memory: 2Gi }

# Keep primary and replica off the same node so one NodeNotReady
# can't take the primary AND its failover target at once.
affinity:
enablePodAntiAffinity: true
topologyKey: kubernetes.io/hostname
podAntiAffinityType: preferred
A few things worth dwelling on:
  • shared_preload_libraries: [timescaledb] is mandatory, not optional. TimescaleDB hooks the planner and executor, so it has to be loaded at server start — before the first client connects. Leave it out and CREATE EXTENSION timescaledb fails with a message telling you to add it to the preload list and restart. CNPG applies this as a managed Postgres parameter, so it's set correctly on the primary and every replica.
  • storage.size becomes an EBS  volume. CNPG creates a PersistentVolumeClaim per instance; on EKS the EBS CSI driver  fulfills it. Make your default (or an explicit) StorageClass  gp3 — you get configurable baseline IOPS and throughput decoupled from volume size , which matters a lot for a write-heavy append workload. For time-series ingest, disk headroom and IOPS are the real capacity ceiling long before CPU is.
  • Anti-affinity is preferred, not required. You want the primary and its replica on different nodes so a single node failure can't take both. But required would wedge scheduling on a small or single-AZ node group, so preferred is the safe default — spread when you can, don't deadlock when you can't. If you run a multi-AZ node group, consider topology.kubernetes.io/zone as the topology key to spread across AZs.

Trap #2: postInitApplicationSQL runs exactly once, forever

CNPG's bootstrap.initdb block lets you seed the database when the cluster is first created — the natural home for CREATE EXTENSION and your schema:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
bootstrap:
initdb:
database: edge
owner: workshop
postInitApplicationSQL:
- CREATE EXTENSION IF NOT EXISTS timescaledb;
- |
CREATE TABLE IF NOT EXISTS sensor_readings (
ts_ms BIGINT NOT NULL,
sensor TEXT NOT NULL,
site_id TEXT NOT NULL,
value DOUBLE PRECISION,
partition_time TIMESTAMPTZ NOT NULL
);
 - SELECT create_hypertable('sensor_readings', 'partition_time', if_not_exists => TRUE);
# postInitApplicationSQL runs as the superuser — hand the table to the
# app role so it can OWN the hypertable (needed to build a continuous
# aggregate on it later, not just to run DML).
- ALTER TABLE sensor_readings OWNER TO workshop;
This works, and it's the right place for CREATE EXTENSION and the base table. But read the name carefully: it's postInit — it runs once, at initdb, when the data directory is first created. A helm upgrade does not re-run it. Neither does a replica rebuild (replicas clone the primary's data directory; they never run initdb). So any schema you expect to evolve — a trigger you'll tweak, a function you'll fix a bug in — cannot live only here, or clusters bootstrapped before the fix keep the old definition forever.
The pattern that fixes it: put the base, write-once DDL in postInitApplicationSQL, and put any evolving DDL in an idempotent post-upgrade Job that re-applies it on every helm upgrade. In my pipeline the dashboard uses a Postgres pg_notify  trigger as a push channel, and the trigger definition changes over time, so it lives in a Helm hook Job:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
metadata:
annotations:
"helm.sh/hook": post-install,post-upgrade
"helm.sh/hook-weight": "25"
# ...
command:
- sh
- -c
- |
until psql "$TIMESCALE_DSN" -c 'SELECT 1' >/dev/null 2>&1; do sleep 3; done
psql "$TIMESCALE_DSN" <<'SQL'
CREATE OR REPLACE FUNCTION notify_sensor_reading() RETURNS trigger AS $notify$
BEGIN
PERFORM pg_notify('sensor_readings_change', 'batch');
RETURN NULL;
END;
$notify$ LANGUAGE plpgsql;
DROP TRIGGER IF EXISTS sensor_readings_notify ON sensor_readings;
CREATE TRIGGER sensor_readings_notify
AFTER INSERT ON sensor_readings
FOR EACH STATEMENT EXECUTE FUNCTION notify_sensor_reading();
SQL
CREATE OR REPLACE + DROP ... IF EXISTS makes it a no-op when nothing changed and a real update when the definition did. Base DDL in initdb, mutable DDL in a post-upgrade Job — that split is the whole lesson.

Trap #2a: use a named dollar-quote tag in postInitApplicationSQL

While we're here — notice the function body above is wrapped in $notify$ ... $notify$, not the usual bare $$ ... $$. That is not a style choice. CNPG's instance manager hands each postInitApplicationSQL entry to the backend after some string processing that collapses a bare $$ to a single $, turning AS $$ BEGIN ... END; $$ into AS $ BEGIN ... END; $ — a syntax error (SQLSTATE 42601) that aborts initdb and wedges the entire cluster bootstrap. A named dollar-quote tag  survives intact. (In the post-upgrade Job, where the SQL goes straight to psql, bare $$ would be fine — but staying consistent with a named tag costs nothing.)

Trap #3: don't copy the password — consume CNPG's -app Secret directly

CNPG generates the application-role password itself and stores it in a Secret named <cluster>-app. Your first instinct — mine too — is to write a little post-install Job that reads that Secret, assembles a postgres://… DSN, and writes it into a Secret your apps mount. Don't. That Job copies the password into a second Secret (doubling the exposure surface), needs RBAC to create Secrets rather than just read one, and — the real problem — produces a point-in-time snapshot that goes stale the moment the password rotates. Nothing re-runs it, so a rotation silently breaks every consumer.
You don't need any of it, because CNPG already did the work. The <cluster>-app Secret  is type: kubernetes.io/basic-auth but carries more than username/password — it ships ready-made connection strings as additional keys: host, port, dbname, pgpass, a full uri (the postgresql://… DSN), jdbc-uri, and FQDN variants. So a consumer just references the uri key:
1
2
3
4
5
6
env:
- name: TIMESCALE_DSN
valueFrom:
secretKeyRef:
name: timescaledb-cloud-app # the CNPG-generated Secret
key: uri # a complete postgresql:// DSN
No Job, no copy, no second RBAC grant. The uri already points at the -rw Service CNPG publishes (<cluster>-rw), which always tracks the current primary — so you never have to care which pod is primary after a failover; the Service follows the promotion. (There's a matching -ro Service for read-only replicas if you want to send reads there.)
One rotation caveat worth internalising: an environment variable is read once at process start, so a rotated password won't reach a running pod injected this way — it needs a restart. For rotation-aware apps, mount the Secret as a volume instead (not via subPath, which doesn't receive updates) and have the app reload its connection pool on change. CNPG documents a full rotation flow using the External Secrets Operator  that updates the password in place in -app and uses the cnpg.io/reload: "true" label to trigger CNPG to apply the new password — which only works because consumers read the live Secret rather than a copy of it.
If a consumer genuinely can't be changed and demands a fixed Secret name or key you don't control, copying is a last resort — but then rotation becomes your problem to automate, not CNPG's to hand you.

HA and observability, nearly for free

Two more things the operator gives you with one line each.
Failover is automatic once instances: 2 and the anti-affinity above are set — CNPG watches the primary, and on failure promotes the replica and repoints the -rw Service. The anti-affinity is what makes that promotion meaningful: if both pods were on the same node, the event that killed your primary would take the failover target with it.
Metrics: CNPG ships a built-in Prometheus exporter  on port 9187 and will emit its own PodMonitor if you ask it to:
1
2
monitoring:
enablePodMonitor: true
A kube-prometheus-stack  in the cluster discovers it and you get connection counts, replication lag, and the rest without wiring an exporter sidecar yourself. (This needs the Prometheus Operator CRDs installed cluster-wide first, or the PodMonitor has nothing to register against.)

One cluster, more than one database

A small bonus of the operator model: a CNPG Cluster is a Postgres instance, and you can declare additional databases on it with a Database custom resource instead of standing up a second cluster. I use one TimescaleDB instance to also host a second database for a colocated streaming engine's metadata store — same instance, same managed role and credentials, logically separated catalog. If you have a small sidecar service that just needs "a Postgres," this is cheaper than another Cluster.

When not to self-host this

Self-hosting on EKS is the right call when you need TimescaleDB inside your VPC — low-latency, private-network access from in-cluster producers and consumers, data that never leaves your account, and the freedom to tune Postgres parameters directly. The operator makes the day-2 story genuinely manageable.
It is not free of operational weight. You own major-version upgrades, backup verification, capacity planning, and the EBS volumes. If none of your consumers are in-VPC and you'd rather not own any of that, Tiger Cloud  (hosted TimescaleDB) or a managed time-series service like Amazon Timestream for InfluxDB  are legitimate alternatives — different trade-offs, less control, less toil. Pick self-hosting when in-VPC placement or Postgres-level control is the thing you actually need.

Takeaways

  1. CNPG runs the database; the image carries TimescaleDB. You don't need a TimescaleDB-specific operator — just the right operand image under CloudNativePG.
  2. Use a CNPG-tag-shaped image (16.4-ts2.16), not the official pg16-ts… tag the validator rejects. Pin it exactly.
  3. shared_preload_libraries: [timescaledb] or CREATE EXTENSION fails — set it in the Cluster spec so every replica inherits it.
  4. postInitApplicationSQL runs once at initdb. Base DDL there; anything that evolves goes in an idempotent post-upgrade Job. And use a named dollar-quote tag.
  5. Consume CNPG's -app Secret directly — reference its ready-made uri key (which targets the -rw Service) instead of copying the password into your own Secret, so failover and password rotation stay transparent to your apps.
Get those five right and TimescaleDB on EKS is a boringly reliable place to land time-series data — which is exactly what you want from a database.

Related reading

# kubernetes# amazon-eks# postgresql
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article