Prometheus to Kafka on AWS
The observability layer settles where metrics are stored on AWS: Amazon Managed Service for Prometheus, fed by remote write. This page is about the other reason to move metrics: to put them on a stream — for a Flink anomaly detector, a data-lake landing job, a billing pipeline, or a second vendor — where MSK is the carrier.
The most common tool for that hop is Telefónica's prometheus-kafka-adapter : Apache-2.0, about 400 stars, around 600 lines of Go, last released as 1.9.1 in April 2024, with a dependency bump in June 2026. It receives Prometheus
remote_write, turns every sample into a JSON message and produces it to Kafka. It is small enough to read in one sitting, which is what this page does — and then runs it, because the most important thing about it is not in the README.What it does, in one picture
1
2
3
4
5
6
7
8
Prometheus ──remote_write (snappy protobuf, v1)──▶ adapter :8080/receive
│ decode WriteRequest
│ for each series, for each sample:
│ filter (MATCH) ─▶ topic (template)
│ JSON or Avro-JSON message
▼
librdkafka in-memory queue ──▶ Kafka / MSK
(HTTP 200 is returned here)
One Kafka message per sample, no message key, the full label set in every message:
1
2
{"labels":{"__name__":"go_gc_duration_seconds","instance":"pka:8080","job":"gone","quantile":"0.5"},
"name":"go_gc_duration_seconds","timestamp":"2026-10-01T07:25:38Z","value":"0.000057369"}
1. It acknowledges before Kafka does
The producer is configured with
acks=all — and with go.delivery.reports: false. acks=all controls what the broker waits for before acknowledging; delivery reports are how the client learns of that acknowledgement. With reports off, nobody reads it. The handler calls producer.Produce(), which only places the message on librdkafka's local queue, and returns HTTP 200 to Prometheus as soon as every message in the batch is queued.So the 200 means "in this pod's memory", not "in Kafka". I ran the 1.9.1 image pointed at a broker that does not exist, with Prometheus 3.5 scraping itself and writing to it:
| after 7 minutes with no Kafka broker | value |
|---|---|
| Prometheus prometheus_remote_storage_samples_total | 43,360 |
| Prometheus samples_failed_total / samples_retried_total | 0 / 0 |
| adapter objects_written_total | 43,360 |
| adapter objects_failed_total | 0 |
| adapter /healthz | {"status":"UP"} |
| adapter log lines about expired or lost messages | 0 |
librdkafka gives up on a queued message after
message.timeout.ms — five minutes by default — and reports the failure on the delivery channel the adapter has turned off. Everything in that table says healthy, and the oldest samples have already been dropped. The same happens to whatever is queued when the pod is stopped: there is no signal handler and no Flush(), so every rolling update and every node drain loses the in-flight queue.What Prometheus thinks it guarantees is gone. Remote write's at-least-once behaviour depends on the receiver refusing a batch it cannot persist; this receiver never refuses one for a broker problem. Only when the local queue fills (100,000 messages by default) does
Produce() fail with Local: Queue full — issue #90 — and the 500 finally makes Prometheus retry. By then up to 100,000 samples are queued per pod with nothing durable behind them.objects_written_total is the trap on the dashboard: it counts messages queued, so an alert on it stays green through an MSK outage. Monitor the consumer side instead — topic bytes-in on MSK, or consumer lag — and alert when it falls below what Prometheus says it sent.2. A batch that never succeeds stops remote write
Prometheus retries a 5xx response with backoff and does not move past the batch. The adapter returns 500 for several conditions that are properly permanent:
- a
KAFKA_TOPICtemplate that fails to render for some label set — the topic becomes"", and producing to it fails; - the template's
substringfunction givenstart >= end, which panics; gin's recovery middleware turns that into a 500; Produce()failing for any reason on any one message, after the messages before it in the batch were already queued — so each retry re-sends those as duplicates.
The first two are deterministic: the same batch fails the same way forever, that remote-write shard stops, and Prometheus's write-ahead log keeps growing behind it until its segments are truncated (about two hours), at which point the stuck samples are lost from remote write as well. Keep topic templates trivial, or use a single topic and let consumers route by
name.3. What it does to the data
| property | what you get | consequence |
|---|---|---|
| timestamp | RFC 3339 to the second — milliseconds are truncated (time.Unix(ts/1000, 0)) | 0 of 45,105 messages in the test had a sub-second part; two samples of one series within a second become indistinguishable |
| value | a string ("0.000057369") | every consumer parses floats; the Avro schema is all strings too |
| staleness markers | the special NaN Prometheus writes when a series disappears is formatted as "NaN" | indistinguishable from a real NaN: 2,431 NaN messages in the test, of which 51 were staleness markers — no consumer can tell which |
| native histograms, exemplars, metadata | dropped — the loop reads Samples only | a native histogram simply never appears in Kafka; neither does metric type (issue #102) |
| remote write 2.0 | not supported | leave Prometheus's protobuf_message at the default prometheus.WriteRequest |
| message key | none — PartitionAny | one series' samples spread across partitions; per-series ordering is lost (issues #75, #87) |
| a sample that fails to serialize | logged, then still produced as an empty message | a missing continue in serializers.go |
None of these are exotic. A consumer computing a rate from Kafka has to reorder by timestamp, cannot see staleness, and gets one-second resolution.
4. Which MSK authentication it can use
MSK offers four ways in. The adapter's configuration code supports two of them:
| MSK mode | port | adapter | how |
|---|---|---|---|
| IAM | 9098 | no | needs the MSK IAM SASL signer; the adapter has only PLAIN/SCRAM. MSK Serverless is IAM-only, so it is out entirely |
| SASL/SCRAM | 9096 | yes | KAFKA_SECURITY_PROTOCOL=sasl_ssl, KAFKA_SASL_MECHANISM=SCRAM-SHA-512; the credential lives in a Secrets Manager secret named AmazonMSK_… and encrypted with a customer-managed KMS key |
| mutual TLS | 9094 | yes | client certificate from AWS Private CA; all three of KAFKA_SSL_CLIENT_CERT_FILE, …_KEY_FILE and KAFKA_SSL_CA_CERT_FILE must be set or TLS is silently not configured — point the CA at the image's bundle, /etc/ssl/certs/ca-certificates.crt, because MSK's broker certificates chain to a public CA |
| TLS, unauthenticated | 9094 | no | security.protocol is only applied inside the mTLS and SASL branches (issue #118) |
| plaintext | 9092 | yes | the default; do not, beyond a lab |
With SCRAM, the adapter reads the password from an environment variable, so it reaches the pod as a Kubernetes Secret. Sync it from Secrets Manager with External Secrets under an IRSA role rather than copying it in; the Helm chart in the repository has no SASL settings at all, so this is a manifest you write.
5. What it costs in bytes
The test measured 223 bytes per sample on average (p95 266) as JSON, and compression is off by default (
KAFKA_COMPRESSION=none). The same stream compressed about 19× with zlib; Kafka compresses per batch, so expect less, but an order of magnitude is realistic. For comparison, Prometheus's own storage uses one to two bytes per sample.A worked estimate, before compression and before replication:
| active series | 1,000,000 |
| scrape interval | 30 s → 33,300 samples/s |
| JSON at 223 B/sample | 7.4 MB/s → about 640 GB/day |
| MSK, replication factor 3 | about 1.9 TB/day of broker storage |
| with KAFKA_COMPRESSION=zstd | roughly a tenth of that |
Two settings change that by more than any broker choice:
- Filter in Prometheus, not in the adapter.
write_relabel_configson theremote_writeblock drops series before they leave Prometheus. The adapter'sMATCHfilter runs after the data has crossed the network and been decoded, and only supports exact label equality. - Turn compression on.
zstdorlz4; MSK stores and replicates whatever the producer sends.
A per-metric topic template (
metrics.{{ index . "__name__" }}) looks tidy and creates one topic per metric name — thousands of topics, each with partitions, against MSK's per-broker partition guidance. One topic, keyed consumers.6. Running it on EKS anyway
If the adapter fits — a provisioned MSK cluster, SCRAM or mTLS, consumers that tolerate the data model above, and a loss window you have written down — these are the settings the chart does not give you:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
# Deployment excerpt — the repository's chart pins image 1.8.0 and sets none of this
spec:
replicas: 2
template:
spec:
securityContext: # the image has no USER: it runs as root by default
runAsNonRoot: true
runAsUser: 65534
seccompProfile: { type: RuntimeDefault }
terminationGracePeriodSeconds: 10 # there is no flush to wait for
containers:
- name: adapter
image: telefonica/prometheus-kafka-adapter:1.9.1
env:
- { name: KAFKA_BROKER_LIST, value: "b-1.example.kafka.us-west-2.amazonaws.com:9096,b-2.example.kafka.us-west-2.amazonaws.com:9096" }
- { name: KAFKA_TOPIC, value: metrics }
- { name: KAFKA_COMPRESSION, value: zstd }
- { name: KAFKA_SECURITY_PROTOCOL, value: sasl_ssl }
- { name: KAFKA_SASL_MECHANISM, value: SCRAM-SHA-512 }
- { name: KAFKA_SASL_USERNAME, valueFrom: { secretKeyRef: { name: msk-scram, key: username } } }
- { name: KAFKA_SASL_PASSWORD, valueFrom: { secretKeyRef: { name: msk-scram, key: password } } }
- { name: GIN_MODE, value: release }
ports: [{ containerPort: 8080 }]
readinessProbe: { httpGet: { path: /healthz, port: 8080 } } # says nothing about Kafka
resources:
requests: { cpu: 250m, memory: 256Mi }
limits: { memory: 512Mi } # the librdkafka queue lives here
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities: { drop: ["ALL"] }
And on the Prometheus side, the two settings that matter:
1
2
3
4
5
6
7
8
remote_write:
- url: http://prometheus-kafka-adapter.monitoring.svc:8080/receive
write_relabel_configs:
- source_labels: [__name__]
regex: "(http_server_requests_seconds_count|up|kube_pod_status_ready)"
action: keep
queue_config:
max_shards: 10 # each shard is one concurrent POST to the adapter
The ALB is not involved: the adapter is a ClusterIP Service, and the only thing that should reach it is Prometheus.
7. Or remove it
The alternatives on AWS, by what you actually want:
| you want | use | why |
|---|---|---|
| metrics stored and queryable | Amazon Managed Service for Prometheus, remote write with SigV4 | no Kafka at all; it is what remote write was designed for |
| metrics on a Kafka stream, on MSK | OpenTelemetry Collector (ADOT or contrib): prometheus receiver → kafka exporter | its Kafka exporter supports MSK IAM authentication, so Serverless works and no password exists; OTLP keeps millisecond timestamps, numeric values and histograms; it waits for the broker before acknowledging its own pipeline |
| both | the same Collector with two exporters | one scrape, AMP and MSK |
| metrics landed in S3 for Athena | the Collector's S3 exporter, or MSK → MSK Connect S3 sink | skip the stream if nothing consumes it live |
| to keep this adapter | patch it — below | the code is small |
If you keep it, these are the changes that make it honest, as a sketch rather than a tested patch:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
// handlers.go — wait for Kafka before answering Prometheus
deliveries := make(chan kafka.Event, cap(metrics)) // room for every report
for _, m := range metrics {
if err := producer.Produce(&kafka.Message{TopicPartition: part, Key: seriesKey, Value: m}, deliveries); err != nil {
c.AbortWithStatus(http.StatusServiceUnavailable) // retryable
return
}
}
for range metrics {
if msg := (<-deliveries).(*kafka.Message); msg.TopicPartition.Error != nil {
c.AbortWithStatus(http.StatusServiceUnavailable)
return
}
}
…plus
producer.Flush() on SIGTERM, time.UnixMilli(sample.Timestamp), a numeric value with an explicit stale flag from value.IsStaleNaN, a message key derived from the series labels, continue after a serialization error, and a template failure counted and dropped rather than returned as a 500. Waiting for delivery per request costs latency; raise Prometheus's max_shards to keep throughput.Verified, and not
Run locally on 2026-10-01, with Docker:
telefonica/prometheus-kafka-adapter:1.9.1, prom/prometheus:v3.5.0, and apache/kafka:3.9.1 in KRaft mode. Verified: the 200-with-no-broker behaviour and every number in the first table; the second-resolution timestamps; NaN staleness markers indistinguishable from real NaNs; 223 bytes per sample and the zlib ratio on 45,105 messages; the CA bundle present in the image.Read in the source at commit
18efe53 (June 2026): the producer configuration, the missing flush, the 500 paths, the substring panic, the Samples-only loop, the missing continue, the authentication branches, and the Helm chart's pinned image and missing settings. The wedge in section 2 follows from those paths and Prometheus's documented retry behaviour; it was not reproduced.Not run on AWS. No MSK cluster was created. The authentication table is from the adapter's code and MSK's documented modes; the OpenTelemetry Collector's MSK IAM support should be checked against the Collector version you deploy; the cost estimate is arithmetic on the measured message size, not an MSK bill.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article