CloudWatch Builder Series - Part 3 : OpenTelemetry
A hands-on guide to native OpenTelemetry ingestion in CloudWatch. Covers collector configurations for EKS, ECS, and Lambda, PromQL query patterns, migration paths from existing setups, cost modeling at scale, and Telemetry Configuration Rules for automatic org-wide enablement.
Series: CloudWatch Builder Series (3 articles)
- 3CloudWatch Builder Series - Part 3 : OpenTelemetry This article
OpenTelemetry Metrics and Telemetry Configuration
CloudWatch now natively ingests metrics via the OpenTelemetry Protocol (OTLP) and exposes them through PromQL. This can be a alternative to the classic PutMetricData/EMF model with a per-GB pricing approach — no per-metric charges, no dimension limits, and 15 months of storage included.
Telemetry Configuration Rules let you auto-enable metric and log collection across all AWS resources in your organization — no manual agent setup per resource.
Capability | Details |
|---|---|
Ingestion protocol | OTLP (OpenTelemetry Protocol) over HTTPS |
Endpoint | https://monitoring.\{region}.amazonaws.com/v1/metrics |
Authentication | AWS SigV4 or bearer token |
Labels per data point | Up to 150 (vs. 30 dimensions in classic CloudWatch) |
Metric types | Gauge, Sum, Histogram |
Storage | 15 months included, no separate charge |
Query language | PromQL |
Pricing | Per-GB ingestion only — no per-metric, no per-API-call charges |
Clients | OTel SDKs (Java, Python, Go, .NET, Node.js), OTel Collector, CloudWatch Agent |
When to use: OTel Metrics vs. PutMetricData vs. EMF
Criteria | OTel Metrics (OTLP) | PutMetricData | Embedded Metric Format (EMF) |
|---|---|---|---|
Pricing model | Per-GB ingestion | $0.30/metric/month (first 10K) | Per-GB log ingestion + metric extraction |
Dimensions/labels | 150 labels | 30 dimensions | 30 dimensions |
Query language | PromQL | CloudWatch Metrics math | CloudWatch Metrics math |
High cardinality | Built for it — no penalty for pod names, request IDs | Expensive — each unique combo = new billable metric | Moderate — limited by dimensions |
Vendor lock-in | None — standard OTLP works with Prometheus, Grafana, Datadog | AWS-specific API | AWS-specific log format |
Storage | 15 months included | Standard retention tiers (3h/1min/5min/1h) | Log retention + metric retention |
Best for | Application metrics, container metrics, high-cardinality | Low-volume infrastructure metrics, alarms on simple counts | Metrics embedded in structured logs |
Migration effort | Swap exporter config | N/A (existing) | N/A (existing) |
Decision guide:
- You have more than 10,000 custom metrics — OTel (per-GB is cheaper than $0.30/metric/month at scale)
- You need more than 30 dimensions — OTel (150 labels, add pod names, request IDs, custom tags freely)
- You use Prometheus/Grafana already — OTel (same collector, same queries, add CloudWatch as destination)
- You have a handful of simple metrics with alarms — PutMetricData (simpler, works fine at low volume)
- You want metrics extracted from logs — EMF (still valid for log-derived metrics)
Architecture

Implementation
What each step does
Step | What it does | Who runs it |
|---|---|---|
1. Enable resource tags on telemetry | Allows CloudWatch to attach resource tags as PromQL labels | Account admin (one-time) |
2. Enable OTel enrichment | Makes existing AWS vended metrics (EC2, RDS, ELB) queryable via PromQL without any agent changes | Account admin (one-time) |
3. Configure collector/agent | Sets up the pipeline to send your custom application metrics to the OTLP endpoint | Platform/DevOps team |
4. Query with PromQL | Validates metrics are flowing and demonstrates query patterns | Developer/SRE |
5. Create PromQL alarms | Replaces classic metric alarms with PromQL-based alarms | SRE/Ops |
Step 1: Enable resource tags on telemetry
One-time setup — allows resource tags (environment, team, cost-center) to appear as PromQL labels on vended metrics.
aws observabilityadmin start-telemetry-enrichmentStep 2: Enable OTel enrichment for vended metrics
This makes your existing AWS service metrics (EC2 CPUUtilization, RDS connections, etc.) queryable via PromQL — enriched with resource ARN and tag labels. No agent changes required.
aws cloudwatch start-otel-enrichmentAfter enabling, you can immediately query vended metrics with PromQL:
# EC2 CPU by instance tag "environment=production"
aws_ec2_cpuutilization{@resource.aws_ec2_instance.tag.environment="production"}Step 3: Configure the collection pipeline
Choose one of the following based on your environment:
Option A — CloudWatch Agent (recommended for most workloads)
Deploy the agent on EC2, ECS task, or as a DaemonSet in EKS. Your app sends metrics to the agent's local OTLP listener (same host/pod), and the agent handles auth + forwarding to CloudWatch.
CloudWatch Agent configuration (
amazon-cloudwatch-agent.json ):{ "metrics": { "metrics_collected": { "otlp": { "grpc_endpoint": "127.0.0.1:4317", "http_endpoint": "127.0.0.1:4318" } } }}Your application sends metrics to the local agent via OTLP (port 4317 for gRPC, 4318 for HTTP). The agent forwards to CloudWatch.
Option B — Custom OTel Collector (full control)
Deploy as a sidecar, DaemonSet, or standalone Deployment in your cluster. You own the full pipeline config — receivers, processors, exporters — and point the exporter to CloudWatch's OTLP endpoint with SigV4 auth.
OTel Collector configuration (
otel-collector-config.yaml ):receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
http:
endpoint: 0.0.0.0:4318
prometheus:
config:
scrape_configs:
- job_name: 'my-app'
scrape_interval: 30s
static_configs:
- targets: ['localhost:9090']
processors:
batch:
timeout: 60s
send_batch_size: 1000
resource:
attributes:
- key: environment
value: production
action: upsert
- key: service.name
value: order-service
action: upsert
exporters:
otlphttp/cloudwatch:
endpoint: https://monitoring.us-east-1.amazonaws.com/v1/metrics
auth:
authenticator: sigv4auth
compression: gzip
extensions:
sigv4auth:
region: us-east-1
service: monitoring
service:
extensions: [sigv4auth]
pipelines:
metrics:
receivers: [otlp, prometheus]
processors: [batch, resource]
exporters: [otlphttp/cloudwatch]Option C — Direct SDK (no collector needed)
Embed directly in your application code. The OTel SDK exports metrics straight to CloudWatch's OTLP endpoint — no agent or collector process to manage. Best for Lambda or simple services where running a sidecar adds unnecessary complexity.
Python example using the OTel SDK:
Required environment variables (set before running your app):
export OTEL_EXPORTER_OTLP_METRICS_ENDPOINT="https://monitoring.us-east-1.amazonaws.com/v1/metrics"export OTEL_EXPORTER_OTLP_METRICS_PROTOCOL="http/protobuf"export OTEL_METRICS_EXPORTER="otlp"export OTEL_SERVICE_NAME="order-service"
IAM permission on the calling identity:
{ "Effect": "Allow", "Action": ["cloudwatch:PutMetricData"], "Resource": "*"}from opentelemetry import metricsfrom opentelemetry.sdk.metrics import MeterProviderfrom opentelemetry.sdk.metrics.export import PeriodicExportingMetricReaderfrom opentelemetry.exporter.otlp.proto.http.metric_exporter import OTLPMetricExporter
# Point directly at the CloudWatch OTLP endpoint.# Authentication is handled automatically via the default AWS credential chain# (environment variables, IAM role, instance profile, etc.)# when using the ADOT Python SDK distribution.exporter = OTLPMetricExporter( endpoint="https://monitoring.us-east-1.amazonaws.com/v1/metrics")
reader = PeriodicExportingMetricReader(exporter, export_interval_millis=60000)provider = MeterProvider(metric_readers=[reader])metrics.set_meter_provider(provider)
# Create and record metricsmeter = metrics.get_meter("order-service")order_counter = meter.create_counter( name="orders.processed", description="Total orders processed", unit="1")latency_histogram = meter.create_histogram( name="orders.latency", description="Order processing latency", unit="ms")
# Use in your applicationorder_counter.add(1, {"region": "us-east-1", "customer_tier": "enterprise", "status": "success"})latency_histogram.record(142.5, {"region": "us-east-1", "endpoint": "/api/v2/orders"})Option D — Managed Prometheus Collector (EKS — zero pods to manage)
For EKS workloads, CloudWatch provides a fully managed collector that auto-discovers and scrapes Prometheus endpoints. AWS runs the collector for you — no DaemonSet, no sidecar, no pods in your cluster. You provide a scrape config and CloudWatch discovers and scrapes your Prometheus endpoints via VPC networking. Best for teams already exposing /metrics endpoints.
aws amp create-scraper \ --alias "eks-metrics-scraper" \ --source eksConfiguration="{clusterArn='arn:aws:eks:us-east-1:123456789012:cluster/my-cluster', \ securityGroupIds=['sg-12345'], \ subnetIds=['subnet-abc123','subnet-def456']}" \ --scrape-configuration configurationBlob=$(cat prometheus-config.yaml | base64 -w 0) \ --destination cloudWatchConfiguration="{datasetArn='arn:aws:cloudwatch:us-east-1:123456789012:dataset/default'}"Prometheus scrape config (encode to base64):
global:
scrape_interval: 30s
scrape_configs:
- job_name: 'kubernetes-pods'
kubernetes_sd_configs:
- role: pod
relabel_configs:
- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
action: keep
regex: true
- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
action: replace
target_label: __metrics_path__
regex: (.+)Step 4: Query with PromQL
PromQL (Prometheus Query Language) is the query language used to retrieve and aggregate OTel metrics in CloudWatch. If you've used Prometheus or Grafana, it's the same syntax —
rate() , histogram_quantile() , avg by() — now running natively in CloudWatch without managing a Prometheus server.Once metrics flow in, query them in CloudWatch Query Studio or via API:
# Request rate by service over 5 minutes
rate(orders_processed_total[5m])
# P99 latency from histogram
histogram_quantile(0.99, rate(orders_latency_bucket[5m]))
# CPU by pod in production namespace
avg by (k8s_pod_name) (
rate(container_cpu_usage_seconds_total{k8s_namespace_name="production"}[5m])
)
# Filter by resource tag (vended metrics with enrichment enabled)
avg(aws_ec2_cpuutilization{@resource.aws_ec2_instance.tag.team="payments"})
# Top 5 highest memory pods
topk(5, container_memory_working_set_bytes{k8s_namespace_name="production"})Label structure for OTLP metrics in CloudWatch:
Prefix | What it contains | Example |
|---|---|---|
@resource. | Resource attributes (service, host, cloud) | @resource.service.name="order-service" |
@scope. | Instrumentation scope | @scope.name="io.opentelemetry.sdk" |
(no prefix) | Datapoint attributes | region="us-east-1" , status="success" |
@resource.aws_* | AWS-enriched attributes (vended metrics) | @resource.aws_ec2_instance.tag.team="payments" |
Step 5: Create PromQL alarms
# Alarm: P99 latency exceeds 500msaws cloudwatch put-metric-alarm \ --alarm-name "OrderLatencyP99High" \ --evaluation-criteria '{ "PromQLCriteria": { "Query": "histogram_quantile(0.99, rate(orders_latency_bucket{service_name=\"order-service\"}[5m])) > 500", "PendingPeriod": 300, "RecoveryPeriod": 600 } }' \ --evaluation-interval 60 \ --alarm-actions arn:aws:sns:us-east-1:123456789012:ops-alerts
# Alarm: Error rate exceeds 5%aws cloudwatch put-metric-alarm \ --alarm-name "OrderErrorRateHigh" \ --evaluation-criteria '{ "PromQLCriteria": { "Query": "rate(orders_processed_total{status=\"error\"}[5m]) / rate(orders_processed_total[5m]) * 100 > 5", "PendingPeriod": 120, "RecoveryPeriod": 300 } }' \ --evaluation-interval 60 \ --alarm-actions arn:aws:sns:us-east-1:123456789012:ops-alerts
Telemetry Configuration Rules (auto-enable at scale)
Telemetry Configuration Rules automatically enable metric and log collection for AWS resources across your organization — no manual per-resource setup.
What it solves
- New VPC created but no flow logs enabled
- EKS cluster running but control plane logging not turned on
- CloudTrail not configured for new accounts in the org
- WAF, ALB, or Route 53 Resolver logging missing on new resources
- OTel vended metric enrichment not enabled in new accounts
- Telemetry gaps discovered only during incidents
How it works
You define rules that specify: which resource types, which telemetry types, which accounts/OUs. CloudWatch discovers matching resources and automatically configures collection.
# Rule 1: Auto-enable VPC Flow Logs for all VPCs in production OUaws observabilityadmin create-telemetry-rule-for-organization \ --rule-name "vpc-flow-logs-prod" \ --rule '{ "ResourceType": "AWS::EC2::VPC", "TelemetryType": "Logs", "TelemetrySourceTypes": ["VPC_FLOW_LOGS"], "DestinationConfiguration": { "DestinationType": "cloud-watch-logs", "RetentionInDays": 30, "VPCFlowLogParameters": { "TrafficType": "ALL", "MaxAggregationInterval": 60 } }, "Scope": "ou-prod-12345", "AllRegions": true }'
# Rule 2: Auto-enable EKS control plane logs for all EKS clustersaws observabilityadmin create-telemetry-rule-for-organization \ --rule-name "eks-logs-all" \ --rule '{ "ResourceType": "AWS::EKS::Cluster", "TelemetryType": "Logs", "TelemetrySourceTypes": ["EKS_AUDIT_LOGS", "EKS_API_LOGS"], "DestinationConfiguration": { "DestinationType": "cloud-watch-logs", "RetentionInDays": 30 }, "AllRegions": true }'
# Rule 3: Auto-enable OTel vended metric enrichment org-wideaws observabilityadmin create-telemetry-rule-for-organization \ --rule-name "otel-enrichment-org" \ --rule '{ "ResourceType": "AWS::CloudWatch::OTelEnrichment", "TelemetryType": "Metrics", "DestinationConfiguration": { "DestinationType": "cloud-watch-logs" }, "AllRegions": true }'Telemetry Configuration Rules automate log collection (VPC Flow Logs, EKS control plane logs, CloudTrail, WAF, ALB, Route 53 Resolver, S3 access logs) and OTel vended metric enrichment across your organization. For OS/app-level metrics on EC2 (detailed monitoring, CloudWatch Agent), deploy via SSM or user data. For EKS Container Insights metrics, use the EKS add-on or managed collector.
Cost Analysis: OTel per-GB vs. Classic per-metric
Scenario A: Application with 500 custom metrics, moderate cardinality
Model | Calculation | Monthly cost |
|---|---|---|
Classic (PutMetricData) | 500 metrics x $0.30/metric | $150/month |
OTel (per-GB) | ~9 GB/month ingestion (500 metrics with 450 bytes per point at 1-min resolution) x $0.50 | ~$4.50/month (check current pricing) |
Scenario B: Kubernetes cluster with 5,000 unique time series (pod-level)
Model | Calculation | Monthly cost |
|---|---|---|
Classic (PutMetricData) | 5,000 metrics x $0.30 (first 10K tier) | $1,500/month |
OTel (per-GB) | ~118 GB/month ingestion (5000 * 550 bytes * 1 min ) | ~$59/month (check current pricing) |
At high cardinality (pod-level metrics across hundreds of pods), OTel is dramatically cheaper because you pay per-GB regardless of how many unique label combinations exist.
Scenario C: High-cardinality with 50,000 unique time series
Model | Calculation | Monthly cost |
|---|---|---|
Classic (PutMetricData) | 10K x $0.30 + 40K x $0.10 | $7,000/month |
OTel (per-GB) | ~1188 GB/month ingestion (50000 * 550 bytes * 1 min ) | ~$594/month (check current pricing) |
Key insight: OTel per-GB pricing removes the "cardinality tax." With PutMetricData, every unique combination of metric name + dimensions = a billable metric. With OTel, you pay only for the bytes ingested — add labels freely without cost penalty.
PromQL API queries add $0.01 per million samples scanned. Console/dashboard queries are free.
Migration Path: PutMetricData/EMF to OTel
Current State | Migration Step | Effort |
|---|---|---|
PutMetricData via SDK | Replace SDK calls with OTel SDK meter/counter/histogram | Medium (code change) |
EMF via CloudWatch Agent | Add OTLP receiver to agent config; update app to emit OTLP | Low-Medium |
Prometheus exporters (self-managed) | Point collector/agent at CloudWatch OTLP endpoint instead of self-hosted Prometheus | Low (config change only) |
ADOT Collector (existing) | Swap EMF exporter for otlphttp exporter pointing to CloudWatch | Low (config change only) |
No custom metrics yet | Start with OTel SDK from day one | Greenfield (easiest) |
For existing Prometheus setups, the migration is a single config change — replace the remote_write endpoint:
# Before: sending to self-managed Prometheus or AMP
exporters:
prometheusremotewrite:
endpoint: https://aps-workspaces.us-east-1.amazonaws.com/workspaces/ws-xxx/api/v1/remote_write
# After: sending to CloudWatch OTLP
exporters:
otlphttp/cloudwatch:
endpoint: https://monitoring.us-east-1.amazonaws.com/v1/metrics
auth:
authenticator: sigv4auth
Common Patterns
Pattern 1: EKS cluster with full-stack observability
# OTel Collector DaemonSet config for EKS
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
prometheus:
config:
scrape_configs:
- job_name: 'kube-state-metrics'
static_configs:
- targets: ['kube-state-metrics:8080']
- job_name: 'node-exporter'
static_configs:
- targets: ['node-exporter:9100']
processors:
batch:
timeout: 60s
resource:
attributes:
- key: k8s.cluster.name
value: production-east
action: upsert
exporters:
otlphttp/cloudwatch:
endpoint: https://monitoring.us-east-1.amazonaws.com/v1/metrics
auth:
authenticator: sigv4auth
extensions:
sigv4auth:
region: us-east-1
service: monitoring
service:
extensions: [sigv4auth]
pipelines:
metrics:
receivers: [otlp, prometheus]
processors: [batch, resource]
exporters: [otlphttp/cloudwatch]Pattern 2: Lambda function with OTel layer
# Get the latest ADOT Python layer ARN for your region from:# https://aws-otel.github.io/docs/getting-started/lambda/lambda-python
aws lambda update-function-configuration \ --function-name my-function \ --layers "arn:aws:lambda:us-east-1:901920570463:layer:aws-otel-python-amd64-ver-1-25-0:1" \ --environment "Variables={ AWS_LAMBDA_EXEC_WRAPPER=/opt/otel-instrument, OTEL_EXPORTER_OTLP_METRICS_ENDPOINT=https://monitoring.us-east-1.amazonaws.com/v1/metrics, OTEL_EXPORTER_OTLP_METRICS_PROTOCOL=http/protobuf, OTEL_METRICS_EXPORTER=otlp, OTEL_SERVICE_NAME=my-function, OTEL_RESOURCE_ATTRIBUTES=service.name=my-function }"Pattern 3: Multi-account with centralized PromQL queries
Combine with OTel enrichment and metrics centralization:
# Step 1: Enable enrichment in each source account
aws observabilityadmin start-telemetry-enrichment
aws cloudwatch start-otel-enrichment
# Step 2: Enable metrics centralization (from delegated admin)
# In the DESTINATION (monitoring) account — create a sink
aws oam create-sink \
--name "central-metrics-sink" \
--tags Environment=production
# Create a policy allowing the org to link
aws oam put-sink-policy \
--sink-identifier "arn:aws:oam:us-east-1:123456789012:sink/sink-id" \
--policy '{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {"AWS": "*"},
"Action": ["oam:CreateLink", "oam:UpdateLink"],
"Resource": "*",
"Condition": {
"ForAnyValue:StringEquals": {
"aws:PrincipalOrgID": "o-your-org-id"
}
}
}]
}'
# In each SOURCE account — create a link to the sink
aws oam create-link \
--label-template "\$AccountName" \
--resource-types '["AWS::CloudWatch::Metric"]' \
--sink-identifier "arn:aws:oam:us-east-1:123456789012:sink/sink-id"
# Step 3: Query all accounts from destination with PromQL
# "Show me CPU across all production EC2 instances in the org"
# avg(aws_ec2_cpuutilization{@resource.aws_ec2_instance.tag.environment="production"})
Things to Know Before You Start
Constraint | Detail |
|---|---|
OTel metrics and classic metrics are separate stores | You cannot mix PromQL queries with classic CloudWatch Metrics math. Choose one path per metric. |
PromQL alarms are separate from classic metric alarms | Existing classic alarms continue to work but cannot reference OTel metrics |
150 labels is per data point, not per metric name | Each individual measurement can carry up to 150 key-value pairs |
Endpoint requires HTTPS | Plain HTTP is rejected. TLS is mandatory. |
SigV4 auth requires IAM permissions | The calling identity needs cloudwatch:PutMetricData or the new OTLP-specific permissions |
OTel enrichment is account-level | Must be enabled in each account where you want vended metrics in PromQL |
Collector needs outbound HTTPS to monitoring.{region}.amazonaws.com | Ensure security groups/NACLs allow egress on 443 |
Telemetry rules apply to new and existing resources | Rules discover existing resources and configure them, not just new ones |
References
Series: CloudWatch Builder Series (3 articles)
- 3CloudWatch Builder Series - Part 3 : OpenTelemetry This article
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article