AWS Builder Center

CloudWatch Builder Series - Part 3 : OpenTelemetry

A hands-on guide to native OpenTelemetry ingestion in CloudWatch. Covers collector configurations for EKS, ECS, and Lambda, PromQL query patterns, migration paths from existing setups, cost modeling at scale, and Telemetry Configuration Rules for automatic org-wide enablement.

Series: CloudWatch Builder Series (3 articles)

  1. 3
    CloudWatch Builder Series - Part 3 : OpenTelemetry This article

OpenTelemetry Metrics and Telemetry Configuration



CloudWatch now natively ingests metrics via the OpenTelemetry Protocol (OTLP) and exposes them through PromQL. This can be a alternative to the classic PutMetricData/EMF model with a per-GB pricing approach — no per-metric charges, no dimension limits, and 15 months of storage included.
Telemetry Configuration Rules let you auto-enable metric and log collection across all AWS resources in your organization — no manual agent setup per resource.
Capability
Details
Ingestion protocol
OTLP (OpenTelemetry Protocol) over HTTPS
Endpoint
 https://monitoring.\{region}.amazonaws.com/v1/metrics 
Authentication
AWS SigV4 or bearer token
Labels per data point
Up to 150 (vs. 30 dimensions in classic CloudWatch)
Metric types
Gauge, Sum, Histogram
Storage
15 months included, no separate charge
Query language
PromQL
Pricing
Per-GB ingestion only — no per-metric, no per-API-call charges
Clients
OTel SDKs (Java, Python, Go, .NET, Node.js), OTel Collector, CloudWatch Agent



When to use: OTel Metrics vs. PutMetricData vs. EMF

Criteria
OTel Metrics (OTLP)
PutMetricData
Embedded Metric Format (EMF)
Pricing model
Per-GB ingestion
$0.30/metric/month (first 10K)
Per-GB log ingestion + metric extraction
Dimensions/labels
150 labels
30 dimensions
30 dimensions
Query language
PromQL
CloudWatch Metrics math
CloudWatch Metrics math
High cardinality
Built for it — no penalty for pod names, request IDs
Expensive — each unique combo = new billable metric
Moderate — limited by dimensions
Vendor lock-in
None — standard OTLP works with Prometheus, Grafana, Datadog
AWS-specific API
AWS-specific log format
Storage
15 months included
Standard retention tiers (3h/1min/5min/1h)
Log retention + metric retention
Best for
Application metrics, container metrics, high-cardinality
Low-volume infrastructure metrics, alarms on simple counts
Metrics embedded in structured logs
Migration effort
Swap exporter config
N/A (existing)
N/A (existing)
Decision guide:
  • You have more than 10,000 custom metrics — OTel (per-GB is cheaper than $0.30/metric/month at scale)
  • You need more than 30 dimensions — OTel (150 labels, add pod names, request IDs, custom tags freely)
  • You use Prometheus/Grafana already — OTel (same collector, same queries, add CloudWatch as destination)
  • You have a handful of simple metrics with alarms — PutMetricData (simpler, works fine at low volume)
  • You want metrics extracted from logs — EMF (still valid for log-derived metrics)



Architecture


Implementation

What each step does

Step
What it does
Who runs it
1. Enable resource tags on telemetry
Allows CloudWatch to attach resource tags as PromQL labels
Account admin (one-time)
2. Enable OTel enrichment
Makes existing AWS vended metrics (EC2, RDS, ELB) queryable via PromQL without any agent changes
Account admin (one-time)
3. Configure collector/agent
Sets up the pipeline to send your custom application metrics to the OTLP endpoint
Platform/DevOps team
4. Query with PromQL
Validates metrics are flowing and demonstrates query patterns
Developer/SRE
5. Create PromQL alarms
Replaces classic metric alarms with PromQL-based alarms
SRE/Ops

Step 1: Enable resource tags on telemetry

One-time setup — allows resource tags (environment, team, cost-center) to appear as PromQL labels on vended metrics.
aws observabilityadmin start-telemetry-enrichment

Step 2: Enable OTel enrichment for vended metrics

This makes your existing AWS service metrics (EC2 CPUUtilization, RDS connections, etc.) queryable via PromQL — enriched with resource ARN and tag labels. No agent changes required.
aws cloudwatch start-otel-enrichment
After enabling, you can immediately query vended metrics with PromQL:
# EC2 CPU by instance tag "environment=production"
aws_ec2_cpuutilization{@resource.aws_ec2_instance.tag.environment="production"}

Step 3: Configure the collection pipeline

Choose one of the following based on your environment:
Option A — CloudWatch Agent (recommended for most workloads)
Deploy the agent on EC2, ECS task, or as a DaemonSet in EKS. Your app sends metrics to the agent's local OTLP listener (same host/pod), and the agent handles auth + forwarding to CloudWatch.
CloudWatch Agent configuration ( amazon-cloudwatch-agent.json ):
{
"metrics": {
"metrics_collected": {
"otlp": {
"grpc_endpoint": "127.0.0.1:4317",
"http_endpoint": "127.0.0.1:4318"
}
}
}
}
Your application sends metrics to the local agent via OTLP (port 4317 for gRPC, 4318 for HTTP). The agent forwards to CloudWatch.
Option B — Custom OTel Collector (full control)
Deploy as a sidecar, DaemonSet, or standalone Deployment in your cluster. You own the full pipeline config — receivers, processors, exporters — and point the exporter to CloudWatch's OTLP endpoint with SigV4 auth.
OTel Collector configuration ( otel-collector-config.yaml ):
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
http:
endpoint: 0.0.0.0:4318
prometheus:
config:
scrape_configs:
- job_name: 'my-app'
scrape_interval: 30s
static_configs:
- targets: ['localhost:9090']
processors:
batch:
timeout: 60s
send_batch_size: 1000
resource:
attributes:
- key: environment
value: production
action: upsert
- key: service.name
value: order-service
action: upsert
exporters:
otlphttp/cloudwatch:
endpoint: https://monitoring.us-east-1.amazonaws.com/v1/metrics
auth:
authenticator: sigv4auth
compression: gzip
extensions:
sigv4auth:
region: us-east-1
service: monitoring
service:
extensions: [sigv4auth]
pipelines:
metrics:
receivers: [otlp, prometheus]
processors: [batch, resource]
exporters: [otlphttp/cloudwatch]
Option C — Direct SDK (no collector needed)
Embed directly in your application code. The OTel SDK exports metrics straight to CloudWatch's OTLP endpoint — no agent or collector process to manage. Best for Lambda or simple services where running a sidecar adds unnecessary complexity.
Python example using the OTel SDK:
Required environment variables (set before running your app):
export OTEL_EXPORTER_OTLP_METRICS_ENDPOINT="https://monitoring.us-east-1.amazonaws.com/v1/metrics"
export OTEL_EXPORTER_OTLP_METRICS_PROTOCOL="http/protobuf"
export OTEL_METRICS_EXPORTER="otlp"
export OTEL_SERVICE_NAME="order-service"

IAM permission on the calling identity:
{
"Effect": "Allow",
"Action": ["cloudwatch:PutMetricData"],
"Resource": "*"
}
from opentelemetry import metrics
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.metrics.export import PeriodicExportingMetricReader
from opentelemetry.exporter.otlp.proto.http.metric_exporter import OTLPMetricExporter

# Point directly at the CloudWatch OTLP endpoint.
# Authentication is handled automatically via the default AWS credential chain
# (environment variables, IAM role, instance profile, etc.)
# when using the ADOT Python SDK distribution.
exporter = OTLPMetricExporter(
endpoint="https://monitoring.us-east-1.amazonaws.com/v1/metrics"
)

reader = PeriodicExportingMetricReader(exporter, export_interval_millis=60000)
provider = MeterProvider(metric_readers=[reader])
metrics.set_meter_provider(provider)

# Create and record metrics
meter = metrics.get_meter("order-service")
order_counter = meter.create_counter(
name="orders.processed",
description="Total orders processed",
unit="1"
)
latency_histogram = meter.create_histogram(
name="orders.latency",
description="Order processing latency",
unit="ms"
)

# Use in your application
order_counter.add(1, {"region": "us-east-1", "customer_tier": "enterprise", "status": "success"})
latency_histogram.record(142.5, {"region": "us-east-1", "endpoint": "/api/v2/orders"})
Option D — Managed Prometheus Collector (EKS — zero pods to manage)
For EKS workloads, CloudWatch provides a fully managed collector that auto-discovers and scrapes Prometheus endpoints. AWS runs the collector for you — no DaemonSet, no sidecar, no pods in your cluster. You provide a scrape config and CloudWatch discovers and scrapes your Prometheus endpoints via VPC networking. Best for teams already exposing /metrics endpoints.
aws amp create-scraper \
--alias "eks-metrics-scraper" \
--source eksConfiguration="{clusterArn='arn:aws:eks:us-east-1:123456789012:cluster/my-cluster', \
securityGroupIds=['sg-12345'], \
subnetIds=['subnet-abc123','subnet-def456']}" \
--scrape-configuration configurationBlob=$(cat prometheus-config.yaml | base64 -w 0) \
--destination cloudWatchConfiguration="{datasetArn='arn:aws:cloudwatch:us-east-1:123456789012:dataset/default'}"
Prometheus scrape config (encode to base64):
global:
scrape_interval: 30s
scrape_configs:
- job_name: 'kubernetes-pods'
kubernetes_sd_configs:
- role: pod
relabel_configs:
- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
action: keep
regex: true
- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
action: replace
target_label: __metrics_path__
regex: (.+)

Step 4: Query with PromQL

PromQL (Prometheus Query Language) is the query language used to retrieve and aggregate OTel metrics in CloudWatch. If you've used Prometheus or Grafana, it's the same syntax —  rate() ,  histogram_quantile() ,  avg by()  — now running natively in CloudWatch without managing a Prometheus server.
Once metrics flow in, query them in CloudWatch Query Studio or via API:
# Request rate by service over 5 minutes
rate(orders_processed_total[5m])
# P99 latency from histogram
histogram_quantile(0.99, rate(orders_latency_bucket[5m]))
# CPU by pod in production namespace
avg by (k8s_pod_name) (
rate(container_cpu_usage_seconds_total{k8s_namespace_name="production"}[5m])
)
# Filter by resource tag (vended metrics with enrichment enabled)
avg(aws_ec2_cpuutilization{@resource.aws_ec2_instance.tag.team="payments"})
# Top 5 highest memory pods
topk(5, container_memory_working_set_bytes{k8s_namespace_name="production"})
Label structure for OTLP metrics in CloudWatch:
Prefix
What it contains
Example
 @resource. 
Resource attributes (service, host, cloud)
 @resource.service.name="order-service" 
 @scope. 
Instrumentation scope
 @scope.name="io.opentelemetry.sdk" 
(no prefix)
Datapoint attributes
 region="us-east-1" ,  status="success" 
 @resource.aws_* 
AWS-enriched attributes (vended metrics)
 @resource.aws_ec2_instance.tag.team="payments" 

Step 5: Create PromQL alarms

# Alarm: P99 latency exceeds 500ms
aws cloudwatch put-metric-alarm \
--alarm-name "OrderLatencyP99High" \
--evaluation-criteria '{
"PromQLCriteria": {
"Query": "histogram_quantile(0.99, rate(orders_latency_bucket{service_name=\"order-service\"}[5m])) > 500",
"PendingPeriod": 300,
"RecoveryPeriod": 600
}
}' \
--evaluation-interval 60 \
--alarm-actions arn:aws:sns:us-east-1:123456789012:ops-alerts

# Alarm: Error rate exceeds 5%
aws cloudwatch put-metric-alarm \
--alarm-name "OrderErrorRateHigh" \
--evaluation-criteria '{
"PromQLCriteria": {
"Query": "rate(orders_processed_total{status=\"error\"}[5m]) / rate(orders_processed_total[5m]) * 100 > 5",
"PendingPeriod": 120,
"RecoveryPeriod": 300
}
}' \
--evaluation-interval 60 \
--alarm-actions arn:aws:sns:us-east-1:123456789012:ops-alerts



Telemetry Configuration Rules (auto-enable at scale)

Telemetry Configuration Rules automatically enable metric and log collection for AWS resources across your organization — no manual per-resource setup.

What it solves

  • New VPC created but no flow logs enabled
  • EKS cluster running but control plane logging not turned on
  • CloudTrail not configured for new accounts in the org
  • WAF, ALB, or Route 53 Resolver logging missing on new resources
  • OTel vended metric enrichment not enabled in new accounts
  • Telemetry gaps discovered only during incidents

How it works

You define rules that specify: which resource types, which telemetry types, which accounts/OUs. CloudWatch discovers matching resources and automatically configures collection.
# Rule 1: Auto-enable VPC Flow Logs for all VPCs in production OU
aws observabilityadmin create-telemetry-rule-for-organization \
--rule-name "vpc-flow-logs-prod" \
--rule '{
"ResourceType": "AWS::EC2::VPC",
"TelemetryType": "Logs",
"TelemetrySourceTypes": ["VPC_FLOW_LOGS"],
"DestinationConfiguration": {
"DestinationType": "cloud-watch-logs",
"RetentionInDays": 30,
"VPCFlowLogParameters": {
"TrafficType": "ALL",
"MaxAggregationInterval": 60
}
},
"Scope": "ou-prod-12345",
"AllRegions": true
}'

# Rule 2: Auto-enable EKS control plane logs for all EKS clusters
aws observabilityadmin create-telemetry-rule-for-organization \
--rule-name "eks-logs-all" \
--rule '{
"ResourceType": "AWS::EKS::Cluster",
"TelemetryType": "Logs",
"TelemetrySourceTypes": ["EKS_AUDIT_LOGS", "EKS_API_LOGS"],
"DestinationConfiguration": {
"DestinationType": "cloud-watch-logs",
"RetentionInDays": 30
},
"AllRegions": true
}'

# Rule 3: Auto-enable OTel vended metric enrichment org-wide
aws observabilityadmin create-telemetry-rule-for-organization \
--rule-name "otel-enrichment-org" \
--rule '{
"ResourceType": "AWS::CloudWatch::OTelEnrichment",
"TelemetryType": "Metrics",
"DestinationConfiguration": {
"DestinationType": "cloud-watch-logs"
},
"AllRegions": true
}'
Telemetry Configuration Rules automate log collection (VPC Flow Logs, EKS control plane logs, CloudTrail, WAF, ALB, Route 53 Resolver, S3 access logs) and OTel vended metric enrichment across your organization. For OS/app-level metrics on EC2 (detailed monitoring, CloudWatch Agent), deploy via SSM or user data. For EKS Container Insights metrics, use the EKS add-on or managed collector.

Cost Analysis: OTel per-GB vs. Classic per-metric

Scenario A: Application with 500 custom metrics, moderate cardinality

Model
Calculation
Monthly cost
Classic (PutMetricData)
500 metrics x $0.30/metric
$150/month
OTel (per-GB)
~9 GB/month ingestion (500 metrics with 450 bytes per point at 1-min resolution) x $0.50
~$4.50/month (check current pricing)

Scenario B: Kubernetes cluster with 5,000 unique time series (pod-level)

Model
Calculation
Monthly cost
Classic (PutMetricData)
5,000 metrics x $0.30 (first 10K tier)
$1,500/month
OTel (per-GB)
~118 GB/month ingestion (5000 * 550 bytes * 1 min )
~$59/month (check current pricing)
At high cardinality (pod-level metrics across hundreds of pods), OTel is dramatically cheaper because you pay per-GB regardless of how many unique label combinations exist.

Scenario C: High-cardinality with 50,000 unique time series

Model
Calculation
Monthly cost
Classic (PutMetricData)
10K x $0.30 + 40K x $0.10
$7,000/month
OTel (per-GB)
~1188 GB/month ingestion (50000 * 550 bytes * 1 min )
~$594/month (check current pricing)
Key insight: OTel per-GB pricing removes the "cardinality tax." With PutMetricData, every unique combination of metric name + dimensions = a billable metric. With OTel, you pay only for the bytes ingested — add labels freely without cost penalty.
PromQL API queries add $0.01 per million samples scanned. Console/dashboard queries are free.



Migration Path: PutMetricData/EMF to OTel

Current State
Migration Step
Effort
PutMetricData via SDK
Replace SDK calls with OTel SDK meter/counter/histogram
Medium (code change)
EMF via CloudWatch Agent
Add OTLP receiver to agent config; update app to emit OTLP
Low-Medium
Prometheus exporters (self-managed)
Point collector/agent at CloudWatch OTLP endpoint instead of self-hosted Prometheus
Low (config change only)
ADOT Collector (existing)
Swap EMF exporter for otlphttp exporter pointing to CloudWatch
Low (config change only)
No custom metrics yet
Start with OTel SDK from day one
Greenfield (easiest)
For existing Prometheus setups, the migration is a single config change — replace the remote_write endpoint:
# Before: sending to self-managed Prometheus or AMP
exporters:
prometheusremotewrite:
endpoint: https://aps-workspaces.us-east-1.amazonaws.com/workspaces/ws-xxx/api/v1/remote_write
# After: sending to CloudWatch OTLP
exporters:
otlphttp/cloudwatch:
endpoint: https://monitoring.us-east-1.amazonaws.com/v1/metrics
auth:
authenticator: sigv4auth



Common Patterns

Pattern 1: EKS cluster with full-stack observability

# OTel Collector DaemonSet config for EKS
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
prometheus:
config:
scrape_configs:
- job_name: 'kube-state-metrics'
static_configs:
- targets: ['kube-state-metrics:8080']
- job_name: 'node-exporter'
static_configs:
- targets: ['node-exporter:9100']
processors:
batch:
timeout: 60s
resource:
attributes:
- key: k8s.cluster.name
value: production-east
action: upsert
exporters:
otlphttp/cloudwatch:
endpoint: https://monitoring.us-east-1.amazonaws.com/v1/metrics
auth:
authenticator: sigv4auth
extensions:
sigv4auth:
region: us-east-1
service: monitoring
service:
extensions: [sigv4auth]
pipelines:
metrics:
receivers: [otlp, prometheus]
processors: [batch, resource]
exporters: [otlphttp/cloudwatch]

Pattern 2: Lambda function with OTel layer

# Get the latest ADOT Python layer ARN for your region from:
# https://aws-otel.github.io/docs/getting-started/lambda/lambda-python

aws lambda update-function-configuration \
--function-name my-function \
--layers "arn:aws:lambda:us-east-1:901920570463:layer:aws-otel-python-amd64-ver-1-25-0:1" \
--environment "Variables={
AWS_LAMBDA_EXEC_WRAPPER=/opt/otel-instrument,
OTEL_EXPORTER_OTLP_METRICS_ENDPOINT=https://monitoring.us-east-1.amazonaws.com/v1/metrics,
OTEL_EXPORTER_OTLP_METRICS_PROTOCOL=http/protobuf,
OTEL_METRICS_EXPORTER=otlp,
OTEL_SERVICE_NAME=my-function,
OTEL_RESOURCE_ATTRIBUTES=service.name=my-function
}"

Pattern 3: Multi-account with centralized PromQL queries

Combine with OTel enrichment and metrics centralization:
# Step 1: Enable enrichment in each source account
aws observabilityadmin start-telemetry-enrichment
aws cloudwatch start-otel-enrichment
# Step 2: Enable metrics centralization (from delegated admin)
# In the DESTINATION (monitoring) account — create a sink
aws oam create-sink \
--name "central-metrics-sink" \
--tags Environment=production
# Create a policy allowing the org to link
aws oam put-sink-policy \
--sink-identifier "arn:aws:oam:us-east-1:123456789012:sink/sink-id" \
--policy '{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {"AWS": "*"},
"Action": ["oam:CreateLink", "oam:UpdateLink"],
"Resource": "*",
"Condition": {
"ForAnyValue:StringEquals": {
"aws:PrincipalOrgID": "o-your-org-id"
}
}
}]
}'
# In each SOURCE account — create a link to the sink
aws oam create-link \
--label-template "\$AccountName" \
--resource-types '["AWS::CloudWatch::Metric"]' \
--sink-identifier "arn:aws:oam:us-east-1:123456789012:sink/sink-id"
# Step 3: Query all accounts from destination with PromQL
# "Show me CPU across all production EC2 instances in the org"
# avg(aws_ec2_cpuutilization{@resource.aws_ec2_instance.tag.environment="production"})



Things to Know Before You Start

Constraint
Detail
OTel metrics and classic metrics are separate stores
You cannot mix PromQL queries with classic CloudWatch Metrics math. Choose one path per metric.
PromQL alarms are separate from classic metric alarms
Existing classic alarms continue to work but cannot reference OTel metrics
150 labels is per data point, not per metric name
Each individual measurement can carry up to 150 key-value pairs
Endpoint requires HTTPS
Plain HTTP is rejected. TLS is mandatory.
SigV4 auth requires IAM permissions
The calling identity needs cloudwatch:PutMetricData or the new OTLP-specific permissions
OTel enrichment is account-level
Must be enabled in each account where you want vended metrics in PromQL
Collector needs outbound HTTPS to monitoring.{region}.amazonaws.com
Ensure security groups/NACLs allow egress on 443
Telemetry rules apply to new and existing resources
Rules discover existing resources and configure them, not just new ones



References

Series: CloudWatch Builder Series (3 articles)

  1. 3
    CloudWatch Builder Series - Part 3 : OpenTelemetry This article
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article