Observability for ML Pipelines on AWS: Following a Model Run with OpenTelemetry
A practical architecture for tracing and troubleshooting ML pipelines on AWS using OpenTelemetry, SageMaker Pipelines, EventBridge, CloudWatch, and ADOT. The article focuses on correlating model runs across asynchronous processing, training, evaluation, and model registration without treating an ML pipeline as a single long-running trace.
An ML pipeline can finish with a green status and still leave an operations team with a bad answer to the most important question: what actually happened during this run?
A pipeline orchestrator can tell us that data preparation succeeded, training completed, evaluation passed, and a model was deployed. That is execution state. It is not necessarily observability.
When a production ML workflow slows down or behaves unexpectedly, the real investigation usually crosses several boundaries at once. A preprocessing container may spend most of its time waiting on object storage. A training job may sit in provisioning longer than expected. Model registration may retry after a transient API failure. Deployment may succeed while the new endpoint takes longer than normal to become useful. Each component can expose logs and metrics, yet the causal path of a single model run remains fragmented.
This is where OpenTelemetry becomes useful.
The useful mental model is not “instrument every container.” It is treat the ML pipeline run as a distributed transaction. Give the run one correlation identity, represent meaningful stages as spans, preserve that context when the workflow crosses process boundaries, and combine that trace with the native AWS metrics and logs that already describe the infrastructure.
That produces an observability model that follows the lifecycle of a model instead of the lifecycle of an individual pod or job.
Pipeline success is too coarse a signal
Consider a typical production workflow:
Data validation
|
Feature preparation
|
Training
|
Evaluation
|
Model registration
|
Deployment
|
Post-deployment validationThe orchestrator normally knows whether each stage succeeded. But an SRE or ML platform engineer needs a different set of answers:
- Which stage consumed most of the wall-clock time?
- Was that time spent computing, waiting for data, provisioning capacity, or retrying an API call?
- Which dataset and model version were involved?
- Did the training job become slower even though the pipeline still met its timeout?
- Did deployment complete normally while inference latency deteriorated afterward?
- Are retries increasing before they become visible as outright failures?
- Can one run be followed across Python code, Kubernetes workloads, AWS service calls, logs, and metrics?
Those questions require correlation.
Amazon SageMaker AI already publishes useful operational telemetry to Amazon CloudWatch, including metrics for processing jobs, training jobs, batch transform jobs, and inference endpoints. Training and processing workloads can expose CPU, memory, disk, and GPU-related signals, while endpoint telemetry provides request and latency information. Those signals are necessary, but they are still organized around resources and jobs.
OpenTelemetry adds the missing execution context.
Make the pipeline run the trace
For an ML workflow, I prefer to model one logical pipeline run as the root trace.
The major stages become child spans:
ml.pipeline
|
+ data.validate
|
+ features.prepare
|
+ model.train
|
+ model.evaluate
|
+ model.register
|
+ model.deploy
|
+ deployment.verifyThis sounds simple, but it changes the way incidents are investigated.
Instead of opening several dashboards and trying to reconstruct a timeline from timestamps, an engineer can begin with a failed or slow pipeline run and move down the trace.
The
model.train span might show that the training logic itself took eleven minutes, but the total stage took twenty-four minutes. A nested span around data acquisition might explain six minutes. Another span around an AWS API operation could expose retries during job creation. Native CloudWatch GPU metrics could then tell us whether the expensive training instances were actually busy during the remaining time.The trace does not replace the metrics. It gives them context.
A practical AWS architecture
A useful architecture has four layers.
The first layer is the pipeline itself. This might be Amazon SageMaker Pipelines, AWS Step Functions, Apache Airflow running on Amazon EKS, or another orchestrator.
The second layer is instrumentation. Python, Java, Go, or other pipeline components emit OpenTelemetry traces, metrics, and logs. Auto-instrumentation is valuable for HTTP, AWS SDK, database, and framework activity, but important ML lifecycle operations still deserve manual spans.
The third layer is collection. On AWS, AWS Distro for OpenTelemetry, commonly called ADOT, provides AWS-supported OpenTelemetry SDK and Collector components. Workloads can send OTLP telemetry to an ADOT Collector rather than coupling application code to a specific monitoring backend.
The fourth layer is storage and analysis. Traces can flow to AWS X-Ray, metrics can flow to Amazon CloudWatch or Amazon Managed Service for Prometheus, and logs can remain in CloudWatch Logs. SageMaker AI's native CloudWatch telemetry continues to provide service-level and infrastructure signals.
Conceptually:
Pipeline Orchestrator
|
OpenTelemetry context
|
+-------------+-------------+
| | |
Data job Training job Deploy job
| | |
+------ OTLP telemetry ------+
|
ADOT Collector
|
+----------+----------+
| | |
X-Ray CloudWatch AMP
traces metrics/logs metricsThe ADOT Collector becomes a routing and processing layer, not the source of truth. Instrumentation stays based on OpenTelemetry, which keeps the application side portable.
Instrument lifecycle decisions, not every line of code
Over-instrumentation is one of the easiest ways to make tracing expensive and useless at the same time.
A span should represent an operation an engineer might actually investigate.
For a training stage, useful spans could include:
model.train
feature_set.load
training_job.submit
training_job.wait
artifact.persistThat is much more useful than generating thousands of tiny spans around ordinary Python functions.
A minimal Python example looks like this:
import os
from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.trace import Status, StatusCode
resource = Resource.create(
{
"service.name": "ml-training-worker",
"service.namespace": "risk-modeling",
"deployment.environment.name": os.getenv("ENVIRONMENT", "dev"),
}
)
provider = TracerProvider(resource=resource)
provider.add_span_processor(
BatchSpanProcessor(
OTLPSpanExporter(
endpoint=os.environ["OTEL_EXPORTER_OTLP_TRACES_ENDPOINT"]
)
)
)
trace.set_tracer_provider(provider)
tracer = trace.get_tracer("ml.pipeline")
def train_model(context):
with tracer.start_as_current_span("model.train") as span:
span.set_attribute("ml.pipeline.name", context.pipeline_name)
span.set_attribute("ml.pipeline.run_id", context.run_id)
span.set_attribute("ml.stage", "training")
span.set_attribute("ml.dataset.version", context.dataset_version)
span.set_attribute("ml.model.name", context.model_name)
try:
result = context.run_training()
span.set_attribute("ml.training.outcome", "success")
return result
except Exception as exc:
span.record_exception(exc)
span.set_status(Status(StatusCode.ERROR))
span.set_attribute("ml.training.outcome", "failure")
raiseThe point of the custom attributes is not to invent a huge private telemetry vocabulary. It is to establish a few stable fields that let engineers identify the pipeline, stage, dataset, model, environment, and outcome.
Use established OpenTelemetry semantic conventions wherever they apply. For ML-specific lifecycle context that is not covered by a standard convention, keep the internal namespace small and documented.
The correlation ID matters more than the dashboard
The hardest part of ML observability is often not exporting telemetry. It is maintaining identity as the workflow moves between systems.
A useful run identifier should be attached consistently to:
- the root OpenTelemetry trace
- pipeline metadata
- job tags where supported
- application logs
- model metadata
- deployment records
For managed AWS metrics that cannot be given arbitrary OpenTelemetry attributes, use deterministic resource names or tags that map back to the same pipeline run.
For example, a training job might include a shortened pipeline run identifier in its job name while the full identifier is stored as a trace attribute and structured log field.
That gives you a join key without trying to force every backend into the same data model.
There is another subtlety: asynchronous stages are not always true parent-child operations from a tracing perspective. Context can be lost when a managed service starts a job later or when a workflow fans out into parallel work.
Do not manufacture a parent-child relationship just to make a trace look tidy.
When execution is asynchronous or fan-out based, span links can describe causality without pretending that one process directly called another. When trace propagation across a boundary is unavailable, preserve a correlation identifier in pipeline parameters or metadata and use it to reconnect the operational story.
The trace should reflect reality, not the diagram we wish we had.
Keep run IDs out of metric dimensions
A pipeline run identifier is excellent trace metadata and terrible metric metadata.
Suppose every training run emits:
ml_training_duration_seconds{
pipeline_run_id="7f3...",
model_version="2026-08-10-1742",
dataset_id="..."
}That looks convenient until every run creates a new combination of metric attributes.
High-cardinality metrics consume resources in telemetry SDKs and backends and can eventually make dashboards, alerts, and cost behavior difficult to predict.
Use bounded metric attributes instead:
stage="training"
environment="prod"
outcome="success"
region="us-east-1"
model_family="fraud-risk"Keep unique identifiers such as run IDs, request IDs, artifact hashes, and highly granular model versions in traces and logs.
This separation is one of the most important design choices in a production telemetry model:
Metrics tell you that a class of pipeline runs is unhealthy. Traces tell you which run and why. Logs provide the detailed evidence.
Use AWS-native telemetry instead of recreating it
OpenTelemetry should not become an excuse to rebuild telemetry AWS services already produce.
For SageMaker AI training and processing jobs, CloudWatch already provides resource utilization metrics. For inference endpoints, SageMaker AI publishes invocation and latency telemetry. CloudWatch Logs captures output from several SageMaker AI workload types.
Keep those signals.
The OpenTelemetry layer should answer questions that native resource metrics cannot answer easily:
- Which pipeline run created this training job?
- Which upstream stage delayed it?
- Which model version was being evaluated?
- How many retries happened before the API call succeeded?
- Which deployment followed this evaluation?
- What happened immediately after the model entered production?
This is the difference between collecting more telemetry and creating better observability.
Separate pipeline reliability from model quality
Another common mistake is to put every ML concern under “observability.”
A perfectly reliable pipeline can train a bad model.
A highly accurate model can be delivered by an unreliable pipeline.
Those are different failure domains.
Operational telemetry should cover concerns such as:
pipeline completion latency
stage duration
failure rate
retry rate
queue or provisioning delay
resource saturation
deployment success
inference latency
inference error rateModel monitoring should cover concerns such as:
data quality
feature drift
prediction drift
model quality
bias
business outcome degradationThey should meet at the same operational context, but they should not be treated as the same signal.
For example, a model-quality alarm might identify a production model version. That version should map back to the deployment trace that introduced it, which should map back to the evaluation and training run that produced it.
That is where telemetry becomes operationally powerful. The model-quality system tells you what changed in behavior. The distributed trace helps explain how that model reached production.
Define SLOs around the lifecycle
Once the telemetry has consistent structure, ML platform SLOs become much more useful.
Instead of an alert that says “training job failed,” consider lifecycle-oriented objectives:
- 99 percent of production pipeline runs complete within the agreed delivery window.
- 99.5 percent of model deployment attempts complete without manual intervention.
- 95 percent of training jobs begin execution within an acceptable provisioning window.
- Post-deployment validation completes before traffic promotion.
- Inference availability and latency remain inside service objectives after a model rollout.
The trace then provides the diagnostic path when the SLO is violated.
CloudWatch Application Signals can also provide service-level telemetry and SLO capabilities for supported application workloads, while X-Ray traces can help with dependency-level analysis. For EKS-hosted components, AWS provides OpenTelemetry-based integration through its observability tooling and ADOT.
The practical goal is not to put every ML stage into one product. It is to make the lifecycle queryable.
Sampling should follow workload economics
Pipeline control traffic and inference traffic should not necessarily use the same trace sampling policy.
A team might have hundreds of pipeline runs per day but millions of inference requests.
Sampling all pipeline lifecycle spans may be perfectly reasonable because the volume is small and every failed production training run matters.
Sampling every inference request may be unnecessarily expensive.
That suggests two policies:
Pipeline lifecycle:
high sampling, potentially 100 percent for important production runs
Online inference:
probabilistic or tail-based sampling according to traffic and investigation needsThe exact percentages depend on workload volume, incident requirements, and telemetry cost.
What matters is that sampling is a deliberate reliability and cost decision. It should not be whatever default happened to ship with the first SDK configuration.
Instrument the observability pipeline too
The telemetry path can fail.
Collectors can become memory constrained. Exporters can be throttled. Network paths can fail. A configuration change can silently drop attributes that dashboards depend on.
If the organization depends on OpenTelemetry to investigate ML incidents, then the Collector itself becomes part of the reliability surface.
At minimum, monitor:
accepted telemetry
exported telemetry
dropped telemetry
export failures
collector memory
queue pressure
receiver errorsA pipeline should also continue its primary work if telemetry export experiences a transient failure. Observability is supposed to explain failures, not create a new one.
Batching, bounded queues, sane retry behavior, memory limits, and explicit collector health monitoring are operational requirements, not polishing work.
What the incident workflow looks like
Imagine that a production model pipeline normally completes in twenty-five minutes but suddenly takes forty-two.
The orchestrator still reports success.
A metric alert identifies abnormal pipeline duration. The engineer opens the trace for the affected run and sees that
model.train accounts for most of the increase.Inside that stage, the actual training span looks normal. The delay sits in
training_job.wait .The SageMaker AI CloudWatch metrics show that the training instances were healthy once the job started. Structured logs show no application errors. The trace attributes provide the training job name and dataset version, and the job timeline confirms that the delay occurred before useful computation began.
The investigation has now moved from “ML got slower” to a much narrower operational question about job startup and capacity.
That is the value of tracing the lifecycle.
The trace does not magically diagnose every ML problem. It reduces the search space.
The design principle I would keep
OpenTelemetry is most valuable in ML platforms when it is treated as correlation infrastructure.
Do not replace CloudWatch metrics that already work.
Do not put unique run identifiers on every metric.
Do not create spans around every function.
Do not pretend that model quality and platform reliability are the same discipline.
Instead:
- use traces to model causality
- use metrics to detect aggregate degradation
- use logs for detailed evidence
- retain AWS-native telemetry for managed services
- carry a stable model and pipeline identity across the lifecycle
- use ADOT as the OpenTelemetry integration layer on AWS
When those pieces are aligned, a pipeline stops looking like a collection of jobs.
It becomes one observable system.
And that is a much better starting point for operating machine learning in production.
Suggested tags
observability opentelemetry machine-learning amazon-sagemaker aws-distro-for-opentelemetry amazon-cloudwatch aws-x-ray amazon-eks Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article