AWS Builder Center

Tracing an Incident Across Kubernetes, OpenTelemetry and Async Messaging Without Losing Causality

Distributed tracing looks straightforward when the request path is synchronous. A client calls an API. The API calls another service. That service queries a database. Trace context travels through HTTP or gRPC headers, and the resulting span tree roughly resembles the execution path. Production systems rarely remain that simple.

DevOps Cloud Platform & Site Reliability Engineer | AWS Certified Solutions Architect – Professional



A request enters Kubernetes, publishes an event, returns successfully, waits in a queue, is consumed by another workload, triggers another message, retries after a transient failure, and eventually reaches a downstream service. By the time an engineer investigates the incident, the original HTTP request may have completed minutes earlier.
The difficult observability problem is no longer collecting spans.
It is preserving causality.
A useful tracing architecture must answer:
Which upstream event caused this downstream work, even when execution crossed queues, processes, retries, batches, and time boundaries?
That requires treating asynchronous tracing differently from synchronous RPC tracing.

The synchronous mental model breaks at the queue

Consider an application running on Amazon EKS:
Client
|
v
API Gateway / Load Balancer
|
v
orders-api
|
+------> PostgreSQL
|
+------> SQS: order-events
|
v
order-worker
|
+------> payment-api
|
+------> SQS: fulfillment-events
|
v
fulfillment-worker

The first half is easy to trace.
HTTP request
|
+-- orders-api
|
+-- database query
|
+-- SQS send

The problem begins after the message enters SQS.
The producer may terminate.
The message may wait.
A consumer may receive several messages in one poll.
Processing may fail.
The message may become visible again.
Another pod may receive it.
Eventually it may be sent to a dead-letter queue.
Trying to represent this entire lifecycle as a traditional parent-child span tree creates misleading traces.
The queue is not simply another RPC hop.
It is a temporal and execution boundary.

Trace context is not business identity

One of the most common mistakes in asynchronous observability is assuming that  trace_id  solves correlation.
It does not.
A trace identifier represents telemetry context. A business identifier represents domain identity.
For an order-processing system, useful identifiers might include:
trace_id
order_id
message_id
payment_id
workflow_id
correlation_id

These values answer different questions.
 trace_id :
Which telemetry operations belong to this distributed execution?
 order_id :
Which business transaction is affected?
 message_id :
Which specific transport message was processed?
 workflow_id :
Which long-running business workflow owns this operation?
Do not overload one identifier to perform all four jobs.
A production investigation often starts from something other than a trace ID:
Customer reports order ORD-74219 is stuck.

The observability system must therefore support:
order_id
|
v
logs / traces / events
|
v
message_id
|
v
consumer execution
|
v
downstream effects

This is why domain correlation attributes should accompany trace context.

Propagate context with the message

OpenTelemetry explicitly recognizes that messaging requires context propagation beyond transport-level tracing.
Its messaging semantic conventions distinguish message creation, sending, receiving, processing, and settlement. They also describe a message creation context that producers should propagate so consumer processing can be correlated with the operation that created the message.
For a producer, the conceptual flow is:
Current application context
|
v
Create / Send span
|
v
Inject trace context
|
v
Message metadata
|
v
Queue

For a consumer:
Queue
|
v
Receive message
|
v
Extract context
|
v
Create processing span
|
v
Application logic

The exact carrier depends on the messaging technology.
Amazon SQS supports message attributes for application metadata and also exposes message system attributes, including  AWSTraceHeader , for AWS tracing integration.
A message envelope might conceptually carry:
{
"body": {
"orderId": "ORD-74219",
"eventType": "OrderAccepted"
},
"metadata": {
"traceparent": "00-...",
"correlation_id": "workflow-92814"
}
}

The important architectural rule is:
Context must cross the asynchronous boundary explicitly.
If the producer creates a trace and the consumer independently creates another trace, both traces may be individually correct while the end-to-end causal chain is lost.

Parent-child is not always the right relationship

Suppose a consumer polls ten SQS messages.
Those messages may originate from ten independent traces.
What should be the parent of the consumer's receive span?
A span can have only one parent.
Choosing one message arbitrarily would imply that the other nine messages were caused by the first.
They were not.
OpenTelemetry addresses this through span links.
Conceptually:
Producer Trace A ----\
Producer Trace B -----\
Producer Trace C ------> Consumer Process Span
Producer Trace D -----/ ^
Producer Trace E ----/ |
span links

The consumer span can have its own execution context while linking to the creation contexts of the messages it processes.
OpenTelemetry's current messaging conventions use span links as the default correlation mechanism for producer and consumer contexts because this works across different messaging models and, critically, supports batching where one consumer operation relates to multiple messages.
For single-message processing, using the message creation context as the parent can sometimes be reasonable.
For batch processing, links are usually the more accurate model.
This distinction matters during incident analysis.
Tracing systems should represent actual causal relationships, not merely produce aesthetically pleasing trees.

Separate receive from process

Another common instrumentation mistake is treating message receipt and message processing as the same operation.
They are different.
For a pull-based consumer:
ReceiveMessage
|
v
Message available to application
|
v
ProcessMessage
|
v
Delete / acknowledge

The receive operation measures interaction with the broker.
The process operation measures application work.
OpenTelemetry's messaging model distinguishes  receive ,  process , and  settle  operations. It recommends  CONSUMER  span kind for processing spans, while receive and settle operations have different semantics.
This separation becomes valuable during incidents.
Imagine:
Receive latency: 40 ms
Processing latency: 18 s
Settlement latency: 30 ms

The broker is not the bottleneck.
Now consider:
Receive latency: 12 s
Processing latency: 300 ms

That points investigation in a very different direction.
If both are represented as one span called  consume , that distinction disappears.

Retries are new executions, not invisible continuations

Retries are where many trace models become misleading.
Consider:
Message M1
|
v
worker-pod-a
|
+--> processing fails
|
v
visibility timeout
|
v
worker-pod-b
|
+--> processing succeeds

These are two execution attempts.
They should not look like one uninterrupted operation.
A useful model is:
Message creation context
|
+------ link ------> Process attempt 1
|
+------ link ------> Process attempt 2

Each attempt should record enough transport information to explain what happened.
For SQS,  ApproximateReceiveCount  can help identify repeated delivery attempts. AWS documents this as a message system attribute available from  ReceiveMessage .
At the application layer, useful telemetry might include:
messaging.message.id
messaging.destination.name
messaging.operation.type
delivery_attempt
workflow_id
order_id

But be disciplined with attributes.
Do not blindly attach entire message bodies to spans.
That creates cardinality, cost, privacy, and security problems.

Kubernetes identity must survive pod churn

Tracing the application is only half the problem.
During an incident, engineers also need to understand where an operation executed.
Kubernetes workloads are ephemeral.
A pod may disappear before anyone investigates.
Therefore telemetry should carry resource identity such as:
service.name
service.version
service.instance.id
k8s.cluster.name
k8s.namespace.name
k8s.deployment.name
k8s.pod.name
k8s.node.name

The distinction between service identity and pod identity is important.
Dashboards and SLOs usually care about:
service.name = fulfillment-worker

Incident debugging may care about:
pod = fulfillment-worker-76c87f9b7c-m4r8p
node = ip-10-0-42-17

If every query begins with pod names, observability becomes fragile because Kubernetes constantly replaces them.
Start with the stable workload identity and drill into ephemeral infrastructure identity only when necessary.
Amazon EKS supports the AWS Distro for OpenTelemetry, and AWS documents ADOT as an AWS-supported OpenTelemetry distribution that can send telemetry from EKS workloads to destinations including CloudWatch and AWS X-Ray.
A typical collection architecture is:
Application Pods
|
| OTLP
v
OpenTelemetry Collector
|
+------> trace backend
|
+------> metrics backend
|
+------> logs pipeline

The collector should be treated as telemetry infrastructure, not as the source of application semantics.
Instrumentation still needs to establish the right causal relationships.

Correlate logs with traces deliberately

Distributed traces rarely contain enough detail to diagnose every failure.
Logs remain essential.
The useful pattern is not:
traces over here
logs over there

It is:
trace_id
span_id
service.name
business correlation fields

included in structured logs.
Then an engineer can move from:
failed consumer span

to:
all logs emitted by this processing attempt

and then to:
all activity for order ORD-74219

These are different navigation paths.
A structured log might look like:
{
"severity": "ERROR",
"service": "payment-worker",
"trace_id": "...",
"span_id": "...",
"order_id": "ORD-74219",
"message_id": "...",
"event": "payment_authorization_failed"
}

Avoid using free-text parsing as your primary correlation strategy.
Observability systems work much better when identity is structured at emission time.

Metrics explain scope, traces explain examples

There is another trap: trying to solve queue incidents entirely with tracing.
Tracing is not a replacement for queue metrics.
Suppose consumer latency suddenly increases.
The investigation should begin with aggregate signals:
queue depth
age of oldest message
publish rate
consume rate
processing latency
error rate
retry rate
dead-letter volume
consumer replicas
CPU / memory saturation

Those signals tell you whether the problem affects one message or the entire system.
Then traces answer:
What happened to representative messages?
The observability flow should be:
Alert
|
v
Metrics establish scope
|
v
Select affected service / queue
|
v
Exemplar or trace search
|
v
Follow causal chain
|
v
Logs provide detailed evidence

Trying to query millions of traces to discover whether a queue is backing up is expensive and unnecessary.
Metrics identify the population.
Traces explain individual paths through that population.

Sampling can destroy the exact trace you need

Asynchronous systems complicate sampling.
A producer may look healthy when its trace begins:
POST /orders -> 200

The actual failure may happen three minutes later:
fulfillment-worker -> repeated processing failure

If sampling decisions are finalized too early, the producer side of the causal chain may already have been discarded.
This is where tail-based sampling can become useful.
Instead of deciding exclusively when a trace begins, a collector can retain telemetry based on outcomes observed later.
Potential policies include retaining:
errors
high latency
selected critical workflows
rare operations
specific service classes

But tail sampling has costs.
Collectors need enough capacity to buffer trace information while decisions are pending. High-volume systems can create significant memory and operational overhead.
There is no universal sampling configuration.
The architecture must balance:
diagnostic completeness
vs
telemetry volume
vs
collector resources
vs
backend cost

For high-value workflows, deterministic or higher sampling rates may be justified.
For high-volume low-risk events, aggressive sampling may be unavoidable.

Dead-letter queues must preserve forensic context

A dead-letter queue is not merely a disposal mechanism.
It is an incident evidence store.
When a message reaches a DLQ, engineers should still be able to determine:
Where was it created?

Which workflow created it?

How many processing attempts occurred?

Which services handled it?

What was the last failure?

Which deployment version processed it?

Can the original trace or related traces be located?

That means the metadata required for correlation must survive redrive.
A DLQ message that contains only:
{
"orderId": "ORD-74219"
}

may be functionally sufficient for replay but operationally weak for diagnosis.
Design message metadata as part of the observability contract, not as an afterthought.

Watch the semantic convention boundary

There is an important implementation caveat.
OpenTelemetry's messaging semantic conventions are currently marked as Development, and the project documents migration considerations for instrumentation using earlier convention versions.
That means platform teams should not scatter assumptions about attribute names across hundreds of dashboards and alerts without version discipline.
Create an internal telemetry contract.
For example:
Required resource identity
Required business correlation fields
Approved messaging attributes
Trace propagation format
Sampling rules
Sensitive-field policy
Collector compatibility version

Then manage changes to that contract like an API.
Otherwise an instrumentation library upgrade can silently break dashboards, queries, and incident tooling.

A stronger incident architecture

Putting the pieces together:
┌─────────────────────┐
│ Client │
└──────────┬──────────┘
|
v
┌─────────────────────┐
│ orders-api │
│ on EKS │
└──────────┬──────────┘
|
inject trace context
+ business identity
|
v
┌─────────────────────┐
│ SQS │
│ order-events │
└──────────┬──────────┘
|
extract context
|
v
┌─────────────────────┐
│ order-worker │
│ on EKS │
└──────────┬──────────┘
|
span link to message
creation context
|
+--------------+--------------+
| |
v v
payment-api fulfillment-events
|
v
fulfillment-worker

All workloads
|
| OTLP
v
OpenTelemetry / ADOT Collectors
|
+------> Traces
|
+------> Metrics
|
+------> Logs

The architecture works because correlation exists at several levels:
Trace context
-> technical causality

Span links
-> asynchronous causality

Message identity
-> transport identity

Business correlation
-> domain causality

Kubernetes resource attributes
-> execution identity

Metrics
-> population-level impact

No single identifier replaces the others.

The lesson

Good asynchronous observability is not about drawing one enormous distributed trace.
It is about preserving enough relationships that an engineer can reconstruct what caused what.
The critical design decisions are therefore not merely:
Which tracing backend should we buy?

They are:
Where is context created?

How does it cross messaging boundaries?

When should spans use parents versus links?

How are retries represented?

Which business identifiers survive every hop?

How do logs expose trace identity?

How do Kubernetes resources identify execution?

What telemetry survives dead-lettering?

Where is sampling allowed to discard context?

OpenTelemetry gives us the primitives. Kubernetes gives us dynamic execution infrastructure. AWS services such as Amazon EKS, SQS, ADOT, and X-Ray provide useful integration points.
But none of them automatically preserves causality for the application.
That remains an architecture responsibility.
The standard for a production tracing system should therefore be simple:
Given one failed business operation, can an engineer move backward from the failure to the message that triggered it, to the workload that processed it, to the upstream producer that created it, and forward again through every retry and downstream consequence?
If the answer is no, the system may be collecting distributed traces.
It does not yet have distributed observability.
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article