Tracing an Incident Across Kubernetes, OpenTelemetry and Async Messaging Without Losing Causality
Distributed tracing looks straightforward when the request path is synchronous. A client calls an API. The API calls another service. That service queries a database. Trace context travels through HTTP or gRPC headers, and the resulting span tree roughly resembles the execution path. Production systems rarely remain that simple.
A request enters Kubernetes, publishes an event, returns successfully, waits in a queue, is consumed by another workload, triggers another message, retries after a transient failure, and eventually reaches a downstream service. By the time an engineer investigates the incident, the original HTTP request may have completed minutes earlier.
The difficult observability problem is no longer collecting spans.
It is preserving causality.
A useful tracing architecture must answer:
Which upstream event caused this downstream work, even when execution crossed queues, processes, retries, batches, and time boundaries?
That requires treating asynchronous tracing differently from synchronous RPC tracing.
The synchronous mental model breaks at the queue
Consider an application running on Amazon EKS:
Client | vAPI Gateway / Load Balancer | vorders-api | +------> PostgreSQL | +------> SQS: order-events | v order-worker | +------> payment-api | +------> SQS: fulfillment-events | v fulfillment-worker
The first half is easy to trace.
HTTP request | +-- orders-api | +-- database query | +-- SQS send
The problem begins after the message enters SQS.
The producer may terminate.
The message may wait.
A consumer may receive several messages in one poll.
Processing may fail.
The message may become visible again.
Another pod may receive it.
Eventually it may be sent to a dead-letter queue.
Trying to represent this entire lifecycle as a traditional parent-child span tree creates misleading traces.
The queue is not simply another RPC hop.
It is a temporal and execution boundary.
Trace context is not business identity
One of the most common mistakes in asynchronous observability is assuming that
trace_id solves correlation.It does not.
A trace identifier represents telemetry context. A business identifier represents domain identity.
For an order-processing system, useful identifiers might include:
trace_idorder_idmessage_idpayment_idworkflow_idcorrelation_id
These values answer different questions.
trace_id :Which telemetry operations belong to this distributed execution?
order_id :Which business transaction is affected?
message_id :Which specific transport message was processed?
workflow_id :Which long-running business workflow owns this operation?
Do not overload one identifier to perform all four jobs.
A production investigation often starts from something other than a trace ID:
Customer reports order ORD-74219 is stuck.
The observability system must therefore support:
order_id | vlogs / traces / events | vmessage_id | vconsumer execution | vdownstream effects
This is why domain correlation attributes should accompany trace context.
Propagate context with the message
OpenTelemetry explicitly recognizes that messaging requires context propagation beyond transport-level tracing.
Its messaging semantic conventions distinguish message creation, sending, receiving, processing, and settlement. They also describe a message creation context that producers should propagate so consumer processing can be correlated with the operation that created the message.
For a producer, the conceptual flow is:
Current application context | vCreate / Send span | vInject trace context | vMessage metadata | vQueue
For a consumer:
Queue | vReceive message | vExtract context | vCreate processing span | vApplication logic
The exact carrier depends on the messaging technology.
Amazon SQS supports message attributes for application metadata and also exposes message system attributes, including
AWSTraceHeader , for AWS tracing integration.A message envelope might conceptually carry:
{ "body": { "orderId": "ORD-74219", "eventType": "OrderAccepted" }, "metadata": { "traceparent": "00-...", "correlation_id": "workflow-92814" }}
The important architectural rule is:
Context must cross the asynchronous boundary explicitly.
If the producer creates a trace and the consumer independently creates another trace, both traces may be individually correct while the end-to-end causal chain is lost.
Parent-child is not always the right relationship
Suppose a consumer polls ten SQS messages.
Those messages may originate from ten independent traces.
What should be the parent of the consumer's receive span?
A span can have only one parent.
Choosing one message arbitrarily would imply that the other nine messages were caused by the first.
They were not.
OpenTelemetry addresses this through span links.
Conceptually:
Producer Trace A ----\Producer Trace B -----\Producer Trace C ------> Consumer Process SpanProducer Trace D -----/ ^Producer Trace E ----/ | span links
The consumer span can have its own execution context while linking to the creation contexts of the messages it processes.
OpenTelemetry's current messaging conventions use span links as the default correlation mechanism for producer and consumer contexts because this works across different messaging models and, critically, supports batching where one consumer operation relates to multiple messages.
For single-message processing, using the message creation context as the parent can sometimes be reasonable.
For batch processing, links are usually the more accurate model.
This distinction matters during incident analysis.
Tracing systems should represent actual causal relationships, not merely produce aesthetically pleasing trees.
Separate receive from process
Another common instrumentation mistake is treating message receipt and message processing as the same operation.
They are different.
For a pull-based consumer:
ReceiveMessage | vMessage available to application | vProcessMessage | vDelete / acknowledge
The receive operation measures interaction with the broker.
The process operation measures application work.
OpenTelemetry's messaging model distinguishes
receive , process , and settle operations. It recommends CONSUMER span kind for processing spans, while receive and settle operations have different semantics.This separation becomes valuable during incidents.
Imagine:
Receive latency: 40 msProcessing latency: 18 sSettlement latency: 30 ms
The broker is not the bottleneck.
Now consider:
Receive latency: 12 sProcessing latency: 300 ms
That points investigation in a very different direction.
If both are represented as one span called
consume , that distinction disappears.Retries are new executions, not invisible continuations
Retries are where many trace models become misleading.
Consider:
Message M1 | vworker-pod-a | +--> processing fails | vvisibility timeout | vworker-pod-b | +--> processing succeeds
These are two execution attempts.
They should not look like one uninterrupted operation.
A useful model is:
Message creation context | +------ link ------> Process attempt 1 | +------ link ------> Process attempt 2
Each attempt should record enough transport information to explain what happened.
For SQS,
ApproximateReceiveCount can help identify repeated delivery attempts. AWS documents this as a message system attribute available from ReceiveMessage .At the application layer, useful telemetry might include:
messaging.message.idmessaging.destination.namemessaging.operation.typedelivery_attemptworkflow_idorder_id
But be disciplined with attributes.
Do not blindly attach entire message bodies to spans.
That creates cardinality, cost, privacy, and security problems.
Kubernetes identity must survive pod churn
Tracing the application is only half the problem.
During an incident, engineers also need to understand where an operation executed.
Kubernetes workloads are ephemeral.
A pod may disappear before anyone investigates.
Therefore telemetry should carry resource identity such as:
service.nameservice.versionservice.instance.idk8s.cluster.namek8s.namespace.namek8s.deployment.namek8s.pod.namek8s.node.name
The distinction between service identity and pod identity is important.
Dashboards and SLOs usually care about:
service.name = fulfillment-worker
Incident debugging may care about:
pod = fulfillment-worker-76c87f9b7c-m4r8pnode = ip-10-0-42-17
If every query begins with pod names, observability becomes fragile because Kubernetes constantly replaces them.
Start with the stable workload identity and drill into ephemeral infrastructure identity only when necessary.
Amazon EKS supports the AWS Distro for OpenTelemetry, and AWS documents ADOT as an AWS-supported OpenTelemetry distribution that can send telemetry from EKS workloads to destinations including CloudWatch and AWS X-Ray.
A typical collection architecture is:
Application Pods | | OTLP vOpenTelemetry Collector | +------> trace backend | +------> metrics backend | +------> logs pipeline
The collector should be treated as telemetry infrastructure, not as the source of application semantics.
Instrumentation still needs to establish the right causal relationships.
Correlate logs with traces deliberately
Distributed traces rarely contain enough detail to diagnose every failure.
Logs remain essential.
The useful pattern is not:
traces over herelogs over there
It is:
trace_idspan_idservice.namebusiness correlation fields
included in structured logs.
Then an engineer can move from:
failed consumer span
to:
all logs emitted by this processing attempt
and then to:
all activity for order ORD-74219
These are different navigation paths.
A structured log might look like:
{ "severity": "ERROR", "service": "payment-worker", "trace_id": "...", "span_id": "...", "order_id": "ORD-74219", "message_id": "...", "event": "payment_authorization_failed"}
Avoid using free-text parsing as your primary correlation strategy.
Observability systems work much better when identity is structured at emission time.
Metrics explain scope, traces explain examples
There is another trap: trying to solve queue incidents entirely with tracing.
Tracing is not a replacement for queue metrics.
Suppose consumer latency suddenly increases.
The investigation should begin with aggregate signals:
queue depthage of oldest messagepublish rateconsume rateprocessing latencyerror rateretry ratedead-letter volumeconsumer replicasCPU / memory saturation
Those signals tell you whether the problem affects one message or the entire system.
Then traces answer:
What happened to representative messages?
The observability flow should be:
Alert | vMetrics establish scope | vSelect affected service / queue | vExemplar or trace search | vFollow causal chain | vLogs provide detailed evidence
Trying to query millions of traces to discover whether a queue is backing up is expensive and unnecessary.
Metrics identify the population.
Traces explain individual paths through that population.
Sampling can destroy the exact trace you need
Asynchronous systems complicate sampling.
A producer may look healthy when its trace begins:
POST /orders -> 200
The actual failure may happen three minutes later:
fulfillment-worker -> repeated processing failure
If sampling decisions are finalized too early, the producer side of the causal chain may already have been discarded.
This is where tail-based sampling can become useful.
Instead of deciding exclusively when a trace begins, a collector can retain telemetry based on outcomes observed later.
Potential policies include retaining:
errorshigh latencyselected critical workflowsrare operationsspecific service classes
But tail sampling has costs.
Collectors need enough capacity to buffer trace information while decisions are pending. High-volume systems can create significant memory and operational overhead.
There is no universal sampling configuration.
The architecture must balance:
diagnostic completeness vstelemetry volume vscollector resources vsbackend cost
For high-value workflows, deterministic or higher sampling rates may be justified.
For high-volume low-risk events, aggressive sampling may be unavoidable.
Dead-letter queues must preserve forensic context
A dead-letter queue is not merely a disposal mechanism.
It is an incident evidence store.
When a message reaches a DLQ, engineers should still be able to determine:
Where was it created?
Which workflow created it?
How many processing attempts occurred?
Which services handled it?
What was the last failure?
Which deployment version processed it?
Can the original trace or related traces be located?
That means the metadata required for correlation must survive redrive.
A DLQ message that contains only:
{ "orderId": "ORD-74219"}
may be functionally sufficient for replay but operationally weak for diagnosis.
Design message metadata as part of the observability contract, not as an afterthought.
Watch the semantic convention boundary
There is an important implementation caveat.
OpenTelemetry's messaging semantic conventions are currently marked as Development, and the project documents migration considerations for instrumentation using earlier convention versions.
That means platform teams should not scatter assumptions about attribute names across hundreds of dashboards and alerts without version discipline.
Create an internal telemetry contract.
For example:
Required resource identityRequired business correlation fieldsApproved messaging attributesTrace propagation formatSampling rulesSensitive-field policyCollector compatibility version
Then manage changes to that contract like an API.
Otherwise an instrumentation library upgrade can silently break dashboards, queries, and incident tooling.
A stronger incident architecture
Putting the pieces together:
┌─────────────────────┐ │ Client │ └──────────┬──────────┘ | v ┌─────────────────────┐ │ orders-api │ │ on EKS │ └──────────┬──────────┘ | inject trace context + business identity | v ┌─────────────────────┐ │ SQS │ │ order-events │ └──────────┬──────────┘ | extract context | v ┌─────────────────────┐ │ order-worker │ │ on EKS │ └──────────┬──────────┘ | span link to message creation context | +--------------+--------------+ | | v v payment-api fulfillment-events | v fulfillment-worker
All workloads | | OTLP vOpenTelemetry / ADOT Collectors | +------> Traces | +------> Metrics | +------> Logs
The architecture works because correlation exists at several levels:
Trace context -> technical causality
Span links -> asynchronous causality
Message identity -> transport identity
Business correlation -> domain causality
Kubernetes resource attributes -> execution identity
Metrics -> population-level impact
No single identifier replaces the others.
The lesson
Good asynchronous observability is not about drawing one enormous distributed trace.
It is about preserving enough relationships that an engineer can reconstruct what caused what.
The critical design decisions are therefore not merely:
Which tracing backend should we buy?
They are:
Where is context created?
How does it cross messaging boundaries?
When should spans use parents versus links?
How are retries represented?
Which business identifiers survive every hop?
How do logs expose trace identity?
How do Kubernetes resources identify execution?
What telemetry survives dead-lettering?
Where is sampling allowed to discard context?
OpenTelemetry gives us the primitives. Kubernetes gives us dynamic execution infrastructure. AWS services such as Amazon EKS, SQS, ADOT, and X-Ray provide useful integration points.
But none of them automatically preserves causality for the application.
That remains an architecture responsibility.
The standard for a production tracing system should therefore be simple:
Given one failed business operation, can an engineer move backward from the failure to the message that triggered it, to the workload that processed it, to the upstream producer that created it, and forward again through every retry and downstream consequence?
If the answer is no, the system may be collecting distributed traces.
It does not yet have distributed observability.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article