AWS Builder Center
Following the White Rabbit: Chasing One Trace Through Five AWS Services

Following the White Rabbit: Chasing One Trace Through Five AWS Services

Learn how in-process ADOT instrumentation, a stitched parent span across the SQS boundary, and CloudWatch Transaction Search join a Lambda, SQS, ECS Fargate, Bedrock AgentCore, and S3 pipeline into one unbroken trace.

You can get a single, continuous distributed trace across a full agentic pipeline on AWS, and the one place it usually breaks is the asynchronous SQS hop. This post shows how I stitched Lambda, SQS, ECS Fargate, Amazon Bedrock AgentCore, and S3 into one unbroken trace in Amazon CloudWatch, and how I bridged the message-queue boundary that normally breaks the trace into two.
The demo processes a CVE. A producer Lambda enqueues a CVE id to SQS. An ECS Fargate worker polls the queue, enriches the CVE with context from DynamoDB, and invokes a Bedrock AgentCore runtime running a Strands agent. The agent summarizes the security impact and writes the summary to S3. Every hop emits OpenTelemetry spans that carry the same trace id, so the whole flow reads as one trace with one waterfall.

The architecture

Lambda producer sends a CVE id to SQS, an ECS Fargate worker polls and enriches from DynamoDB, invokes the Bedrock AgentCore runtime, and the agent writes the impact summary to S3, with one trace id flowing across every hop.

AgentCore observability is agent-inward. Out of the box it traces the runtime internals, meaning model inference, tool calls, memory, and the agent loop, plus any boto3 calls the agent makes downstream such as the S3 write. It leaves the upstream Lambda, SQS, and ECS hops for you to instrument. A true end-to-end trace is a distributed-tracing job assembled with per-hop instrumentation and one deliberate bridge at the SQS boundary.

Traces and spans, briefly

A trace is the complete story of one request as it moves through the system, identified by a single trace id. In OpenTelemetry that id is a 16-byte value, written as 32 hex characters. Every unit of work for that request carries the same id, and that shared id is why the five services appear as one connected picture.
A span is one unit of work inside a trace. It records a name, a span id, a parent span id that builds the tree, timestamps, a status, a span kind, and attributes. Span kind is the detail that decides how the SQS hop behaves. A PRODUCER span sends a message that something processes later. A CONSUMER span receives that message. The Lambda send is a PRODUCER span, and the ECS poll is a CONSUMER span.

Why the SQS hop breaks the trace

OpenTelemetry recommends a span link for messaging fan-out. One consumer poll can pull many messages from many producers, so a loose association is the sensible default when a queue decouples sender from receiver. The consequence is that an asynchronous SQS boundary produces linked traces by default, and the service map shows the pipeline joined by a link at SQS.
That default is correct for batching workloads. It also explains why the pipeline appears as separate traces right at the SQS handoff you most want to follow end to end.

The decision: stitch the trace

My worker processes exactly one message per unit of work, so I made a deliberate call to stitch the trace across SQS.
The root lambda.handler span leads to the SQS SendMessage span, which becomes the parent of ecs.worker.process. That worker span branches into worker.enrich_cve (DynamoDB) and agentcore.invoke, which parents agent.handle inside AgentCore, which in turn parents model.inference and the S3 PutObject

The producer injects a W3C  traceparent  into the SQS message attributes on send. The worker extracts that context and starts its CONSUMER span with the producer's span as its actual parent. The same trace id flows forward, the parent id points at the Lambda's SQS span, and the tree stays whole through ECS, AgentCore, and S3.
The payoff is a single waterfall. Because every span shares one id and one parent chain, Transaction Search renders the whole pipeline as one timeline, and you can read latency attributed hop to hop across the entire agentic pipeline: how long the message waited in the queue, how long DynamoDB enrichment took, and that the model call owns most of the wall clock. A linked edge scatters that timeline across separate traces, so the end-to-end latency picture takes manual assembly. Stitching the parent id is what buys the at-a-glance view.
This carries a tradeoff worth stating plainly. Stitching a single parent is clean while each poll handles one message. High-volume systems tend to batch, so a design that grows will likely reach the point where one poll pulls many messages from many producers, and a single stitched parent turns into a fiction. The goal survives that shift even though the mechanism changes. At batch scale you keep per-message hop-to-hop latency by giving each message its own continued context and linking the batch, or by carrying the producer span id as an attribute and rebuilding the timeline in queries. The code that does the stitching carries a comment pointing back to this decision so a future maintainer sees the constraint before changing it.

The producer: inject the context

The Lambda runs under the ADOT managed layer with in-process auto-instrumentation. The one setting people miss is the propagator list. The default Lambda ADOT setup uses only the X-Ray propagator, so you have to add  tracecontext  to emit an actual W3C  traceparent .
environment={
"AWS_LAMBDA_EXEC_WRAPPER": "/opt/otel-instrument", # turns on in-process ADOT
"OTEL_PROPAGATORS": "tracecontext,baggage,xray", # emit a W3C traceparent
"OTEL_SERVICE_NAME": "producer-lambda",
}
The handler opens a PRODUCER span and writes the context onto the message.
with tracer.start_as_current_span("producer.enqueue", kind=SpanKind.PRODUCER) as span:
carrier = {}
propagate.inject(carrier) # writes W3C traceparent into carrier
traceparent = carrier.get("traceparent", "")

message_attributes = {
"correlation_id": {"DataType": "String", "StringValue": correlation_id},
}
if traceparent:
message_attributes["traceparent"] = {"DataType": "String", "StringValue": traceparent}

sqs.send_message(QueueUrl=QUEUE_URL, MessageBody=json.dumps(body),
MessageAttributes=message_attributes)

The worker: make the parent

The worker reads the  traceparent  attribute (falling back to the X-Ray  AWSTraceHeader  that SQS preserves) and passes the extracted context straight into  start_as_current_span . The  context=  argument is what makes the producer's span the direct parent.
def extract_parent_context(message):
attrs = message.get("MessageAttributes") or {}
tp = (attrs.get("traceparent") or {}).get("StringValue")
if tp:
return _w3c.extract(carrier={"traceparent": tp})
xray = (message.get("Attributes") or {}).get("AWSTraceHeader")
if xray and AwsXRayPropagator is not None:
return AwsXRayPropagator().extract(carrier={"X-Amzn-Trace-Id": xray})
return otel_context.get_current()

parent_ctx = extract_parent_context(message)
with tracer.start_as_current_span(
"ecs.worker.process", context=parent_ctx, kind=SpanKind.CONSUMER
) as span:
span.set_attribute("cve.id", cve_id)
with tracer.start_as_current_span("worker.enrich_cve") as enrich:
cve_context = get_cve_context(cve_id, cve_table) # DynamoDB GetItem
enrich.set_attribute("cve.found", cve_context.get("found", False))
invoke_agent(agent_client, build_agent_invocation(enriched, correlation_id))
Two small gotchas cost me time here. The idle long-poll emits a span on every empty  ReceiveMessage , and each one wants to become its own root trace, so I wrapped the poll in  suppress_instrumentation()  and dropped the residual poll spans in the collector. I also learned to leave the AgentCore invoke free of manual trace headers. Passing  traceParent  as an explicit  InvokeAgentRuntime  parameter collides with the OTel botocore header injection and breaks the SigV4 signature. OTel auto-propagation plus a payload backstop does the job.

The agent: honor the inbound context

The AgentCore container turns on built-in observability and shares the same propagator settings. It honors the inbound trace context and parents its spans to the caller.
ENV AGENT_OBSERVABILITY_ENABLED=true \
OTEL_PROPAGATORS=tracecontext,baggage,xray \
OTEL_PYTHON_ID_GENERATOR=xray \
OTEL_SERVICE_NAME=agentcore-agent
ENTRYPOINT ["opentelemetry-instrument", "python", "app.py"]
Inside the entrypoint, a SERVER span wraps the Strands call, and the Strands SDK auto-traces the model inference and token usage below it.
@app.entrypoint
def invoke(payload):
parent_ctx = _parent_context_from_payload(payload) # payload backstop
with tracer.start_as_current_span("agent.handle", context=parent_ctx,
kind=SpanKind.SERVER) as span:
span.set_attribute("cve.id", cve_id)
result = agent(prompt) # chat / ConverseStream spans
s3.put_object(Bucket=RESULTS_BUCKET, Key=f"results/{cve_id}.json", Body=...)

Where the spans land

Every service exports OpenTelemetry spans to X-Ray through CloudWatch Transaction Search, which stores them in the  aws/spans  log group. That shared store is what lets spans from different services stitch into one trace. Transaction Search is an account-wide setting you enable once with  xray update-trace-segment-destination --destination CloudWatchLogs  and an X-Ray resource policy on the log group. In CDK it is expressible with native L1 resources, and I gate its creation so a second stack in the same account does not fight over the singleton.
The AgentCore runtime itself uses in-process ADOT with no collector, because the ADOT Collector is unsupported for AgentCore agent observability. The standalone ECS worker is a different story. It needs a real export path, so it runs an ADOT Collector sidecar that relays OTLP to X-Ray, re-hosted into private ECR because the isolated subnets cannot reach the public registry.
Here are a couple of screenshots from CloudWatch.

The payoff

Here is one captured trace for  CVE-2014-0160 , better known as Heartbleed. It contains sixteen spans across three services, all under one trace id.
producer-lambda SERVER 156.2 ms (root)
└─ producer.enqueue PRODUCER 156.1 ms messaging.system=aws_sqs
└─ SQS.SendMessage CLIENT 153.8 ms
ecs.worker.process CONSUMER 14983.0 ms parent = SQS send span <-- stitched across SQS
├─ worker.enrich_cve INTERNAL 197.9 ms cve.cvss_score=7.5
│ └─ DynamoDB.GetItem CLIENT 48.7 ms
├─ agentcore.invoke CLIENT 14784.7 ms
├─ agent.handle SERVER 12361.6 ms s3.key=results/CVE-2014-0160.json
│ └─ chat global.anthropic.claude-sonnet CLIENT 12276.4 ms ConverseStream, tokens total=693
│ └─ S3.PutObject CLIENT 76.7 ms
The  ecs.worker.process  span is parented to the producer's SQS send span even though SQS decoupled the two, and every downstream span shares the one id. The agent's model call dominates latency at about 12.3 seconds of the roughly 15-second total, which is the kind of insight a single trace hands you for free. You can open the same trace in the CloudWatch GenAI Observability dashboard to see model inference, tool calls, and the 233-in, 460-out token split for that invocation.
A screenshot from CloudWatch GenAI Observability
A screenshot from Cloudwatch GenAI Observability annotated for reference

An expanded view of the trajectory, the span drawn as a service/dependency graph instead of a timeline.
spans drawn as a service/dependency graph instead of a timeline


What to take away

The single trace rests on three moves. Every hop runs in-process ADOT SDK instrumentation and shares the  tracecontext  propagator so a real  traceparent  exists to pass along. The worker stitches the trace by making the producer's span the direct parent of its CONSUMER span. CloudWatch Transaction Search routes every span to one log group so the picture assembles itself. With those moves in place, one trace id gives you the whole pipeline in a single view.

Get the code

The full demo is on GitHub at  sr8494/agentcore-observability-adot . It is a CDK Python app that deploys the whole pipeline, with the producer, worker, and agent instrumentation shown above, plus scripts to seed the CVE table, generate traffic, and verify the trace end to end.
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article