
Following the White Rabbit: Chasing One Trace Through Five AWS Services
Learn how in-process ADOT instrumentation, a stitched parent span across the SQS boundary, and CloudWatch Transaction Search join a Lambda, SQS, ECS Fargate, Bedrock AgentCore, and S3 pipeline into one unbroken trace.
You can get a single, continuous distributed trace across a full agentic pipeline on AWS, and the one place it usually breaks is the asynchronous SQS hop. This post shows how I stitched Lambda, SQS, ECS Fargate, Amazon Bedrock AgentCore, and S3 into one unbroken trace in Amazon CloudWatch, and how I bridged the message-queue boundary that normally breaks the trace into two.
The demo processes a CVE. A producer Lambda enqueues a CVE id to SQS. An ECS Fargate worker polls the queue, enriches the CVE with context from DynamoDB, and invokes a Bedrock AgentCore runtime running a Strands agent. The agent summarizes the security impact and writes the summary to S3. Every hop emits OpenTelemetry spans that carry the same trace id, so the whole flow reads as one trace with one waterfall.
The architecture

AgentCore observability is agent-inward. Out of the box it traces the runtime internals, meaning model inference, tool calls, memory, and the agent loop, plus any boto3 calls the agent makes downstream such as the S3 write. It leaves the upstream Lambda, SQS, and ECS hops for you to instrument. A true end-to-end trace is a distributed-tracing job assembled with per-hop instrumentation and one deliberate bridge at the SQS boundary.
Traces and spans, briefly
A trace is the complete story of one request as it moves through the system, identified by a single trace id. In OpenTelemetry that id is a 16-byte value, written as 32 hex characters. Every unit of work for that request carries the same id, and that shared id is why the five services appear as one connected picture.
A span is one unit of work inside a trace. It records a name, a span id, a parent span id that builds the tree, timestamps, a status, a span kind, and attributes. Span kind is the detail that decides how the SQS hop behaves. A PRODUCER span sends a message that something processes later. A CONSUMER span receives that message. The Lambda send is a PRODUCER span, and the ECS poll is a CONSUMER span.
Why the SQS hop breaks the trace
OpenTelemetry recommends a span link for messaging fan-out. One consumer poll can pull many messages from many producers, so a loose association is the sensible default when a queue decouples sender from receiver. The consequence is that an asynchronous SQS boundary produces linked traces by default, and the service map shows the pipeline joined by a link at SQS.
That default is correct for batching workloads. It also explains why the pipeline appears as separate traces right at the SQS handoff you most want to follow end to end.
The decision: stitch the trace
My worker processes exactly one message per unit of work, so I made a deliberate call to stitch the trace across SQS.

The producer injects a W3C
traceparent into the SQS message attributes on send. The worker extracts that context and starts its CONSUMER span with the producer's span as its actual parent. The same trace id flows forward, the parent id points at the Lambda's SQS span, and the tree stays whole through ECS, AgentCore, and S3.The payoff is a single waterfall. Because every span shares one id and one parent chain, Transaction Search renders the whole pipeline as one timeline, and you can read latency attributed hop to hop across the entire agentic pipeline: how long the message waited in the queue, how long DynamoDB enrichment took, and that the model call owns most of the wall clock. A linked edge scatters that timeline across separate traces, so the end-to-end latency picture takes manual assembly. Stitching the parent id is what buys the at-a-glance view.
This carries a tradeoff worth stating plainly. Stitching a single parent is clean while each poll handles one message. High-volume systems tend to batch, so a design that grows will likely reach the point where one poll pulls many messages from many producers, and a single stitched parent turns into a fiction. The goal survives that shift even though the mechanism changes. At batch scale you keep per-message hop-to-hop latency by giving each message its own continued context and linking the batch, or by carrying the producer span id as an attribute and rebuilding the timeline in queries. The code that does the stitching carries a comment pointing back to this decision so a future maintainer sees the constraint before changing it.
The producer: inject the context
The Lambda runs under the ADOT managed layer with in-process auto-instrumentation. The one setting people miss is the propagator list. The default Lambda ADOT setup uses only the X-Ray propagator, so you have to add
tracecontext to emit an actual W3C traceparent .environment={ "AWS_LAMBDA_EXEC_WRAPPER": "/opt/otel-instrument", # turns on in-process ADOT "OTEL_PROPAGATORS": "tracecontext,baggage,xray", # emit a W3C traceparent "OTEL_SERVICE_NAME": "producer-lambda",}The handler opens a PRODUCER span and writes the context onto the message.
with tracer.start_as_current_span("producer.enqueue", kind=SpanKind.PRODUCER) as span: carrier = {} propagate.inject(carrier) # writes W3C traceparent into carrier traceparent = carrier.get("traceparent", "")
message_attributes = { "correlation_id": {"DataType": "String", "StringValue": correlation_id}, } if traceparent: message_attributes["traceparent"] = {"DataType": "String", "StringValue": traceparent}
sqs.send_message(QueueUrl=QUEUE_URL, MessageBody=json.dumps(body), MessageAttributes=message_attributes)The worker: make the parent
The worker reads the
traceparent attribute (falling back to the X-Ray AWSTraceHeader that SQS preserves) and passes the extracted context straight into start_as_current_span . The context= argument is what makes the producer's span the direct parent.def extract_parent_context(message): attrs = message.get("MessageAttributes") or {} tp = (attrs.get("traceparent") or {}).get("StringValue") if tp: return _w3c.extract(carrier={"traceparent": tp}) xray = (message.get("Attributes") or {}).get("AWSTraceHeader") if xray and AwsXRayPropagator is not None: return AwsXRayPropagator().extract(carrier={"X-Amzn-Trace-Id": xray}) return otel_context.get_current()
parent_ctx = extract_parent_context(message)with tracer.start_as_current_span( "ecs.worker.process", context=parent_ctx, kind=SpanKind.CONSUMER) as span: span.set_attribute("cve.id", cve_id) with tracer.start_as_current_span("worker.enrich_cve") as enrich: cve_context = get_cve_context(cve_id, cve_table) # DynamoDB GetItem enrich.set_attribute("cve.found", cve_context.get("found", False)) invoke_agent(agent_client, build_agent_invocation(enriched, correlation_id))Two small gotchas cost me time here. The idle long-poll emits a span on every empty
ReceiveMessage , and each one wants to become its own root trace, so I wrapped the poll in suppress_instrumentation() and dropped the residual poll spans in the collector. I also learned to leave the AgentCore invoke free of manual trace headers. Passing traceParent as an explicit InvokeAgentRuntime parameter collides with the OTel botocore header injection and breaks the SigV4 signature. OTel auto-propagation plus a payload backstop does the job.The agent: honor the inbound context
The AgentCore container turns on built-in observability and shares the same propagator settings. It honors the inbound trace context and parents its spans to the caller.
ENV AGENT_OBSERVABILITY_ENABLED=true \ OTEL_PROPAGATORS=tracecontext,baggage,xray \ OTEL_PYTHON_ID_GENERATOR=xray \ OTEL_SERVICE_NAME=agentcore-agentENTRYPOINT ["opentelemetry-instrument", "python", "app.py"]Inside the entrypoint, a SERVER span wraps the Strands call, and the Strands SDK auto-traces the model inference and token usage below it.
@app.entrypointdef invoke(payload): parent_ctx = _parent_context_from_payload(payload) # payload backstop with tracer.start_as_current_span("agent.handle", context=parent_ctx, kind=SpanKind.SERVER) as span: span.set_attribute("cve.id", cve_id) result = agent(prompt) # chat / ConverseStream spans s3.put_object(Bucket=RESULTS_BUCKET, Key=f"results/{cve_id}.json", Body=...)Where the spans land
Every service exports OpenTelemetry spans to X-Ray through CloudWatch Transaction Search, which stores them in the
aws/spans log group. That shared store is what lets spans from different services stitch into one trace. Transaction Search is an account-wide setting you enable once with xray update-trace-segment-destination --destination CloudWatchLogs and an X-Ray resource policy on the log group. In CDK it is expressible with native L1 resources, and I gate its creation so a second stack in the same account does not fight over the singleton.The AgentCore runtime itself uses in-process ADOT with no collector, because the ADOT Collector is unsupported for AgentCore agent observability. The standalone ECS worker is a different story. It needs a real export path, so it runs an ADOT Collector sidecar that relays OTLP to X-Ray, re-hosted into private ECR because the isolated subnets cannot reach the public registry.
Here are a couple of screenshots from CloudWatch.
The payoff
Here is one captured trace for
CVE-2014-0160 , better known as Heartbleed. It contains sixteen spans across three services, all under one trace id.producer-lambda SERVER 156.2 ms (root)└─ producer.enqueue PRODUCER 156.1 ms messaging.system=aws_sqs └─ SQS.SendMessage CLIENT 153.8 msecs.worker.process CONSUMER 14983.0 ms parent = SQS send span <-- stitched across SQS├─ worker.enrich_cve INTERNAL 197.9 ms cve.cvss_score=7.5│ └─ DynamoDB.GetItem CLIENT 48.7 ms├─ agentcore.invoke CLIENT 14784.7 ms├─ agent.handle SERVER 12361.6 ms s3.key=results/CVE-2014-0160.json│ └─ chat global.anthropic.claude-sonnet CLIENT 12276.4 ms ConverseStream, tokens total=693│ └─ S3.PutObject CLIENT 76.7 msThe
ecs.worker.process span is parented to the producer's SQS send span even though SQS decoupled the two, and every downstream span shares the one id. The agent's model call dominates latency at about 12.3 seconds of the roughly 15-second total, which is the kind of insight a single trace hands you for free. You can open the same trace in the CloudWatch GenAI Observability dashboard to see model inference, tool calls, and the 233-in, 460-out token split for that invocation.A screenshot from CloudWatch GenAI Observability

An expanded view of the trajectory, the span drawn as a service/dependency graph instead of a timeline.

What to take away
The single trace rests on three moves. Every hop runs in-process ADOT SDK instrumentation and shares the
tracecontext propagator so a real traceparent exists to pass along. The worker stitches the trace by making the producer's span the direct parent of its CONSUMER span. CloudWatch Transaction Search routes every span to one log group so the picture assembles itself. With those moves in place, one trace id gives you the whole pipeline in a single view.Get the code
The full demo is on GitHub at sr8494/agentcore-observability-adot . It is a CDK Python app that deploys the whole pipeline, with the producer, worker, and agent instrumentation shown above, plus scripts to seed the CVE table, generate traffic, and verify the trace end to end.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article