AWS Builder Center

Builder Center

End-to-End Observability for Agent-to-Agent Workflows with CloudWatch Omni

Discover how to stitch a single logical request across many independent Bedrock agents, spanning multiple AWS accounts, into one unified trace in CloudWatch Omni, then attribute the token cost of that request back to a team or project using CUR 2.0.

Terri Johnson

Senior Solutions Architect
As organizations move agentic AI from prototype to production, a single user request rarely stays inside one agent, one process, or even one AWS account. A customer-service orchestrator in the application account calls a research sub-agent in a data account, which calls a pricing tool backed by a third agent in a finance account. Each hop makes its own LLM calls and tool invocations. When latency spikes or a bill arrives, the question is always the same: what did this one request actually do, and what did it cost?
This post walks through building end-to-end observability for Agent-to-Agent (A2A) Bedrock workflows using Amazon CloudWatch Omni . We cover three things that have to work together: stitching every agent hop into a single distributed trace, centralizing those traces across accounts into one monitoring account, and attributing the cost of a logical request back across accounts with CUR 2.0 and a cost allocation tagging strategy.

Architecture at a glance

The diagram below shows the full picture: a logical request fans out across three accounts, with each agent deployed on Amazon Bedrock AgentCore Runtime  and exporting OTLP spans carrying the same W3C trace context, a CloudWatch centralization rule  centralizes those spans into one destination account's Omni space, and CUR 2.0 carries the tagged cost data that reconciles against the token counts on the spans. The instrumentation and trace-context code below is deployment-target agnostic — it runs the same whether the agent is a standalone process, as shown in the walkthrough, or packaged as an AgentCore Runtime service.
Click the diagram to view it full size.
 
 

What is CloudWatch Omni?

Amazon CloudWatch Omni  is an observability experience built on OpenTelemetry  (OTel). Telemetry you already send to CloudWatch shows up in Omni with nothing to reconfigure, and any other workload you instrument with OpenTelemetry can send its telemetry to an OpenTelemetry Protocol (OTLP)  endpoint and appear there too. For agentic workloads, Omni reads the model calls, tool calls, and orchestration steps out of your traces, then presents them as agent traces  you can analyze and evaluate.
Key features for A2A observability:
  • OpenTelemetry -native: Omni ingests OTLP traces, metrics, and logs. Any OTel-instrumented agent participates with no proprietary SDK.
  • Framework support: Works with Strands Agents , LangGraph, LangChain, CrewAI, OpenAI Agents, and the Vercel AI SDK, in Python and Node.js.
  • GenAI-aware spans: The AWS Distro for OpenTelemetry  auto-instruments model and tool calls and records them with gen_ai.* semantic-convention attributes, including token usage per model call.
  • Cross-account by design: Omni does not centralize telemetry itself  — you centralize with CloudWatch centralization rules , and that centralized telemetry, including traces, then appears in your space like any other telemetry.
  • Context propagation: Because it is standard OTel, trace context flows across process, network, and account boundaries using W3C Trace Context , so hops in different accounts land in the same trace.

Understanding the A2A observability problem

Before the code, it helps to be precise about what "end to end" means when agents call agents across accounts.

The anatomy of a logical request

A single logical request is one unit of work from the user's point of view. Underneath, it fans out into a tree:
  • Orchestrator agent (Account A) receives the request and plans.
  • LLM calls — each Bedrock InvokeModel / Converse call the orchestrator makes.
  • Tool invocations — functions the agent runs to retrieve data or act.
  • Sub-agent calls — the orchestrator hands part of the work to another agent, which may live in Account B or C and repeats the whole pattern.
Every one of these is a span. The job of tracing is to make all of them share one trace ID  and nest under the right parent span ID , no matter which account or process they run in.

The three things that must be true

  1. Context must propagate. When the orchestrator calls a sub-agent, it must pass the W3C traceparent  header so the downstream agent continues the same trace instead of starting a new one.
  2. Traces must centralize. Spans emitted in three accounts are useless if you have to log into three accounts to see them. They need to reach one monitoring account.
  3. Cost must attribute. The token counts captured on spans tell you usage; your bill tells you dollars. Tying a logical request's usage to its share of the bill, across accounts, is a separate problem that tagging and CUR 2.0 solve.
The rest of this guide builds each piece.

Prerequisites

  • Python 3.10 or higher (examples use Strands Agents; Node.js is also supported)
  • Two or more AWS accounts: at least one source account where agents run, and one monitoring account where you view traces
  • Access to Amazon Bedrock  models in each source account (for example Claude Sonnet, Amazon Nova)
  • Transaction Search enabled  in each account and Region that receives traces (traces sent before it is enabled are not searchable)
  • The AWS managed policy AWSXrayWriteOnlyAccess attached to the role your compute uses (Lambda, EC2, ECS, or EKS), so the exporter can deliver spans
  • AWS CLI configured with credentials for each account
  • An AWS Organization  containing your source and destination accounts, and permission to create CloudWatch centralization rules  from the management account or a delegated administrator, plus permission to activate cost allocation tags  in AWS Billing

Part 1: Instrumenting an agent for OTLP traces

Start with a single agent. Strands Agents  has built-in OpenTelemetry tracing, so with the AWS Distro for OpenTelemetry  (ADOT) the model and tool calls are captured as spans automatically, no span code required.
Shell conventions used in this guide. The commands come in two flavors — pick the one for your machine. Windows examples use PowerShell: environment variables are set with $env:NAME = "value" and persist only for the current session. macOS / Linux examples use bash or zsh: environment variables use export NAME="value". Python, pip, and the AWS CLI commands themselves are identical on both; only the shell syntax for variables and line continuation differs (PowerShell uses a backtick ` to continue a line, bash uses a backslash \).
Install the distribution (identical on every platform):
1
pip install "aws-opentelemetry-distro>=0.20.0" strands-agents strands-agents-tools -q
The distribution registers the global tracer provider on startup and derives the OTLP trace-export settings. The key rule: do not create your own tracer provider or exporter, attach everything to the distribution's provider. You start the agent under the OpenTelemetry launcher so instrumentation initializes before the framework is imported. The opentelemetry-instrument command is the same on both platforms once it is on your PATH:
1
2
# Windows (PowerShell), macOS, or Linux
opentelemetry-instrument python orchestrator.py
If opentelemetry-instrument is "not found" after install: pip placed the console script in a per-user Scripts/bin directory that is not on your PATH. pip warns about this at install time (for example, The scripts opentelemetry-bootstrap.exe and opentelemetry-instrument.exe are installed in '...\Python313\Scripts' which is not on PATH). The script runs fine — the shell just can't find it by name yet. Note that it is a console script, not a runnable module, so python -m opentelemetry.instrumentation.auto_instrumentation ... does not work; use one of the following.
Fastest unblock — call the script by its full path (no PATH change, works in the current terminal). Use the directory from your own pip warning:
1
2
# Windows (PowerShell)
& "$env:APPDATA\Python\Python313\Scripts\opentelemetry-instrument.exe" python orchestrator.py
1
2
# macOS / Linux
"$(python -m site --user-base)/bin/opentelemetry-instrument" python orchestrator.py
Permanent fix — add that directory to PATH. On Windows this persists for your user; open a new terminal afterward, because an already-open shell keeps its old PATH:
1
2
3
4
5
6
# Windows (PowerShell) — use the exact path pip reported
$scripts = Join-Path $env:APPDATA 'Python\Python313\Scripts'
[Environment]::SetEnvironmentVariable(
'Path', "$([Environment]::GetEnvironmentVariable('Path','User'));$scripts", 'User')
# To use it in THIS session without reopening, also run:
$env:Path += ";$scripts"
1
2
3
# macOS / Linux (bash or zsh) — add to PATH now and persist in your profile
export PATH="$(python -m site --user-base)/bin:$PATH"
echo 'export PATH="$(python -m site --user-base)/bin:$PATH"' >> ~/.zshrc # or ~/.bashrc
Using a virtual environment places the script on PATH (inside an activated venv, pip installs it into the environment's Scripts/bin), which resolves the not-found case — but see the next note before relying on a venv on Windows, because a venv does not fix the spaces-in-path error below.
If you see C:\Program: can't open file ... on Windows: this is a different problem from the one above — the launcher is found, but it fails to re-launch Python. The opentelemetry-instrument wrapper re-executes your script using the full path to python.exe (sys.executable) without quoting it. When that path contains a space — which it does for the default install location C:\Program Files\Python313\python.exe — the space splits the command, so C:\Program is treated as the program and the rest is mangled into a bogus file argument. A venv does not help, because the venv's python.exe sits inside your project folder, and if that path contains spaces (for example My Documents\...) the same split happens. The reliable fix is to hand the launcher a space-free path to Python using its DOS 8.3 short name:
1
2
3
4
# Windows (PowerShell) — resolve the space-free 8.3 short path, then launch
$py = (New-Object -ComObject Scripting.FileSystemObject).GetFile("C:\Program Files\Python313\python.exe").ShortPath
# $py is now something like C:\PROGRA~1\PYTHON~1\python.exe
opentelemetry-instrument $py orchestrator.py
The surest long-term fix is to install Python (or just keep your project) under a path with no spaces, such as C:\Python313 or W:\projects\omni-a2a. Then plain opentelemetry-instrument python orchestrator.py works with no workaround. macOS and Linux do not hit this, because their default interpreter paths (/usr/bin/python3, a venv, pyenv) contain no spaces.
Now write the agent itself. Save the following as orchestrator.py — the same filename the launcher command above runs. The thing to notice is what is not here: there are no imports from the tracing SDK, no tracer provider, no exporter, and no span code. You write a plain Strands agent, and the opentelemetry-instrument launcher wraps it at startup, instrumenting every Bedrock model call and tool call for you. Adding your own tracer provider here would actually break it, because OpenTelemetry uses only the first provider registered and the launcher has already registered one.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
# orchestrator.py — a plain Strands agent. Do NOT add tracing code here;
# the opentelemetry-instrument launcher instruments the Bedrock model
# calls and tool calls automatically, recording each as a span with
# gen_ai.* attributes (including input/output token counts).
from strands import Agent
from strands_tools import http_request

agent = Agent(
tools=[http_request],
# Replace with a model you have enabled in this account/Region.
model="us.anthropic.claude-sonnet-4-5-20250929-v1:0",
system_prompt="You are an orchestrator. Plan the request and call tools or sub-agents as needed.",
)

response = agent("Summarize this quarter's pricing trends for the enterprise tier.")
print(response)
Run it with the launcher command from above (opentelemetry-instrument python orchestrator.py), not with a plain python orchestrator.py — the plain form skips instrumentation and produces no spans.
When this runs, each Bedrock call becomes a span carrying gen_ai.* attributes, and each tool call becomes a child span. On your own compute, set the OTLP endpoint and the destination log group through the standard OTel environment variables; the exporter then signs and delivers spans to X-Ray , where Omni reads them. Within about five minutes the invocation shows up under Agent traces  in your Omni space, with an agent span at the root and a child span for every model and tool call.
If this is the first agent in the account and you see nothing under Agent traces, confirm Transaction Search  is enabled — it routes spans into CloudWatch Logs, and traces sent before it is on are not searchable. This is a single-line AWS CLI command, identical on Windows, macOS, and Linux, and it takes about 10 minutes to take effect:
1
2
# Run once per account and Region (Windows, macOS, or Linux)
aws xray update-trace-segment-destination --destination CloudWatchLogs --region us-east-1

Part 2: Propagating W3C trace context across agents

A single agent is the easy case. The whole point of A2A is that the orchestrator calls another agent, often over HTTP and often in another account. Unless you pass the trace context, the downstream agent starts a brand-new trace and you lose the stitch.
OpenTelemetry solves this with context propagation  using the W3C Trace Context  standard: the traceparent header carries the trace ID, the parent span ID, and sampling flags. The calling agent injects it; the receiving agent extracts it.

Inject context on the outbound call

When the orchestrator calls a sub-agent's endpoint, inject the current context into the request headers:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
from opentelemetry import propagate
import requests

def call_sub_agent(endpoint: str, payload: dict) -> dict:
"""Call a sub-agent in another account, carrying the trace context."""
headers = {"Content-Type": "application/json"}

# Inject W3C traceparent (and tracestate) into the outbound headers.
# Downstream spans will nest under the current span, in the SAME trace.
propagate.inject(headers)

resp = requests.post(endpoint, json=payload, headers=headers, timeout=60)
resp.raise_for_status()
return resp.json()

Extract context on the inbound side

The sub-agent (running in Account B, with its own ADOT instrumentation) extracts the incoming context and attaches it so its own spans continue the trace:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
from opentelemetry import propagate, context, trace
from strands import Agent

tracer = trace.get_tracer(__name__)

def handle_request(event):
"""Entry point for the sub-agent. Continues the caller's trace."""
incoming_headers = event.get("headers", {})

# Extract the W3C traceparent the orchestrator injected.
ctx = propagate.extract(incoming_headers)
token = context.attach(ctx)
try:
with tracer.start_as_current_span("research-sub-agent"):
agent = Agent(
model="us.amazon.nova-pro-v1:0",
system_prompt="You are a research sub-agent. Answer the delegated task.",
)
return str(agent(event["task"]))
finally:
context.detach(token)

Let sampling follow the parent

Sampling decisions must be consistent across the whole trace, or you will get half a request. The recommendation is to leave OTEL_TRACES_SAMPLER unset  on downstream agents. By default the distribution traces every request that starts at your agent, and when a request arrives from another traced service, the agent follows that caller's decision. Tracing every request also keeps span-derived metrics such as token usage accurate.
If the root service needs to sample down to match traffic, set the sampler only there, using a parent-based ratio sampler so children inherit the decision. Set these in the same shell session (or process environment) that launches the agent:
1
2
3
# Windows (PowerShell)
$env:OTEL_TRACES_SAMPLER = "parentbased_traceidratio"
$env:OTEL_TRACES_SAMPLER_ARG = "0.9"
1
2
3
# macOS / Linux (bash or zsh)
export OTEL_TRACES_SAMPLER=parentbased_traceidratio
export OTEL_TRACES_SAMPLER_ARG=0.9
With injection on every outbound hop and extraction on every inbound hop, all the agents — across all the accounts — emit spans that share one trace ID. Omni can now draw the entire logical request as a single tree.

Part 3: Centralizing traces across accounts with CloudWatch centralization rules

Each source account currently delivers its spans to its own account. To see one unified trace, those spans have to reach the single account and Region that hosts your Omni space. It is tempting to reach for CloudWatch Observability Access Manager (OAM)  , the sinks-and-links feature behind CloudWatch cross-account observability , but that is not the mechanism Omni uses. Per the CloudWatch Omni documentation : "Centralization stays with CloudWatch. Omni does not aggregate telemetry across accounts and Regions on its own. Telemetry you centralize with CloudWatch centralization rules appears in your space like any other telemetry." OAM is a real, separate CloudWatch feature, but Omni's own cross-account story runs on CloudWatch centralization rules instead.
The setup has three parts:
  1. Host the Omni space in the account and Region you want to act as the destination, this is the account where you will view the unified trace.
  2. Enable Transaction Search  in every source account. This requirement from Part 1 carries forward unchanged, it is what gets spans into CloudWatch Logs in the first place, and it is also a hard prerequisite for centralizing traces, not just for searching them locally.
  3. Create a CloudWatch centralization rule that names the source accounts/Regions and the destination account/Region.

Centralization rules require AWS Organizations

Unlike OAM, which can link any two accounts directly, centralization rules only work inside an AWS Organization. Per the Omni organization setup guide : "Centralization rules require AWS Organizations and must be created from the management account or by a delegated administrator." There is no non-Organizations path, if your source and destination accounts are not members of the same AWS Organization, you cannot create a centralization rule between them, and this guide's three-account setup (A, B, C feeding one monitoring account) assumes they are.

Create the centralization rule

Centralization rules are configured through the CloudWatch console's Organization settings, not through a dedicated AWS CLI subcommand , there is no aws cloudwatch create-centralization-rule-style one-liner to show here, so this is a console walkthrough rather than a shell block. From the log centralization setup guide  and the metrics centralization setup guide :
  1. Sign in to the CloudWatch console in the AWS Organizations management account (or a delegated administrator account) — not in the destination account itself.
  2. Choose Settings, then the Organization tab, then Configure rule.
  3. Specify the source details: a rule name, the source accounts (a list of accounts, a list of OUs, or the entire organization), and the source Regions.
  4. Specify the destination details: the destination account (the one hosting your Omni space) and the destination Region. Centralization is cross-Region as well as cross-account , the destination Region does not have to match every source Region.
  5. Specify the telemetry to centralize, logs and/or metrics (both are selected by default; you can deselect one).
  6. Review and choose Create centralization policy.
You do not create a separate rule for traces. Per the same guide: "Trace centralization requires Transaction Search enablement for each source account. To centralize traces, enable Transaction Search in all your source accounts. Trace centralization then happens automatically when logs centralization is enabled." So once Transaction Search is on in Account A, B, and C, and the centralization rule covers logs, the agent traces ride along automatically, there is no AWS::XRay::Trace-equivalent resource type to request separately, because the rule operates on log groups (and metrics), not on X-Ray directly.
A few operational notes:
  • Cross-Region, not same-Region. This is the opposite of OAM's constraint. OAM links are scoped to a single Region, but a centralization rule explicitly has independent "Source Regions" and "Destination Region" fields, so you can centralize telemetry from several Regions into one destination Region.
  • AgentCore's cross-account story is a different, OAM-based mechanism — don't conflate the two. Monitoring Amazon Bedrock AgentCore resources across accounts  still uses CloudWatch Observability Access Manager sinks and links (AWS::Oam::Sink / AWS::Oam::Link), the same way classic CloudWatch cross-account observability does, and that AgentCore cross-account view is explicitly single-Region: "Cross-account observability works within a single AWS Region. The monitoring account and source accounts must be in the same Region." If you rely on the AgentCore Observability console's own cross-account view of agent metrics, traces, sessions, and resource data, you are using OAM and are bound by its same-Region rule — separately from whatever Omni centralization rule you've set up for the unified trace view this guide builds.
  • This is a requirement, not a scaling option. Because centralization rules only exist inside an AWS Organization and must be created by the management account or a delegated administrator, there is no smaller-scale alternative to reach for with just two or three accounts — the Organization prerequisite applies from the first source account you add, not just when you scale to many.
Once the centralization rule is active and Transaction Search is enabled in every source account, open the Omni space in the destination account. A trace that began in the orchestrator (Account A) and continued through sub-agents (Accounts B and C) now renders as one tree, with every model and tool span in place, regardless of which account produced it. Keep in mind centralization does not backfi, only telemetry generated after the rule becomes active is replicated, and cross-space queries are not supported , so all the accounts you want to see together must centralize into the same destination space rather than each getting their own.

Part 4: FinOps — attributing cost to a logical request across accounts

Here is the part most observability guides skip. You can now see the whole request. Can you say what it cost, and bill it to the right team?
This requires combining three data sources, because no single one answers the question alone.

What each source gives you

SourceGrainWhat it tells you
Omni trace spansPer model callToken usage (gen_ai.*) per hop, tied to the trace ID
Model invocation logsPer requestFull per-request detail, including token counts, by invocation
CUR 2.0Per usage type, per hour/dayActual dollars, by model, token type, tier, and tags
The critical fact: CUR does not contain per-request line items.  Both classic CUR and CUR 2.0 aggregate Bedrock cost by usage type, operation, and resource over an hour or a day, neither carries a per-requestId identifier. So you cannot read a dollar figure for one trace directly out of CUR. Instead you attribute cost by joining the per-request token detail (from spans and model invocation logs ) to CUR dollars at the model and usage-type grain.

Account for all four token types

CUR 2.0  generates separate line items per token type , each with its own unit price:
  • Input tokens — *-input-tokens
  • Output tokens — *-output-tokens
  • Cache read tokens — *-cache-read-input-token-count (cheaper than input)
  • Cache write tokens — *-cache-write-input-token-count (more expensive than input)
If you only sum input and output tokens, your totals will not match the bill — this is the most common reconciliation gap, especially with heavy prompt caching. The line_item_usage_type field also encodes the service tier (standard, priority, flex) and whether the request used cross-region inference, and those carry different unit prices. Apply the matching rate for each line item.

A cost allocation tagging strategy for A2A

To attribute dollars to a logical request type, a team, or a project across accounts, you need a tag dimension that survives into CUR. Bedrock surfaces several, and they appear in CUR 2.0 as columns:
A practical strategy for A2A:
  1. Pick one shared dimension that identifies the logical workload — for example resourceTags/Workload=enterprise-pricing. Apply it consistently in every account by having each agent invoke Bedrock through an application inference profile  tagged with that value. The tag then rides into CUR in every account the workflow touches.
  2. Add a per-account or per-team tag (for example Team) so you can also slice cost by the account that did the work.
  3. Activate the tags as cost allocation tags  in the AWS Billing console. They do not appear in CUR or Cost Explorer until activated, and after activation allow up to 24 hours for them to populate.
  4. Use CUR 2.0 (AWS Data Exports)  — IAM principal identity and tag data require CUR 2.0; the legacy format omits the IAM principal fields.
With the same Workload tag applied through inference profiles in every account, you can aggregate the full cross-account cost of a logical workload in Cost Explorer or query it in Athena — even though the work ran in three accounts.

Querying it: usage from spans, dollars from CUR

The reconciliation pattern is a join at the model + usage-type grain. Conceptually:
1
2
3
4
5
6
7
8
9
10
11
-- Dollars per model + usage type + workload tag, from CUR 2.0 (Athena)
SELECT
line_item_usage_account_id AS account_id,
resource_tags['user:workload'] AS workload,
line_item_usage_type AS usage_type, -- encodes model, token type, tier, routing
SUM(line_item_unblended_cost) AS cost_usd,
SUM(line_item_usage_amount) AS tokens
FROM cur2_bedrock
WHERE line_item_product_code = 'AmazonBedrock'
AND line_item_usage_start_date >= DATE '2026-10-01'
GROUP BY 1, 2, 3;
Then attribute to a single logical request by taking that request's token counts per model and usage type (from its trace spans / model invocation logs) and applying the effective per-token rate the query above yields for the same model and usage type. The span tells you how many tokens this request used; CUR tells you what a token of that type costs; the shared tag tells you which workload bucket it belongs to across accounts. Multiply and you have a defensible per-request cost, rolled up across every account the request touched.

Putting it all together

An end-to-end A2A observability setup comes down to five moving parts:
  1. Instrument every agent  with the AWS Distro for OpenTelemetry so model and tool calls become gen_ai.* spans, with Transaction Search enabled in each account.
  2. Propagate W3C trace context  on every outbound sub-agent call (inject) and inbound handler (extract), and let sampling follow the parent so no hop is lost.
  3. Centralize with a CloudWatch centralization rule  created from the AWS Organizations management account or a delegated administrator — naming the source accounts/Regions and the destination account/Region — so trace centralization rides automatically on logs centralization once Transaction Search is enabled everywhere, and the whole tree is visible in one place.
  4. Tag consistently  with a shared workload dimension via application inference profiles, and activate those as cost allocation tags.
  5. Reconcile cost  by joining per-request token usage from spans and model invocation logs to CUR 2.0 dollars at the model and usage-type grain , remembering all four token types, service tiers, and cross-region rates, and that CUR has no per-request row.

Conclusion

Agent-to-agent architectures give you flexibility and blast-radius isolation by spreading work across independent agents and accounts. The cost is observability: a single request becomes a scattered set of spans and a scattered set of charges. CloudWatch Omni, built on OpenTelemetry, lets you reassemble the request into one trace; a CloudWatch centralization rule lets you see that trace from one destination account no matter where the hops ran; and a disciplined tagging strategy over CUR 2.0 lets you turn the token counts on those spans into dollars attributed back to the team or workload that incurred them. Instrument once, propagate everywhere, centralize to one account, and tag for cost, and the question "what did this request do, and what did it cost?" finally has a direct answer.
# agentic-ai# agent-to-agent# amazon-bedrock-agentcore# amazon-cloudwatch# observability
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article