Seeing What Your AI Agent Did: Logging and Tracing with CloudWatch and X-Ray
Learn how to see what your AI agent did. Enable CloudWatch Transaction Search for Amazon Bedrock AgentCore, read traces in the GenAI Observability dashboard, add structured logs, protect sensitive data and set up alarms.
Series: Amazon Bedrock (13 articles)
- …
- 12Seeing What Your AI Agent Did: Logging and Tracing with CloudWatch and X-Ray This article
Agents choose their own path, so you need a record of the path they took. Here's how to turn on tracing for an AgentCore agent, read a trace, add your own logs and set up alerts.
With an ordinary program, the same input gives the same steps every time. With an agent, it doesn't. The model decides which tools to call, in what order, and how many times. When a customer says "it gave me a strange answer," the only way to find out why is to look at what actually happened.
That's what observability is for. In this article we'll switch it on for an AgentCore agent, read a trace, add structured logs of our own and put alarms on the things that matter.
What you need
- A deployed AgentCore agent, such as the one from the earlier articles
- Access to the Amazon CloudWatch console in the same Region
- Permission to change CloudWatch settings, at least once, to enable Transaction Search
- About thirty minutes
Tracing and logging add charges for ingested data, trace indexing and log storage. Check the current CloudWatch pricing, and read the cost notes near the end before you leave everything on.
A few terms
The vocabulary is short:
Term | Plain meaning |
|---|---|
Log | A line of text your agent or its platform wrote, such as an error or a note |
Metric | A number tracked over time, such as how many requests failed |
Span | One step in a request, such as a model call or a tool call, with its start time and duration |
Trace | All the spans of one request, strung together in order |
Session | A series of related requests, such as one user's conversation |
A trace is the one to love when you're debugging. It's the story of one request from start to finish.
Step 1: Turn on Transaction Search (once per account)
This is the step people miss. According to the AgentCore documentation, first-time users need to enable CloudWatch Transaction Search once before AgentCore spans and traces become visible. Without it, you'll deploy and invoke happily and see nothing.
In the CloudWatch console:
- Open Settings, choose Account, and select the X-Ray traces tab.
- In the Transaction Search section, choose View settings.
- Choose Enable Transaction Search.
- Choose the option for X-Ray users, and set the percentage of traces to index.
The indexing percentage is a cost control. A lower percentage means fewer traces are stored for search, which is cheaper but means you'll see only a sample. For a first test, a modest figure is fine. For debugging a specific problem, you can raise it temporarily.
After you enable it, the documentation says it can take about ten minutes for spans to become available. If you're doing this for the first time, enable it, make a cup of tea, then continue.
The documentation also describes enabling it through an API, which is handy if you want to repeat the setup across several accounts.
Step 2: Deploy and invoke your agent
For an agent hosted on AgentCore Runtime, the instrumentation is built in. You don't need to add tracing code. Deploy it, then make some requests:
Bash
agentcore deployagentcore invoke --runtime agent --payload '{"prompt":"Where is order 1001?"}'
Send a few different prompts, including one that should use a tool and one that shouldn't, so you have something interesting to look at.
If your agent runs somewhere other than AgentCore Runtime, for example on your own servers or another cloud, you can still send its telemetry to the same dashboard using the AWS Distro for OpenTelemetry. The AgentCore documentation describes the settings for this. That's a bigger topic than we need here.
Step 3: Explore the GenAI Observability dashboard
In the CloudWatch console, open GenAI Observability and choose the Bedrock AgentCore section. The documentation says the page has three views:
- Agents view: an overview of each agent, with its health and activity.
- Sessions view: conversations grouped by session, so you can follow one user's journey.
- Traces view: individual requests, with the spans inside each one.
Start in the Traces view and open one of your requests.
Step 4: Read a trace
A good trace shows you, in order, what the agent did and how long each step took. When you open one, ask these questions:
- What was the sequence? Which model calls and tool calls happened, and in what order?
- Did it pick the right tool? If it used the wrong one, the tool's description probably needs work.
- What went into each tool? Check the arguments the model chose. Wrong input here explains many wrong answers.
- Where did the time go? A slow request often comes down to one slow step, or to too many steps.
- Did anything fail or retry? Repeated calls to the same tool are a red flag for a loop.
- Is anything unexpected? Steps you didn't plan for are worth understanding.
A habit that pays off: after every change to your agent's prompt or tools, send a handful of test requests and read at least one trace before you call it done.
Step 5: Read the logs from the command line
The AgentCore CLI gives you quick access without leaving the terminal:
Bash
agentcore logs runtimeagentcore logs runtime --since 30m --level erroragentcore traces runtime list --since 30m
The runtime's standard output and standard error go to CloudWatch Logs, in a log group whose name starts with
/aws/bedrock-agentcore/runtimes/ followed by your agent's ID and endpoint name. You can also tail them with the AWS CLI:Plain text
aws logs tail /aws/bedrock-agentcore/runtimes/YOUR_AGENT_ID-DEFAULT --follow
Check the log group name in your own account, since it includes your agent's ID.
Step 6: Add your own structured logs
Built-in traces tell you what the platform saw. They don't know your business. Add a few well-chosen log lines of your own, written as JSON so CloudWatch can search and summarise them.
Plain text
import jsonimport loggingimport timelogging.basicConfig(level=logging.INFO, format="%(message)s")logger = logging.getLogger("agent")def log_event(event, **fields):logger.info(json.dumps({"event": event, **fields}))# Inside a tool function:start = time.time()result = look_up_order(order_id) # your own lookuplog_event("tool_call",tool="get_order_status",found=result.get("found"),duration_ms=int((time.time() - start) * 1000),)
Then you can ask CloudWatch Logs Insights questions such as which tools are slowest:
Plain text
fields @timestamp, tool, duration_ms| filter event = "tool_call"| stats avg(duration_ms), max(duration_ms), count(*) by tool
What's worth logging: which tool ran, how long it took, whether it succeeded, which version of the prompt was in use, and a request or session ID so you can match a log line to a trace.
What to leave out: passwords, keys, tokens and anything personal you don't need. Which brings us to the next point.
Step 7: Protect what you log
Logs and traces can contain the very things users type and the model says, and those can include personal or confidential information. Treat them with care.
- Don't log secrets or sensitive personal data. Mask or leave out fields you don't need.
- Set a retention period on your log groups, so data isn't kept forever by default.
- Limit who can read them. Use IAM to restrict access to the logs and the dashboard, in line with the least-privilege approach from the previous article.
- Think about your obligations. If you handle personal data, check what your privacy rules say about storing prompts and responses.
Step 8: Set up alarms
Looking at dashboards only helps when you remember to look. Alarms tell you when to.
In CloudWatch, create alarms for the few things that matter most to your agent, for example:
- Error rate rising above a level you're comfortable with
- Latency exceeding what users will tolerate
- Unusual volume, a sudden spike that could mean a loop or abuse
- A key log event, using a metric filter on your own JSON logs, such as repeated tool failures
AgentCore publishes its metrics to CloudWatch, so check the metric names and namespace available in your account and build alarms from those. Send the alarm to an email address or a notification topic someone actually reads. And connect this to your budget alerts from the cost article, so you're told about both quality problems and money problems.
When no traces appear
Transaction Search isn't enabled. This is the most common cause. Enable it, then wait about ten minutes.
Wrong Region. The dashboard is regional. Check you're looking in the Region where the agent runs.
You haven't invoked the agent recently. Send a new request and refresh.
The agent runs outside AgentCore Runtime. You need to set up the OpenTelemetry exporter yourself, as described in the documentation.
A permissions problem. Check that the agent's role is allowed to write logs and traces, and that your own identity can read them.
Sampling. If you set a low indexing percentage, some requests won't be searchable. Raise it temporarily when debugging.
Cost notes
- Traces, logs and log storage are billed, so volume matters. A busy agent can generate a lot.
- Lower the trace indexing percentage once you've finished debugging.
- Set log retention to what you really need.
- Be careful with very detailed logging in production.
A checklist
- Is Transaction Search enabled in the right account and Region?
- Have you read at least one real trace end to end?
- Do your logs include a session or request ID?
- Are secrets and personal data kept out of the logs?
- Is log retention set?
- Do you have alarms for errors, latency and unusual volume?
Where we go next
We now have an agent that's secure and visible. In the next article we'll put it to work on a real business task: a project write-up with the design, the results and the honest lessons.
AgentCore Observability features, CloudWatch settings, console layouts and pricing change often. Check the current AWS documentation before you follow any step here.
Series: Amazon Bedrock (13 articles)
- …
- 12Seeing What Your AI Agent Did: Logging and Tracing with CloudWatch and X-Ray This article
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article