Build and Observe Production AI Agents with Strands Agents, Amazon Bedrock AgentCore, and Datadog Agent Observability
Learn how to instrument Strands Agents on Amazon Bedrock AgentCore with Datadog Agent Observability using OpenTelemetry, enabling end-to-end tracing, cost monitoring, and structured experimentation for production AI agents.
This post is co-written by
Shashiraj Jeripotula and Kola Akinnibi, AWS
George Morgan, Rashel Hoover, and Will Potts, Datadog
The rise of artificial intelligence (AI) agents marks a fundamental shift in how software applications make decisions and interact with users. Unlike traditional systems that follow predetermined execution paths, AI agents engage in complex, multi-step reasoning — selecting tools, forming plans, and adapting their behavior based on intermediate results. This dynamic reasoning often operates as a "black box," creating a critical challenge for organizations: how can you trust, debug, and optimize what you can't see?
This is where agent observability becomes essential. Observability provides deep insights into how agentic applications perform, reason, and execute tasks — enabling teams to identify performance bottlenecks, debug unexpected behaviors, control costs, and maintain the reliability standards that production workloads demand.
In this post, we explain how to integrate Datadog Agent Observability with Amazon Bedrock AgentCore to gain comprehensive visibility into your AI agent's performance, debug issues faster, optimize inference costs, and run systematic experiments to improve agent quality. We walk through a complete implementation using Strands Agents deployed on AgentCore Runtime, including step-by-step code examples for both the AWS and Datadog configuration.
Why AI observability matters
Traditional application monitoring tracks request-response cycles with predictable inputs and outputs. AI agents introduce fundamentally different challenges:
- Non-deterministic execution paths — The same input can produce different reasoning chains and tool invocations depending on model state, context, and intermediate results
- Multi-step reasoning opacity — Agents may invoke multiple tools, make sequential decisions, and backtrack — all invisible without proper instrumentation
- Cost unpredictability — Token consumption varies dramatically between simple queries and complex multi-turn interactions, making cost forecasting difficult
- Latency variance — Response times depend on the number of reasoning steps, tool calls, and model inference cycles, creating variable user experiences
- Failure modes unique to AI — Rate limiting, token limit exceeded, hallucination-triggered tool errors, and context window overflows require AI-specific monitoring
Organizations operating agents without observability face longer mean time to resolution (MTTR), delayed development cycles, and increased security and cost risks. Datadog Agent Observability addresses these challenges as a complete AI engineering platform supporting teams with pre-production experimentation and production monitoring. It helps teams evaluate, improve, and secure generative AI workloads from individual model inferences to complex multi-agent orchestrations.
How Datadog Agent Observability works
Datadog Agent Observability uses OpenTelemetry (OTEL) to trace and monitor agents deployed on Amazon Bedrock AgentCore. OpenTelemetry is a Cloud Native Computing Foundation (CNCF) project that provides standardized APIs, SDKs, and conventions for collecting distributed traces and metrics from applications.
Agents built with the Strands Agents SDK are instrumented with OpenTelemetry out of the box, emitting agent, LLM (Large Language Models), and tool spans that follow the GenAI semantic conventions . In this solution, the agent code configures a custom OpenTelemetry TracerProvider that exports these spans over OTLP HTTP directly to Datadog's Agent Observability intake. Because AgentCore Runtime ships a default AWS Distribution for OpenTelemetry (ADOT) pipeline that sends traces to Amazon CloudWatch, the sample sets
DISABLE_ADOT_OBSERVABILITY=true so the custom Datadog exporter becomes the sole trace provider and Datadog receives the spans exclusively. The telemetry captures:- Agent-level spans — Complete end-to-end request lifecycle including total latency and final output
- LLM inference spans — Individual model calls with input/output tokens, model ID, temperature, and response latency
- Tool execution spans — Each tool invocation with parameters, execution duration, and return values
- Reasoning step spans — The agent's chain-of-thought decisions between tool calls
This hierarchical trace structure flows from the Strands agent through the OTLP exporter directly to Datadog's
https://trace.agent.{DD_SITE}/v1/traces endpoint. Datadog's backend automatically parses GenAI semantic conventions, extracting metrics like token usage, model latency, and inference costs without requiring custom instrumentation.The following diagram illustrates the data flow:
Figure 1: Architecture overview — Telemetry flows from AgentCore Runtime to Datadog via OTLP
Experimenting and iterating in development
Key Datadog Agent Observability capabilities
Beyond basic tracing, Datadog Agent Observability provides a suite of capabilities specifically designed for AI workloads across the development flywheel, from offline testing and experimentation through production monitoring.
| # | Capability | Description |
|---|---|---|
| 1 | Distributed Tracing | End-to-end visibility across agent reasoning, tool calls, and LLM inferences, visualized as a complete call graph from prompt to response. |
| 2 | Application Stack Correlation | Correlate agent traces with APM services, infrastructure, and RUM user sessions in one connected trace for full-stack root cause analysis. |
| 3 | Performance & Error Monitoring | Track latency percentiles, token throughput, time to first token, requests per second, retries, and errors at every step. |
| 4 | Token & Cost Monitoring | Real-time cost and token tracking by model, per-conversation cost estimates, usage trends, and alerting on spending thresholds. |
| 5 | Security & Safety Monitoring | Defense-in-depth: AI Guard blocks unsafe prompts and tool calls in the execution path, paired with Sensitive Data Scanner redaction. |
| 6 | Pattern & Behavioral Analysis | Automatically clusters production interactions into hierarchical topics, surfacing unexpected user workflows, agent behaviors, and quality drift before they become incidents. |
| 7 | Playground Testing | Test different prompt, model, and tool call changes in an isolated environment to develop hypotheses. |
| 8 | Structured Experimentation | Compare prompts, models, and agent configurations against versioned production datasets to see which version performs best before release. |
| 9 | Evaluation & Annotation | Out-of-the-box and custom evaluators, plus human review and annotation, score quality offline and online and catch drift across releases. |
| 10 | Embedded AI Agent | An agentic feature that runs root cause analysis on failing traces and generates evaluator and dataset candidates from production behavior. |
| 11 | Auto-Instrumentation & OTEL-Native Onboarding | Auto-instrument popular frameworks like OpenAI, LangChain, and Bedrock, or onboard any stack through OpenTelemetry and the HTTP API. |
| 12 | Role-Based Access Control | Role-based access control and enterprise-grade governance, including HIPAA compliance, let teams run AI in production safely. |
Solution overview
This post demonstrates how to deploy a Strands agent on Amazon Bedrock AgentCore Runtime with Datadog Agent Observability. The implementation uses Anthropic Claude models through Amazon Bedrock. Telemetry data flows from the Strands agent through OTEL exporters to Datadog for monitoring, debugging, and optimization.
Key components used in the solution:
- Strands Agents — Python framework for building LLM-powered agents with built-in telemetry support
- Amazon Bedrock AgentCore Runtime — Managed runtime service for hosting and scaling agents on AWS
- Datadog Agent Observability — Enterprise observability service with specialized LLM monitoring, tracing, and experimentation
- OpenTelemetry — Industry-standard protocol for collecting and exporting telemetry data
- Note: Datadog is an AWS Specialization Partner and an AWS Marketplace Seller . To learn more about Datadog for AWS, visit datadoghq.com/solutions/aws/ .
Prerequisites
Before starting this walkthrough, ensure you have the following:
- An AWS account with credentials configured (via AWS CLI or environment variables)
- Amazon Bedrock Model Access for Anthropic Claude Haiku 4.5 in the us-east-1 region
- Amazon Bedrock AgentCore permissions (for Runtime deployment)
- Python 3.10+ installed locally
- uv package manager for Python environments
- A Datadog account with:
- A Datadog API key (available under Organization Settings > API Keys)
- Agent Observability enabled (navigate to Agent Observability in the Datadog console)
Technical implementation guide
The complete working implementation — including the agent code, requirements, and deployment notebook — is available on GitHub. Start by cloning the repository and running the notebook end-to-end, then return here for detailed guidance on the Datadog configuration, monitoring features, and optimization techniques.
Step 1: Clone the repository and run the notebook
1
2
git clone https://github.com/awslabs/amazon-bedrock-agentcore-samples.git
cd amazon-bedrock-agentcore-samples/03-integrations/observability/datadog
Follow the instructions on README.md to install dependencies, build and deploy the agent. Once you've successfully deployed and invoked the agent, continue below for the Datadog-specific configuration details, monitoring walkthrough, and optimization guidance.
Step 2: Understanding the Datadog configuration
The repository's
deploy.py passes the required Datadog environment variables into the AgentCore Runtime container, and utils/travel_agent.py uses them to configure a custom OTel TracerProvider that exports traces directly to Datadog. Here's how it works:DISABLE_ADOT_OBSERVABILITY=true— Bypasses AgentCore's default CloudWatch ADOT pipeline so Datadog receives the traces exclusivelyOTEL_SEMCONV_STABILITY_OPT_IN=gen_ai_latest_experimental— Enables OTel v1.37+ GenAI semantic conventions required by Strands Agents for Datadog Agent Observability views- The agent creates an
OTLPSpanExporterpointing tohttps://trace.agent.{DD_SITE}/v1/traceswith thedd-api-keyanddd-otlp-source=llmobsheaders
The following table lists the required environment variables. Set them in the
.env file (copy from .env.example in the repo):| # | Parameter | Description | Example Value |
|---|---|---|---|
| 1 | DD_API_KEY | Your Datadog API key (from Organization Settings > API Keys) | abc123def456... |
| 2 | DD_SITE | Datadog site for your organization | datadoghq.com (US1), us3.datadoghq.com (US3), us5.datadoghq.com (US5), datadoghq.eu (EU1), ap1.datadoghq.com (AP1) |
| 3 | OTEL_SERVICE_NAME | Service name visible in Datadog Agent Observability | agentcore-travel-agent |
Important: Keep your
DD_API_KEY secure. In production, store it in AWS Secrets Manager or pass it as an encrypted environment variable through AgentCore Runtime configuration.Step 3: View traces in Datadog
After running
python invoke.py from the repo, navigate to Datadog > Agent Observability in your browser. After the agent runs, the trace hierarchy for its executions appears in Agent Observability.
Figure 2: Datadog Agent Observability — Trace waterfall showing the agent's reasoning steps, tool calls, and LLM inferences
The trace view displays:
- The root span representing the full agent invocation (total latency, final output)
- Child spans for each LLM inference call (model ID, input/output tokens, latency)
- Tool spans for each web_search invocation (query parameters, results, execution time)
- Metadata including session ID, model parameters, and token counts
Monitoring and iterating in Datadog
Once traces are flowing, Datadog Agent Observability provides multiple specialized views for understanding and optimizing your agent's behavior.
Trace exploration and debugging
The Traces view provides a searchable, filterable list of all agent invocations. Each trace expands into an execution graph showing the complete execution hierarchy:
Key debugging capabilities:
- Input/Output inspection — View the exact prompts sent to the model and the responses received at each step
- Error highlighting — Failed spans are visually marked in red, with error messages and stack traces
- Span-level attributes — Token counts, latency, model parameters, and custom metadata for each operation
- Cross-request correlation — Link related traces across sessions using session IDs and user identifiers
Annotating traces and adding to datasets
As traces flow in, you can route them into annotation queues to bring human-in-the-loop (HITL) review into the workflow. Domain experts review each trace with full context, prompts, completions, spans, tool calls, and evaluation results while applying structured labels and feedback. Datadog attaches this human judgment to the trace as metadata, so the agent's behavior and the expert assessment travel together. Automations route the traces that meet conditions you define, such as low evaluation scores or specific error types, to the right queue at a sampling rate you control. Reviewed traces then flow into datasets that become the foundation for testing and experimentation.
Experimenting against production datasets
Production signals feed directly into golden datasets. A dataset can hold hundreds of traces that represent the scenarios your agent meets in production, including edge cases, which lets you test against real behavior and avoid overfitting. These datasets are the foundation for structured experiments. You test different configurations side by side such as model selection, prompts, tool definitions, and parameters — all against the same collection of Strands agent traces, then compare the results. Teams gain confidence before deployment that a new variation of the agent configuration improves quality, cost-efficiency, or security, with a clear view of any trade-offs.
Figure 3: Datadog Experiments within Agent Observability provides a side-by-side comparison of experiment runs to show performance over time and tradeoffs across latency, quality, and costs with each potential agent configuration.
Figure 3: Datadog Experiments within Agent Observability provides a side-by-side comparison of experiment runs to show performance over time and tradeoffs across latency, quality, and costs with each potential agent configuration.
Clean up
To avoid incurring ongoing charges, delete the AWS resources created during this walkthrough. Navigate to the sample directory and run the cleanup script:
1
2
cd amazon-bedrock-agentcore-samples/03-integrations/observability/datadog
python cleanup.py
This removes the AgentCore Runtime deployment and associated AWS resources.
Conclusion
The integration of Amazon Bedrock AgentCore with Datadog Agent Observability provides an integrated approach for building, deploying, and operating production-ready AI agents with enterprise-grade observability. This combination provides:
- Managed deployment — Simplify deployment with AgentCore's managed infrastructure
- Comprehensive visibility — Full trace hierarchy from agent reasoning through tool execution and LLM inference
- Cost control — Real-time spend monitoring with per-model and per-conversation cost attribution
- Proactive reliability — AI-specific error detection and alerting before issues impact users
- Continuous optimization — Experimentation frameworks for systematic prompt and model improvement
By following the implementation in this post, you can deploy your first observable agent. As you scale from proof of concept to production, Datadog's Agent Observability grows with you — providing the insights needed to maintain performance, control costs, and continuously improve agent quality.
Get started
- AgentCore samples: github.com/awslabs/amazon-bedrock-agentcore-samples
- Strands Agents: strandsagents.com
- Datadog Agent Observability: datadoghq.com/product/llm-observability
- Datadog on AWS Marketplace: AWS Marketplace Seller Profile
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article