Autonomous Incident Resolution with AWS DevOps Agent and Datadog

When alerts fire at 3 AM, humans spend 40–70 minutes correlating signals to find root cause. AWS DevOps Agent changes this. This guide connects Datadog to DevOps Agent via webhook, creating a closed loop: Datadog detects an anomaly, fires a webhook, and DevOps Agent autonomously investigates — mapping blast radius, tracing dependencies, and identifying root cause before anyone's pager goes off. Covers full setup from registration to validation.

Shashi Shekhar

TechnicalAccountManager@AWS

Published Jun 12, 2026

Last modified Jun 22, 2026

Co-Author: Archana Srinivasan

As builders, we know our observability stacks are as varied as our architectures. Some of us run Datadog. Others rely on New Relic, Splunk, Dynatrace, or a combination of tools stitched together over years of organic growth. The common thread isn't which monitoring tool we use — it's the same broken incident response pattern underneath all of them:

When a cascading failure hits at 3 AM where the CPU usage has spiked past 90% on a series of production EC2 instances and the monitoring tool does its job. It fires the alerts. But then a human has to wake up, open six tabs, and manually correlate signals across services and tools to figure out which alert is the cause, and which are symptoms. That manual correlation is where 40–70 minutes disappear, regardless of whether the alerts came from Datadog, New Relic, or CloudWatch.

AWS DevOps Agent changes this equation by autonomously investigating incidents the moment they're detected — mapping the blast radius, tracing the dependency chain, and identifying root cause without waiting for a human to start the process.

Integration Overview

For customers currently using Datadog, the integration path is straightforward. The following walkthrough provides step-by-step instructions to connect Datadog's monitoring alerts directly to DevOps Agent via webhook, creating a closed loop where:

Datadog detects anomaly → Fires webhook → DevOps Agent investigates autonomously

This guide covers the full setup, from registering Datadog as a capability provider through validating the end-to-end flow. Once configured, any Datadog monitor alert can automatically trigger an autonomous DevOps Agent investigation.

The setup connects three things:

Datadog — your monitoring layer that detects anomalies
A webhook — the bridge that carries alert context (hostname, tags, alert status, event details) to DevOps Agent
AWS DevOps Agent — the autonomous investigator that receives the alert and goes to work

Step 1: Register Datadog as a Capability Provider

In the DevOps Agent console, navigate to Capability Providers, find Datadog under Telemetry, and click Register. Complete the OAuth flow by clicking Allow in Datadog — this establishes the trust relationship.

Make sure you sign up for the us1-east region as it simplifies the next steps.

After sign-up, skip the post sign-up steps and directly access the dashboard: https://app.datadoghq.com/dashboard/lists?p=1

Navigate to AWS DevOps Agent in the AWS console. We will enable DevOps Agent's integration with Datadog first.
a. Access Capability Providers menu.
b. Find Datadog under Telemetry. Click Register.

c. On the next page, supply a value for the server name, leave other defaults (assuming you signed up in the us1-east region) and click Next. If not, you have to choose the endpoint URL to match the signed-up region.

d. Click Add on the next page.

You will be asked to login to your Datadog account if you are not logged in already.

e. On the next step, click the Allow option.

f. You will be navigated back to the AWS console with the integration established.

In the next section, we integrate specific DevOps Agent space(s) to connect to Datadog.

Step 2: Associate Datadog with Your Agent Space

Add the Datadog integration to your Agent Space's Telemetry section. Upon saving, you receive a Webhook URL and Auth Token — save these immediately, they're shown only once.

a. Navigate to your agent space and scroll to the Telemetry section.

b. Click the Add link next to Datadog integration.

c. Click Save at the next step.

You will see an "Association created successfully" message. Note the webhook URL and auth token values. We will need this information later when sending webhooks from Datadog to AWS DevOps Agent to automatically trigger investigations.

⚠️ Important: The webhook URL and auth token are shown only once. Save them immediately.

Step 3: Deploy Datadog Monitoring

Create a Datadog API key, spin up a t3.micro EC2 instance (Amazon Linux 2023), and install the Datadog agent using the provided installation script. Verify the instance appears in your Datadog Fleet dashboard. We will set up an EC2 instance and stress its CPU to trigger a DevOps investigation.

Create a Datadog API Key

We need a Datadog API Key on the AWS resources to be able to enable monitoring from Datadog. Login to your Datadog account and click on your user icon and click on the API keys menu. Alternately, you can access the same menu at:

Datadog API Keys page

Create a new API key with a name of your choice.

On the next page, you will see the API key value. Copy this value as it will be needed later when sending data from Datadog agents running in AWS to be able to send monitoring data to Datadog. Click Finish.

Set Up AWS Infrastructure

Spin up a new t3.micro AL2023 EC2 instance. In the instance profile, attach the following policies to the instance role:

AmazonSSMManagedInstanceCore
CloudWatchAgentServerPolicy

Login to the EC2 instance via SSM Session Manager. Run the following commands to install the Datadog agent (replace DD_API_KEY with the API key value created earlier):

Set the DataDog API Key:

1
export DD_API_KEY="YOUR_API_KEY"

Set the Datadog site:

1
export DD_SITE="datadoghq.com"

And finally, download and install DataDog Agent:

1
bash -c "$(curl -L https://install.datadoghq.com/scripts/install_script_agent7.sh)"

Ignore any warning messages. When the script finishes, you should see a message confirming:

"Your Datadog Agent is running and functioning properly. It will continue to run in the background and submit metrics to Datadog."

Run this command to make sure datadog-agent is running:

1
sudo systemctl status datadog-agent

You should see the datadog agent with status as "running" in the output.

Region Configuration (if not US1)

Note: If you are using a different region other than US1, update the below config for the agent to communicate to Datadog without any errors.

Update the agent config:

1
sudo vi /etc/datadog-agent/datadog.yaml

Update these lines:

1
2
api_key: PASTE_NEW_KEY_HERE
site: datadoghq.com

Site values by region:

Region	Site Value
US1	`datadoghq.com`
US3	`us3.datadoghq.com`
US5	`us5.datadoghq.com`
EU	`datadoghq.eu`
AP1	`ap1.datadoghq.com`
US Gov	`ddog-gov.com`

Step 4: Configure the Webhook in Datadog

In Datadog Integrations, create a webhook named AWSDevOpsAgent using the Webhook URL from Step 2, a structured JSON payload with incident details, and a custom Authorization header with the Bearer token.

a. Navigate to this link to view Datadog agents recognized by Datadog: https://app.datadoghq.com/fleet

You will see that Datadog recognizes an agent running on your EC2 instance.

b. We will now create a webhook that will be used by the monitors to trigger DevOps Agent investigations. Access the integration menu at: https://app.datadoghq.com/integrations

Search for "webhook" and select the Webhooks integration by Datadog. Click Configure.

c. On the next page, click the New button and create a new webhook. Provide the webhook URL and auth token from Step 2.

Payload:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
{
  "eventType": "incident",
  "incidentId": "$AGGREG_KEY-$DATE_POSIX",
  "action": "created",
  "priority": "HIGH",
  "title": "$EVENT_TITLE",
  "description": "$EVENT_MSG",
  "timestamp": "$DATE_POSIX",
  "service": "$TAGS[service]",
  "data": {
    "alertType": "$ALERT_TYPE",
    "alertStatus": "$ALERT_STATUS",
    "alertTransition": "$ALERT_TRANSITION",
    "alertQuery": "$ALERT_QUERY",
    "alertScope": "$ALERT_SCOPE",
    "hostname": "$HOSTNAME",
    "link": "$LINK",
    "tags": "$TAGS"
  }
}

Custom Header (replace AUTH_TOKEN_VALUE with the actual token value from Step 2):

1
Authorization: Bearer \<AUTH_TOKEN_VALUE\>

Click Save.

d. Add a notification rule by accessing this link: https://app.datadoghq.com/monitors/settings/notifications

Create a new rule with the following values. Note that the recipient is the webhook created earlier.

e. View the current system monitors by accessing this link: https://app.datadoghq.com/monitors/manage?p=1

f. Your Datadog account has six predefined system monitors. Access the monitor named "CPU usage is high for host" by clicking on the row with the monitor. Scroll down all the way to the bottom to verify that webhooks will be sent.

g. If you wish to just test the webhook integration without triggering the monitors, you can edit the monitor and click the Test Notifications button. You will see an investigation triggered in DevOps Agent.

Step 5: Trigger and Validate the Investigation

Create a Monitor Notification Rule that routes alerts (tagged integration:host) to the webhook-AWSDevOpsAgent recipient. This ensures relevant monitors automatically trigger DevOps Agent investigations.

a. Connect to your EC2 instance via SSM Session Manager and install the stress utility:

1
sudo dnf install -y stress

b. Stress the two CPU cores by running this command (the script times out after 5 minutes):

1
stress --cpu 2 --timeout 300

c. Navigate back to the "CPU usage is high" monitor and see its status change to ALERT in the next 5 minutes.

d. Verify that a new DevOps Agent investigation is triggered.

Step 6: DevOps Agent Root Cause Analysis

Navigate to AWS console and stress-test the EC2 instance (stress --cpu 2 --timeout 300), watch the "CPU usage is high" monitor transition to ALERT, and confirm a new investigation automatically appears in DevOps Agent.

Open DevOps Agent from the AWS console page, navigate to your AgentSpace and view the findings in Operator Access.

What's Next

This same webhook-driven pattern extends beyond CPU spikes. Memory pressure, disk saturation, application latency, error rate anomalies — any Datadog monitor can now trigger an autonomous investigation. With Datadog detecting anomalies and AWS DevOps Agent autonomously investigating them, you've built a closed loop where every alert arrives with context, not just noise. The blast radius is mapped, the dependency chain is traced, and root cause analysis is underway — all before anyone's pager goes off.

Useful Links

AWS DevOps Agent — Connecting Datadog Telemetry Sources

Note: If you use a region other than US1, update the MCP endpoint in Step 1c to match the region. Refer to the Datadog site endpoints section in the link above.

📢 Published by the ES-TAM-Builder community.

Follow our ES-TAM-Builder Space for more tools, guides, and technical deep-dives from fellow TAMs

# generative-ai

Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.

Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article

Builder Center

Autonomous Incident Resolution with AWS DevOps Agent and Datadog

Shashi Shekhar

Integration Overview

The setup connects three things:

Step 1: Register Datadog as a Capability Provider

Step 2: Associate Datadog with Your Agent Space

Step 3: Deploy Datadog Monitoring

Create a Datadog API Key

Set Up AWS Infrastructure

Region Configuration (if not US1)

Step 4: Configure the Webhook in Datadog

Step 5: Trigger and Validate the Investigation

Step 6: DevOps Agent Root Cause Analysis

What's Next

Useful Links