Autonomous Incident Resolution with AWS DevOps Agent and New Relic
This hands-on guide connects New Relic to AWS DevOps Agent via webhook, creating a closed loop: New Relic detects an anomaly, fires a webhook, and DevOps Agent autonomously investigates — mapping blast radius, tracing dependencies, and identifying root cause before anyone's pager goes off. Covers the full setup, from capability-provider registration through a live CPU-stress test and validation.
Note: This article follows the same pattern as Autonomous Incident Resolution with AWS DevOps Agent and Datadog . The key difference is that this guide uses the New Relic Observability Platform instead of Datadog as the telemetry source. If you're already familiar with the Datadog version, the overall architecture and flow remain the same — only the monitoring tool configuration steps differ.
This guide connects New Relic to DevOps Agent via webhook, creating a closed loop: New Relic detects an anomaly, fires a webhook, and DevOps Agent autonomously investigates — mapping blast radius, tracing dependencies, and identifying root cause before anyone's pager goes off. Covers full setup from registration to validation.
The Problem
As builders, our observability stacks are as varied as our architectures. Whether you run New Relic, Datadog, Splunk, or a combination of tools, the same broken incident response pattern exists underneath:
A cascading failure hits at 3 AM. CPU usage spikes past 90% on production EC2 instances. Your monitoring tool fires alerts. But then a human has to wake up, open multiple tabs, and manually correlate signals across services to figure out which alert is the cause and which are symptoms. That manual correlation is where 40–70 minutes disappear.
AWS DevOps Agent changes this equation by autonomously investigating incidents the moment they're detected — mapping the blast radius, tracing the dependency chain, and identifying root cause without waiting for a human to start the process.
Integration Overview
For customers currently using New Relic, the integration path is straightforward. This walkthrough provides step-by-step instructions to connect New Relic's alerting system directly to DevOps Agent via webhook, creating a closed loop where:
New Relic detects anomaly → Fires webhook → DevOps Agent investigates autonomously
The setup connects three things:
- New Relic — your monitoring layer that detects anomalies
- A webhook destination — the bridge that carries alert context (entity, tags, condition details) to DevOps Agent
- AWS DevOps Agent — the autonomous investigator that receives the alert and goes to work
Prerequisites
- An AWS account with access to AWS DevOps Agent
- A New Relic account (free tier available at https://newrelic.com/signup )
- An EC2 instance (or other workload) to monitor
- Basic familiarity with New Relic Alerts & AI
Section A: Sign Up for New Relic
Create a New Relic account if you don't already have one.
Step 2. Verify your email by clicking the confirmation link sent to your inbox.
Step 3. Set your password and select your Data region in the next step.

Note: The data center does not have to match your physical location. By selecting a non-US based data region, you will incur an additional $.05 per GB Ingested if you upgrade your plan to a paid plan.
Step 4. Once logged in, click the user icon in the top-right corner of the New Relic UI and select API Keys.

Step 5. Create a key of key type Ingest - License. This key is used by the infrastructure agent running on your EC2 instance to send telemetry data to New Relic.

⚠️ Save the key value securely — the actual value will only be displayed once.
Step 6. Click Create a key again and create a User key. This key (format:
NRAK-XXXXXXXXXXXXXXXXXXXXX ) is used for NerdGraph API access and MCP authentication.
⚠️ Important: Save both key values securely (e.g., in a secrets manager or password vault). You will need these values in later steps. The Ingest key is used when installing the New Relic agent on EC2, and the User key is used for API and MCP access.
Step 7. Note your Account ID — this is the seven-digit number visible in the URL bar when you're logged in (e.g.,
https://one.newrelic.com/nr1-core?account=1234567 ). You can also find it under Administration → Access Management → Accounts. You'll need this Account ID when configuring the DevOps Agent capability provider and the MCP server connection.
Step 8. Navigate to Administration → Previews & Trials and upgrade your account to the next cheapest tier (Standard). This is required to enable preview features such as the MCP server integration.

Step 9. Search for "MCP" in the search box on the Previews & Trials page and opt in for New Relic AI MCP Server by clicking Join Preview (or Rejoin Preview if you previously opted out).

Section B: Register New Relic as a Capability Provider
In the DevOps Agent console, navigate to Capability Providers, find New Relic under Telemetry, and click Register. Complete the authentication flow to establish the trust relationship.
Step 1. Navigate to AWS DevOps Agent in the AWS console.
Step 2. Access the Capability Providers menu.
Step 3. Find New Relic under Telemetry. Click Register.
Step 4. Supply the required values and click Add:
- Name: A unique name to identify this registration (e.g.,
my-newrelic-server). 1–128 characters, letters, numbers, hyphens, and underscores only.
- Account ID: Your seven-digit New Relic Account ID from Section A.
- API Key: Your New Relic User API key (format:
NRAK-XXXXXXXXXXXXXXXXXXXXX) created in Section A.
- Region: Select US (United States region) or EU (Europe region) to match the data region you chose during sign-up.

Step 5. Complete the authorization flow and confirm the integration is established.
Section C: Associate New Relic with Your Agent Space
Add the New Relic integration to your Agent Space's Telemetry section. Upon saving, you receive a Webhook URL and Auth Token — save these immediately, they're shown only once.
Step 1. Navigate to your Agent Space and scroll to the Telemetry section.
Step 2. Click Add next to New Relic integration.
Step 3. Click Save.
Step 4. Note the webhook URL and auth token values.
⚠️ Important: The webhook URL and auth token are shown only once. Save them immediately.
Section D: Connect New Relic to AWS Account
Link your New Relic account to AWS so that New Relic can pull metadata and metrics from your AWS environment.
Step 1. In New Relic, navigate to Infrastructure → AWS.

Step 2. Click Add an AWS account.
Step 3. On the Integrate your AWS account page, select the data type(s) you want to ingest. Choose Metrics to spot trends, monitor performance, and stay up-to-date with alerts. Optionally select Logs to capture stack diagnostics. Click Continue.

Step 4. On the Choose a setup method step, select Automate AWS with CloudFormation (Recommended). This automatically configures the required IAM permissions to integrate your AWS account.

Step 5. On the Choose a metrics integration type step, select CloudWatch Metric Streams (Recommended). This integrates all your AWS services with CloudWatch Metric Streams for up-to-date data and faster troubleshooting.

Step 6. Name your CloudWatch Metric Stream (a default name is provided, or customize it) and click Continue.

Step 7. On the Enable Auto Discovery step, review the options and click Continue.

Step 8. Click Launch CloudFormation in AWS to create the required IAM role and CloudWatch Metric Stream resources in your AWS account.
Note: You may be asked to log in to your AWS account.

Step 9. In the AWS CloudFormation console, fill in the required parameters and leave the rest as default:
- NewRelic account ID — your New Relic Account ID from Section A (may be auto-populated).
- NewRelic API Key — your New Relic User API key (format:
NRAK-XXXXXXXXXXXXXXXXXXXXX) from Section A.
- New Relic Ingest License Key — your Ingest - License key from Section A.
Leave all other values as default and click Create stack to deploy the resources.

Step 10. Scroll to the bottom of the page, check the acknowledgment checkboxes confirming that AWS CloudFormation might create IAM resources with custom names, and click Create stack.

Note: The stack creation will create one main stack with multiple nested stacks and will take several minutes.
Section E: Create New Relic Workflow
Create a workflow in New Relic that will route alert notifications to the DevOps Agent webhook destination.
Step 1. In New Relic, navigate to Alerts → Destinations (under the Enrich & Notify section in the left sidebar). Select Webhook from the available destination types.

Step 2. Configure the new webhook destination:
- Destination name:
AWSDevOpsAgent
- Endpoint URL: The Webhook URL from Section C.
- Use authorization: Select Custom Header with:
- Key:
Authorization
- Value:
Bearer <AUTH_TOKEN_VALUE>(use the auth token from Section C; note there is one space betweenBearerand the token value)
Click Save destination.

Step 3. Navigate to Alerts → Alert Conditions and click + New alert condition to create a new condition.

Step 4. Select the Write your own query option.

Step 5. Enter the following NRQL query to monitor EC2 CPU utilization and click Next:
SELECT max(provider.cpuUtilization.Maximum) FROM ComputeSample WHERE provider = 'Ec2Instance'
Step 6. Ignore any warning or error messages such as "Your condition configuration is invalid. No value returned from NRQL query. Adjust your query to return a value." Scroll down to the Set condition thresholds section and configure the threshold to open an alert when the query returns a value above 60 for at least 1 minute. Click Next.

Step 7. On the Add details step, name your alert condition (e.g.,
CPUAbove60AlertCondition ) and create a policy name (e.g., CPUAbove60AlertPolicy ). Leave the Group alert events into issues setting as default. Click Save & set up notifications.
With the alert condition and destination configured, you can now hook them together with a workflow. This way, whenever CPU utilization exceeds 60%, the alert condition will trigger and a webhook will be sent to the DevOps Agent destination.
Step 8. Navigate to Alerts → Workflows and click Add a workflow.
Step 9. Name the workflow
DevOps Agent Investigation Workflow . Under Filter data, set the Policy filter to CPUAbove60AlertPolicy (the policy created in Step 7).
Step 10. Under Notify, click Add destination and select the
AWSDevOpsAgent webhook destination created earlier. Edit the payload template to use the following:{ "eventType": "incident", "incidentId": "{{issueId}}", "action": "{{#eq state 'CLOSED'}}resolved{{else}}created{{/eq}}", "priority": "{{priority}}", "title": "{{annotations.title.[0]}}", "description": "{{accumulations.conditionDescription.[0]}}", "timestamp": "{{createdAt}}", "service": "{{accumulations.conditionProduct.[0]}}", "data": { "alertType": "{{accumulations.conditionName.[0]}}", "alertStatus": "{{state}}", "link": "{{issuePageUrl}}" }}
You can click Send test notification to verify connectivity to the DevOps Agent webhook. Click Save message to save the notification configuration.

Step 11. Click Activate workflow.
Note: You can create multiple workflows with different filters to route different severity levels or environments to different agent spaces.
Section F: Stress Test an EC2 Instance
Set Up AWS Infrastructure
Spin up a new t3.micro AL2023 EC2 instance. Attach the following policies to the instance role:
-
AmazonSSMManagedInstanceCore
-
CloudWatchAgentServerPolicy
Stress the CPU
Connect to the EC2 instance via SSM Session Manager and run:
# Install the stress utilitysudo dnf install -y stress
# Stress the CPU cores (times out after 5 minutes)stress --cpu 2 --timeout 300Verify the Alert
Navigate to Alerts → Issues & Activity in New Relic and observe the alert condition transition to an open issue. Verify that a new DevOps Agent investigation is triggered in the AWS console.
Note: It may take 10–15 minutes for the alert condition to trigger. This is due to the end-to-end data pipeline: EC2 basic monitoring sends CPU metrics to CloudWatch every 5 minutes, CloudWatch Metric Streams delivers data to New Relic via Kinesis Firehose (1–3 minutes), and then New Relic needs at least one full evaluation cycle to confirm the threshold is breached.
You can also view the notification log for the workflow and verify that alerts were sent to the webhook destination.

Section G: DevOps Agent Root Cause Analysis
Open DevOps Agent from the AWS console, navigate to your AgentSpace, and view the findings in Operator Access. You should see an investigation triggered by the New Relic MCP Server with status In Progress.

DevOps Agent will autonomously:
- Map the blast radius of affected services
- Trace the dependency chain
- Correlate telemetry signals from New Relic
- Identify probable root cause
- Present findings with supporting evidence
Section H: Cleanup
Once you are done testing, clean up the resources to avoid unnecessary costs.
Step 1. Log in to New Relic.
Step 2. Delete the Workflow. Navigate to Alerts → Workflows, find the
DevOps Agent Investigation Workflow , and delete it.
Step 3. Delete the Alert Policy. Navigate to Alerts → Alert Policies, find
CPUAbove60AlertPolicy , and delete it.
Step 4. Delete the Alert Condition. Navigate to Alerts → Alert Conditions, find
CPUAbove60AlertCondition , and delete it.
Step 5. Delete the Destination. Navigate to Alerts → Destinations, find the
AWSDevOpsAgent webhook destination, and delete it.
Step 6. Go to one.newrelic.com > All capabilities > Infrastructure > AWS > Manage services. From your Edit AWS account page, select Unlink this account.
Step 7. Log in to the AWS Console and navigate to CloudFormation. Find the nested stack with name
NewRelic-Metric-Streams-NewrelicMetricStreamsStack-xxxxxxxxxxxx . In the Resources tab, locate the S3 bucket with Logical ID S3FirehoseEventsBucket . Empty the bucket if it's not empty.Step 8. Delete the CloudFormation stack (default name is
NewRelic-Metric-Streams ). Make sure the stack is deleted successfully.Step 9. In the AWS Console, navigate to IAM → Roles, find the role named
NewRelicInfrastructure-Integrations , and delete it.Step 10. In the AWS Console, navigate to EC2 → Instances, find the EC2 instance you launched earlier, and terminate it.
Step 11. (Optional) If you no longer need the New Relic account's paid features, downgrade it to the Free tier. Navigate to [user menu] → Administration → Access management → Plans and billing, and select the option to downgrade your edition. See New Relic documentation for details.
Step 12. In the AWS DevOps Agent console, navigate to your Agent Space and remove the New Relic integration from the Telemetry section. Alternatively, delete the Agent Space entirely if it was created only for testing.
Step 13. Navigate to Capability Providers in the DevOps Agent console and deregister the New Relic capability provider.
What's Next
This same webhook-driven pattern extends beyond CPU spikes. Memory pressure, disk saturation, application latency, error rate anomalies, Apdex score degradation — any New Relic alert condition can now trigger an autonomous investigation.
With New Relic detecting anomalies and AWS DevOps Agent autonomously investigating them, you've built a closed loop where every alert arrives with context, not just noise. The blast radius is mapped, the dependency chain is traced, and root cause analysis is underway — all before anyone's pager goes off.
Ideas to Explore
- Connect APM conditions (transaction errors, throughput drops) to DevOps Agent
- Route Synthetic monitor failures for automated investigation
- Use NRQL alert conditions for custom business metric anomalies
- Integrate with New Relic AIOps for correlated issue grouping before handoff
Useful Links
- Autonomous Incident Resolution with AWS DevOps Agent and Datadog — the companion article using Datadog
📢 Published by the ES-TAM-Builder community.
# generative-ai Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article