AWS Builder Center

Autonomous Incident Resolution with AWS DevOps Agent and New Relic

This hands-on guide connects New Relic to AWS DevOps Agent via webhook, creating a closed loop: New Relic detects an anomaly, fires a webhook, and DevOps Agent autonomously investigates — mapping blast radius, tracing dependencies, and identifying root cause before anyone's pager goes off. Covers the full setup, from capability-provider registration through a live CPU-stress test and validation.

TechnicalAccountManager@AWS
Note: This article follows the same pattern as  Autonomous Incident Resolution with AWS DevOps Agent and Datadog . The key difference is that this guide uses the New Relic Observability Platform instead of Datadog as the telemetry source. If you're already familiar with the Datadog version, the overall architecture and flow remain the same — only the monitoring tool configuration steps differ.
This guide connects New Relic to DevOps Agent via webhook, creating a closed loop: New Relic detects an anomaly, fires a webhook, and DevOps Agent autonomously investigates — mapping blast radius, tracing dependencies, and identifying root cause before anyone's pager goes off. Covers full setup from registration to validation.



The Problem

As builders, our observability stacks are as varied as our architectures. Whether you run New Relic, Datadog, Splunk, or a combination of tools, the same broken incident response pattern exists underneath:
A cascading failure hits at 3 AM. CPU usage spikes past 90% on production EC2 instances. Your monitoring tool fires alerts. But then a human has to wake up, open multiple tabs, and manually correlate signals across services to figure out which alert is the cause and which are symptoms. That manual correlation is where 40–70 minutes disappear.
AWS DevOps Agent changes this equation by autonomously investigating incidents the moment they're detected — mapping the blast radius, tracing the dependency chain, and identifying root cause without waiting for a human to start the process.



Integration Overview

For customers currently using New Relic, the integration path is straightforward. This walkthrough provides step-by-step instructions to connect New Relic's alerting system directly to DevOps Agent via webhook, creating a closed loop where:
New Relic detects anomaly → Fires webhook → DevOps Agent investigates autonomously

The setup connects three things:

  1. New Relic — your monitoring layer that detects anomalies
  1. A webhook destination — the bridge that carries alert context (entity, tags, condition details) to DevOps Agent
  1. AWS DevOps Agent — the autonomous investigator that receives the alert and goes to work



Prerequisites

  • An AWS account with access to AWS DevOps Agent
  • An EC2 instance (or other workload) to monitor
  • Basic familiarity with New Relic Alerts & AI



Section A: Sign Up for New Relic

Create a New Relic account if you don't already have one.
Step 1. Navigate to  https://newrelic.com/signup  and sign up with your email address.
Step 2. Verify your email by clicking the confirmation link sent to your inbox.
Step 3. Set your password and select your Data region in the next step.
New Relic Data Region Selection

Note: The data center does not have to match your physical location. By selecting a non-US based data region, you will incur an additional $.05 per GB Ingested if you upgrade your plan to a paid plan.
Step 4. Once logged in, click the user icon in the top-right corner of the New Relic UI and select API Keys.
New Relic API Keys Menu

Step 5. Create a key of key type Ingest - License. This key is used by the infrastructure agent running on your EC2 instance to send telemetry data to New Relic.
New Relic Create API Key

⚠️ Save the key value securely — the actual value will only be displayed once.
Step 6. Click Create a key again and create a User key. This key (format:  NRAK-XXXXXXXXXXXXXXXXXXXXX ) is used for NerdGraph API access and MCP authentication.
New Relic User Key

⚠️ Important: Save both key values securely (e.g., in a secrets manager or password vault). You will need these values in later steps. The Ingest key is used when installing the New Relic agent on EC2, and the User key is used for API and MCP access.
Step 7. Note your Account ID — this is the seven-digit number visible in the URL bar when you're logged in (e.g.,  https://one.newrelic.com/nr1-core?account=1234567 ). You can also find it under Administration → Access Management → Accounts. You'll need this Account ID when configuring the DevOps Agent capability provider and the MCP server connection.
New Relic Administration

Step 8. Navigate to Administration → Previews & Trials and upgrade your account to the next cheapest tier (Standard). This is required to enable preview features such as the MCP server integration.
New Relic Upgrade Plan

Step 9. Search for "MCP" in the search box on the Previews & Trials page and opt in for New Relic AI MCP Server by clicking Join Preview (or Rejoin Preview if you previously opted out).
New Relic MCP Preview Opt-In




Section B: Register New Relic as a Capability Provider

In the DevOps Agent console, navigate to Capability Providers, find New Relic under Telemetry, and click Register. Complete the authentication flow to establish the trust relationship.
Step 1. Navigate to AWS DevOps Agent in the AWS console.
Step 2. Access the Capability Providers menu.
Step 3. Find New Relic under Telemetry. Click Register.
Step 4. Supply the required values and click Add:
  • Name: A unique name to identify this registration (e.g.,  my-newrelic-server ). 1–128 characters, letters, numbers, hyphens, and underscores only.
  • Account ID: Your seven-digit New Relic Account ID from Section A.
  • API Key: Your New Relic User API key (format:  NRAK-XXXXXXXXXXXXXXXXXXXXX ) created in Section A.
  • Region: Select US (United States region) or EU (Europe region) to match the data region you chose during sign-up.
New Relic Capability Provider Values

Step 5. Complete the authorization flow and confirm the integration is established.



Section C: Associate New Relic with Your Agent Space

Add the New Relic integration to your Agent Space's Telemetry section. Upon saving, you receive a Webhook URL and Auth Token — save these immediately, they're shown only once.
Step 1. Navigate to your Agent Space and scroll to the Telemetry section.
Step 2. Click Add next to New Relic integration.
Step 3. Click Save.
Step 4. Note the webhook URL and auth token values.
⚠️ Important: The webhook URL and auth token are shown only once. Save them immediately.



Section D: Connect New Relic to AWS Account

Link your New Relic account to AWS so that New Relic can pull metadata and metrics from your AWS environment.
Step 1. In New Relic, navigate to Infrastructure → AWS.
New Relic Infrastructure AWS

Step 2. Click Add an AWS account.
Step 3. On the Integrate your AWS account page, select the data type(s) you want to ingest. Choose Metrics to spot trends, monitor performance, and stay up-to-date with alerts. Optionally select Logs to capture stack diagnostics. Click Continue.
New Relic Integrate AWS Account - Choose Data Type

Step 4. On the Choose a setup method step, select Automate AWS with CloudFormation (Recommended). This automatically configures the required IAM permissions to integrate your AWS account.
New Relic Integrate AWS Account - Choose Setup Method

Step 5. On the Choose a metrics integration type step, select CloudWatch Metric Streams (Recommended). This integrates all your AWS services with CloudWatch Metric Streams for up-to-date data and faster troubleshooting.
New Relic Integrate AWS Account - Choose Metrics Integration Type

Step 6. Name your CloudWatch Metric Stream (a default name is provided, or customize it) and click Continue.
New Relic Integrate AWS Account - Name CloudWatch Metric Stream

Step 7. On the Enable Auto Discovery step, review the options and click Continue.
New Relic Integrate AWS Account - Enable Auto Discovery

Step 8. Click Launch CloudFormation in AWS to create the required IAM role and CloudWatch Metric Stream resources in your AWS account.
Note: You may be asked to log in to your AWS account.
New Relic Integrate AWS Account - Launch CloudFormation

Step 9. In the AWS CloudFormation console, fill in the required parameters and leave the rest as default:
  • NewRelic account ID — your New Relic Account ID from Section A (may be auto-populated).
  • NewRelic API Key — your New Relic User API key (format:  NRAK-XXXXXXXXXXXXXXXXXXXXX ) from Section A.
  • New Relic Ingest License Key — your Ingest - License key from Section A.
Leave all other values as default and click Create stack to deploy the resources.
AWS CloudFormation Stack Parameters for New Relic Integration

Step 10. Scroll to the bottom of the page, check the acknowledgment checkboxes confirming that AWS CloudFormation might create IAM resources with custom names, and click Create stack.
AWS CloudFormation Acknowledge and Create Stack

Note: The stack creation will create one main stack with multiple nested stacks and will take several minutes.



Section E: Create New Relic Workflow

Create a workflow in New Relic that will route alert notifications to the DevOps Agent webhook destination.
Step 1. In New Relic, navigate to Alerts → Destinations (under the Enrich & Notify section in the left sidebar). Select Webhook from the available destination types.
New Relic Alerts - Destinations - Select Webhook

Step 2. Configure the new webhook destination:
  • Destination name:  AWSDevOpsAgent 
  • Endpoint URL: The Webhook URL from Section C.
  • Use authorization: Select Custom Header with:
  • Key:  Authorization 
  • Value:  Bearer <AUTH_TOKEN_VALUE>  (use the auth token from Section C; note there is one space between  Bearer  and the token value)
Click Save destination.
New Relic Webhook Destination Configuration

Step 3. Navigate to Alerts → Alert Conditions and click + New alert condition to create a new condition.
New Relic Alerts - Alert Conditions - New Alert Condition

Step 4. Select the Write your own query option.
New Relic Alert Condition - Write Your Own Query

Step 5. Enter the following NRQL query to monitor EC2 CPU utilization and click Next:
SELECT max(provider.cpuUtilization.Maximum) FROM ComputeSample WHERE provider = 'Ec2Instance'
New Relic Alert Condition - NRQL Query

Step 6. Ignore any warning or error messages such as "Your condition configuration is invalid. No value returned from NRQL query. Adjust your query to return a value." Scroll down to the Set condition thresholds section and configure the threshold to open an alert when the query returns a value above 60 for at least 1 minute. Click Next.
New Relic Alert Condition - Set Condition Thresholds

Step 7. On the Add details step, name your alert condition (e.g.,  CPUAbove60AlertCondition ) and create a policy name (e.g.,  CPUAbove60AlertPolicy ). Leave the Group alert events into issues setting as default. Click Save & set up notifications.
New Relic Alert Condition - Add Details and Save

With the alert condition and destination configured, you can now hook them together with a workflow. This way, whenever CPU utilization exceeds 60%, the alert condition will trigger and a webhook will be sent to the DevOps Agent destination.
Step 8. Navigate to Alerts → Workflows and click Add a workflow.
Step 9. Name the workflow  DevOps Agent Investigation Workflow . Under Filter data, set the Policy filter to  CPUAbove60AlertPolicy  (the policy created in Step 7).
New Relic Workflow - Configure Filter

Step 10. Under Notify, click Add destination and select the  AWSDevOpsAgent  webhook destination created earlier. Edit the payload template to use the following:
{
"eventType": "incident",
"incidentId": "{{issueId}}",
"action": "{{#eq state 'CLOSED'}}resolved{{else}}created{{/eq}}",
"priority": "{{priority}}",
"title": "{{annotations.title.[0]}}",
"description": "{{accumulations.conditionDescription.[0]}}",
"timestamp": "{{createdAt}}",
"service": "{{accumulations.conditionProduct.[0]}}",
"data": {
"alertType": "{{accumulations.conditionName.[0]}}",
"alertStatus": "{{state}}",
"link": "{{issuePageUrl}}"
}
}
New Relic Workflow - Notify Webhook Payload

You can click Send test notification to verify connectivity to the DevOps Agent webhook. Click Save message to save the notification configuration.
New Relic Workflow - Send Test Notification and Save Message

Step 11. Click Activate workflow.
Note: You can create multiple workflows with different filters to route different severity levels or environments to different agent spaces.



Section F: Stress Test an EC2 Instance

Set Up AWS Infrastructure

Spin up a new t3.micro AL2023 EC2 instance. Attach the following policies to the instance role:
  •  AmazonSSMManagedInstanceCore 
  •  CloudWatchAgentServerPolicy 

Stress the CPU

Connect to the EC2 instance via SSM Session Manager and run:
# Install the stress utility
sudo dnf install -y stress

# Stress the CPU cores (times out after 5 minutes)
stress --cpu 2 --timeout 300

Verify the Alert

Navigate to Alerts → Issues & Activity in New Relic and observe the alert condition transition to an open issue. Verify that a new DevOps Agent investigation is triggered in the AWS console.
Note: It may take 10–15 minutes for the alert condition to trigger. This is due to the end-to-end data pipeline: EC2 basic monitoring sends CPU metrics to CloudWatch every 5 minutes, CloudWatch Metric Streams delivers data to New Relic via Kinesis Firehose (1–3 minutes), and then New Relic needs at least one full evaluation cycle to confirm the threshold is breached.
You can also view the notification log for the workflow and verify that alerts were sent to the webhook destination.
New Relic Workflow - Notification Log




Section G: DevOps Agent Root Cause Analysis

Open DevOps Agent from the AWS console, navigate to your AgentSpace, and view the findings in Operator Access. You should see an investigation triggered by the New Relic MCP Server with status In Progress.
DevOps Agent Investigation Triggered by New Relic

DevOps Agent will autonomously:
  • Map the blast radius of affected services
  • Trace the dependency chain
  • Correlate telemetry signals from New Relic
  • Identify probable root cause
  • Present findings with supporting evidence



Section H: Cleanup

Once you are done testing, clean up the resources to avoid unnecessary costs.
Step 1. Log in to New Relic.
Step 2. Delete the Workflow. Navigate to Alerts → Workflows, find the  DevOps Agent Investigation Workflow , and delete it.
New Relic - Delete Workflow

Step 3. Delete the Alert Policy. Navigate to Alerts → Alert Policies, find  CPUAbove60AlertPolicy , and delete it.
New Relic - Delete Alert Policy

Step 4. Delete the Alert Condition. Navigate to Alerts → Alert Conditions, find  CPUAbove60AlertCondition , and delete it.
New Relic - Delete Alert Condition

Step 5. Delete the Destination. Navigate to Alerts → Destinations, find the  AWSDevOpsAgent  webhook destination, and delete it.
New Relic - Delete Destination

Step 6. Go to one.newrelic.com > All capabilities > Infrastructure > AWS > Manage services. From your Edit AWS account page, select Unlink this account.
Step 7. Log in to the AWS Console and navigate to CloudFormation. Find the nested stack with name  NewRelic-Metric-Streams-NewrelicMetricStreamsStack-xxxxxxxxxxxx . In the Resources tab, locate the S3 bucket with Logical ID  S3FirehoseEventsBucket . Empty the bucket if it's not empty.
Step 8. Delete the CloudFormation stack (default name is  NewRelic-Metric-Streams ). Make sure the stack is deleted successfully.
Step 9. In the AWS Console, navigate to IAM → Roles, find the role named  NewRelicInfrastructure-Integrations , and delete it.
Step 10. In the AWS Console, navigate to EC2 → Instances, find the EC2 instance you launched earlier, and terminate it.
Step 11. (Optional) If you no longer need the New Relic account's paid features, downgrade it to the Free tier. Navigate to [user menu] → Administration → Access management → Plans and billing, and select the option to downgrade your edition. See  New Relic documentation  for details.
Step 12. In the AWS DevOps Agent console, navigate to your Agent Space and remove the New Relic integration from the Telemetry section. Alternatively, delete the Agent Space entirely if it was created only for testing.
Step 13. Navigate to Capability Providers in the DevOps Agent console and deregister the New Relic capability provider.



What's Next

This same webhook-driven pattern extends beyond CPU spikes. Memory pressure, disk saturation, application latency, error rate anomalies, Apdex score degradation — any New Relic alert condition can now trigger an autonomous investigation.
With New Relic detecting anomalies and AWS DevOps Agent autonomously investigating them, you've built a closed loop where every alert arrives with context, not just noise. The blast radius is mapped, the dependency chain is traced, and root cause analysis is underway — all before anyone's pager goes off.

Ideas to Explore

  • Connect APM conditions (transaction errors, throughput drops) to DevOps Agent
  • Route Synthetic monitor failures for automated investigation
  • Use NRQL alert conditions for custom business metric anomalies
  • Integrate with New Relic AIOps for correlated issue grouping before handoff



Useful Links



📢 Published by the ES-TAM-Builder community.
Follow our  ES-TAM-Builder  Space for more tools, guides, and technical deep-dives from fellow TAMs
 # generative-ai 
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article