AWS Builder Center
Automate incident investigation: connect Amazon Managed Grafana alerts to AWS DevOps Agent

Automate incident investigation: connect Amazon Managed Grafana alerts to AWS DevOps Agent

Amazon Managed Grafana can't call a webhook directly, so how do you make a Grafana alert kick off an AWS DevOps Agent investigation automatically? This guide shows the supported AWS SNS + Lambda pattern, end to end, including the cross-account setup.

Technical Account Manager|WWPS
Monitoring tells you something broke. This guide connects that signal to an AI agent that starts investigating on its own — no human needed to kick it off.
When an alert fires in Amazon Managed Grafana, wouldn't it be great if an investigation just… started? This walkthrough connects Amazon Managed Grafana to the AWS DevOps Agent so that a breaching metric automatically opens an investigation — and the agent can read your Grafana dashboards while it works.
No prior experience with either service is assumed. Every step explains what you're doing, why, and how to check it worked.
Time to complete: ~45–60 minutes • Level: intermediate (AWS Console + a few CLI commands)

What you'll build

By the end, you'll have this automatic flow:
A metric crosses a threshold in Grafana → an alert fires → the AWS DevOps Agent automatically opens an investigation and starts digging in, even pulling data back from your Grafana dashboards.
Think of it as a "call for help" button that dials the AI agent the moment a real problem appears.

How it works (in plain English)

There are two connections between Grafana and the DevOps Agent, and you'll set up both:
  1. The "trigger" connection — how an alert starts an investigation.
  1. The "read" connection — how the agent looks at your Grafana dashboards once it's investigating.
Here's the key thing to know about the trigger connection:
Amazon Managed Grafana cannot call a webhook directly. It can, however, send a message to Amazon SNS. So we put a small AWS Lambda function in the middle: SNS receives the alert, Lambda reformats it and forwards it to the DevOps Agent. This is a normal, supported pattern — just one extra hop.
If you use self-managed Grafana, Grafana OSS, or Grafana Cloud (not the Amazon-managed version), those can call a webhook directly and you can skip SNS and Lambda. Everything else here still applies.

Architecture

AWS Secrets Manager
(stores webhook URL + secret)
|
reads |
v
Grafana alert --> Amazon SNS --> AWS Lambda --> DevOps Agent webhook --> Investigation
fires (mailbox) (translator) (the "call for help") starts

...and separately, the agent reads your Grafana dashboards while it investigates
Why cross-account works with no extra networking: neither connection uses cross-account IAM roles. The trigger path is authenticated by the webhook secret over public HTTPS; the read path is authenticated by a Grafana service-account token. Both endpoints are public HTTPS.
Throughout this guide:
  • "Grafana account" = the account where Amazon Managed Grafana lives.
  • "Agent account" = the account where the DevOps Agent Space lives.
  • Single account? They're the same — just do everything in one place.

A quick glossary

Term
What it means, simply
Agent Space
Your workspace in the DevOps Agent console where investigations happen.
Webhook
A special URL. Send it a message and it triggers something — here, an investigation.
SNS topic
An AWS "mailbox" that receives messages and passes them on.
Lambda
A small piece of code that runs on demand — here, it reformats and forwards the alert.
Contact point
In Grafana, where an alert sends its notification (e.g., to SNS).

Before you begin

Make sure you have:
  • An AWS DevOps Agent Space created.
  • An Amazon Managed Grafana workspace, Grafana version 9 or later, status Active.
  • Permission in the Grafana account to create SNS topics, Lambda functions, IAM policies, and Secrets Manager secrets.
  • The ability to log in to the Grafana web UI (to add a service account, contact point, and alert rule).

The setup at a glance

Part A — In the DevOps Agent (Agent account)
  • Step 1: Let the agent read your Grafana dashboards
  • Step 2: Create the webhook (your "call for help" URL)
Part B — In Amazon Managed Grafana (Grafana account)
  • Step 3: Create the SNS mailbox + store the webhook secret safely
  • Step 4: Deploy the Lambda translator + give it permissions
  • Step 5: Connect Grafana's alerts to SNS
  • Step 6: Create an alert rule and test the whole thing



Part A — In the DevOps Agent

Step 1 — Let the agent read your Grafana dashboards

Why: When the agent investigates, it's far more useful if it can look at the same dashboards and metrics you do. This gives it read-only access.
First, in the Grafana web UI, create a read-only login for the agent:
  • Go to Administration → Service accounts → Add service account.
  • Give it the Viewer role.
  • Choose Add service account token and copy the token (starts with  glsa_ ).
Heads up: Amazon Managed Grafana tokens expire after 30 days maximum. Set a reminder to rotate it — if it expires, the agent quietly loses access.
Then, in the DevOps Agent console, register Grafana:
  • Go to Capability Providers → Telemetry → Grafana → Register.
  • Service Name: any friendly name, e.g.  my-grafana .
  • Grafana URL: e.g.  https://g-xxxxxxxxxx.grafana-workspace.us-east-1.amazonaws.com .
  • Service Account Access Token: paste the  glsa_...  token.
  • Choose Submit.
Finally, attach it to your Agent Space: open the Space → Capabilities → Telemetry → Add → select the registration → Save.
Check it worked: On your Agent Space's Capabilities tab, Grafana now appears under Telemetry.

Step 2 — Create the webhook (your "call for help" URL)

Why: This is the URL that, when sent a message, tells the DevOps Agent to start an investigation.
In the DevOps Agent console, open your Agent Space → Capabilities tab. Find the Agent Space Webhook table and choose Add webhook. Then pick how the webhook is secured:
Option
What it means
When to choose it
HMAC (recommended)
Each message is cryptographically signed. More secure.
Best default; adds tamper-proofing and replay protection.
API key (bearer token)
A simple secret token sent with each message.
Simpler; fine over trusted HTTPS.
If you configure the Grafana integration in the console and it auto-creates a webhook, that one uses the API key (bearer) type.
Immediately copy and save the webhook URL and its secret. The secret is shown only once. The URL looks like:
https://event-ai.us-east-1.api.aws/webhook/generic/<WEBHOOK_ID>
Lost the secret? Open the webhook → Actions → Edit → Rotate webhook. The URL stays the same; you get a new secret.
Check it worked: You have the webhook URL and secret saved safely — you'll use them in Step 3.

Reference: the webhook payload format

You won't build this by hand (the Lambda does it), but for reference the webhook accepts a JSON message like this:
{
"eventType": "incident",
"incidentId": "string",
"action": "created|updated|closed|resolved",
"priority": "CRITICAL|HIGH|MEDIUM|LOW|MINIMAL",
"title": "string",
"description": "string (optional)",
"timestamp": "ISO-8601 (optional)",
"service": "string (optional)",
"data": {}
}
Notes on the fields:
  •  incidentId  must be unique; repeats are ignored.
  •  data  is accepted, but not read by the agent.
  • Only  title ,  description ,  priority , and the incident ID reach the agent — so put the useful details in  description . (The Lambda fills this from your Grafana alert's  summary  text.)



Part B — In Amazon Managed Grafana

For all CLI commands below, replace  <ACCOUNT_ID>  with your Grafana account ID and adjust the region if you're not using  us-east-1 .

Step 3 — Create the SNS mailbox + store the secret safely

Why: SNS is the mailbox Grafana can send to, and Secrets Manager keeps the webhook secret out of your code.
Create the SNS topic. Name it starting with  grafana  — Grafana's default permissions only allow it to send to topics named  grafana* :
aws sns create-topic --name grafana-devops-agent-alerts --region us-east-1
Store the webhook URL + secret from Step 2:
aws secretsmanager create-secret \
--name devops-agent/webhook \
--secret-string '{"url":"https://event-ai.us-east-1.api.aws/webhook/generic/<WEBHOOK_ID>","secret":"<WEBHOOK_SECRET>"}' \
--region us-east-1
Check it worked: Both commands return an ARN with no errors.

Step 4 — Deploy the Lambda translator + give it permissions

Why: Grafana's alert format and the webhook's expected format differ. This Lambda translates between them, authenticates the message, and forwards it. It supports HMAC or bearer auth via the  AUTH_MODE  environment variable.
Save this as  lambda_function.py :
import base64, hashlib, hmac, json, logging, os, urllib.request
from datetime import datetime, timezone
import boto3

logger = logging.getLogger(); logger.setLevel(logging.INFO)
SECRET_ID = os.environ.get("SECRET_ID", "devops-agent/webhook")
AUTH_MODE = os.environ.get("AUTH_MODE", "hmac").lower() # "hmac" or "bearer"
PRIORITY_MAP = {"critical":"CRITICAL","high":"HIGH","warning":"MEDIUM",
"medium":"MEDIUM","low":"LOW","info":"MINIMAL","minimal":"MINIMAL"}
_sm = boto3.client("secretsmanager"); _cfg = None

def _get_config():
global _cfg
if _cfg is None:
_cfg = json.loads(_sm.get_secret_value(SecretId=SECRET_ID)["SecretString"])
return _cfg

def _iso_now():
return datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%S.000Z")

def _build_payload(message):
alert, labels, annotations, status = {}, {}, {}, "firing"
if isinstance(message, dict):
status = message.get("status", "firing")
alerts = message.get("alerts") or []
if alerts:
alert = alerts[0]; labels = alert.get("labels", {}) or {}
annotations = alert.get("annotations", {}) or {}
name = labels.get("alertname", "GrafanaAlert")
fp = alert.get("fingerprint") or str(abs(hash(json.dumps(labels, sort_keys=True))))
sev = str(labels.get("severity", "high")).lower()
desc = annotations.get("summary") or annotations.get("description") or name
return {
"eventType": "incident",
"incidentId": f"{name}-{fp}",
"action": "resolved" if status == "resolved" else "created",
"priority": PRIORITY_MAP.get(sev, "HIGH"),
"title": name, "description": desc,
"service": labels.get("job", "amazon-managed-grafana"),
"timestamp": alert.get("startsAt") or _iso_now(),
"data": {"metadata": {**labels, "_source": "amazon-managed-grafana"}},
}

def _post(url, secret, payload):
body = json.dumps(payload); ts = _iso_now()
headers = {"Content-Type": "application/json", "x-amzn-event-timestamp": ts}
if AUTH_MODE == "bearer":
headers["Authorization"] = f"Bearer {secret}"
else:
sig = base64.b64encode(hmac.new(secret.encode(), f"{ts}:{body}".encode(),
hashlib.sha256).digest()).decode()
headers["x-amzn-event-signature"] = sig
req = urllib.request.Request(url, data=body.encode(), method="POST", headers=headers)
with urllib.request.urlopen(req, timeout=10) as resp:
return resp.status, resp.read().decode(errors="replace")

def lambda_handler(event, context):
cfg = _get_config(); results = []
for record in event.get("Records", []):
raw = record["Sns"]["Message"]
try: message = json.loads(raw)
except (json.JSONDecodeError, TypeError): message = raw
payload = _build_payload(message)
status, text = _post(cfg["url"], cfg["secret"], payload)
logger.info("Webhook responded %s for %s: %s", status, payload["incidentId"], text)
results.append({"incidentId": payload["incidentId"], "status": status})
return {"processed": len(results), "results": results}
Package and create the function (Python 3.12+). Set  AUTH_MODE  to match the webhook type you chose in Step 2 ( hmac  or  bearer ):
zip function.zip lambda_function.py
aws lambda create-function --function-name devops-webhook \
--runtime python3.12 --handler lambda_function.lambda_handler \
--role arn:aws:iam::<ACCOUNT_ID>:role/<lambda-role> \
--environment "Variables={SECRET_ID=devops-agent/webhook,AUTH_MODE=hmac}" \
--timeout 15 --zip-file fileb://function.zip --region us-east-1
If the Lambda runs inside a VPC, ensure it has internet egress (a NAT gateway) to reach the webhook. Outside a VPC, it has access by default.
Give the Lambda permission to read the secret (attach to its execution role):
{
"Effect": "Allow",
"Action": "secretsmanager:GetSecretValue",
"Resource": "arn:aws:secretsmanager:us-east-1:<ACCOUNT_ID>:secret:devops-agent/webhook-*"
}
Let Grafana publish to your SNS topic. If you named the topic  grafana-* , the default policy already allows it. Otherwise, add this to the Grafana workspace role:
{
"Effect": "Allow",
"Action": "sns:Publish",
"Resource": "arn:aws:sns:us-east-1:<ACCOUNT_ID>:<your-topic-name>"
}
Connect the SNS mailbox to the Lambda:
aws sns subscribe --protocol lambda \
--topic-arn arn:aws:sns:us-east-1:<ACCOUNT_ID>:grafana-devops-agent-alerts \
--notification-endpoint arn:aws:lambda:us-east-1:<ACCOUNT_ID>:function:devops-webhook \
--region us-east-1

aws lambda add-permission --function-name devops-webhook \
--statement-id sns-invoke --action lambda:InvokeFunction \
--principal sns.amazonaws.com \
--source-arn arn:aws:sns:us-east-1:<ACCOUNT_ID>:grafana-devops-agent-alerts \
--region us-east-1
Check it worked: Publish a fake alert and confirm the Lambda forwards it:
aws sns publish --topic-arn arn:aws:sns:us-east-1:<ACCOUNT_ID>:grafana-devops-agent-alerts \
--message '{"status":"firing","alerts":[{"labels":{"alertname":"HelloTest","severity":"high","job":"web"},"annotations":{"summary":"Just testing the pipeline"},"startsAt":"2026-01-01T00:00:00Z","fingerprint":"test1"}]}' \
--region us-east-1

aws logs tail /aws/lambda/devops-webhook --region us-east-1 --since 5m --format short
You should see a log line like  Webhook responded 200 ... {"message": "Webhook received"} . A 200 means the agent received it.

Step 5 — Connect Grafana's alerts to SNS

Why: This tells Grafana where to send a notification when an alert fires.
In the Grafana web UI, go to Alerting → Contact points → Add contact point. Choose integration type Amazon SNS and fill in:
  • Topic ARN:  arn:aws:sns:us-east-1:<ACCOUNT_ID>:grafana-devops-agent-alerts 
  • Region:  us-east-1 
  • Auth Provider: AWS SDK Default (leave access key / secret / assume-role empty — it uses the workspace role)
  • Message format: JSON (so the Lambda can parse it)
  • The Amazon SNS API URL: leave blank (defaults to the regional endpoint)
Save the contact point.
Check it worked: Use the Test button on the contact point, then re-run the  aws logs tail ...  command. You should see another  Webhook responded 200 .

Step 6 — Create an alert rule and test end to end

Why: This is the condition that decides when to raise an alarm.
In the Grafana web UI, go to Alerting → Notification policies and route your alerts (e.g. anything labeled  severity=high ) to the Amazon SNS contact point from Step 5.
Then go to Alerting → Alert rules → New alert rule and define your condition. Give it a  severity  label and a  summary  annotation — the  summary  becomes the description the agent reads, so make it useful.
Example: alert when an Amazon Bedrock model responds slower than 8 seconds
  • Data source: CloudWatch
  • Metric: Namespace  AWS/Bedrock , Metric  InvocationLatency , Dimension  ModelId = <your-model-id> , Statistic Maximum, period 60s
  • Condition:  WHEN last() OF A IS ABOVE 8000 
  • Evaluate: every  1m , with a short  for  duration (e.g.,  1m ), routed to the SNS contact point.
Watch the units: this metric is in milliseconds, so "8 seconds" is 8000, not 8.
Check the whole thing worked — follow the trail:
  1. In Grafana, the alert rule shows state Firing.
  1. In the Lambda logs, you see  Webhook responded 200 .
  1. In the DevOps Agent console, a new investigation appears in your Agent Space.

You're done

Your monitoring now automatically summons the DevOps Agent when something breaks:
Grafana alert --> Amazon SNS --> Lambda --> DevOps Agent --> Investigation
^
+-- reads webhook URL + secret from AWS Secrets Manager

Troubleshooting

What you see
Likely cause
Fix
Webhook returns a 4xx
Lambda auth type doesn't match the webhook
Make sure  AUTH_MODE  ( hmac / bearer ) matches how you created the webhook
200 but no investigation
Malformed payload, or duplicate incident ID
Duplicates are ignored — ensure each alert has a unique ID (the Lambda handles this)
200, then investigation cancels instantly
Monthly investigation limit reached
Contact your AWS account team to raise it
Grafana alert never reaches SNS
Grafana lacks  sns:Publish  on the topic
Use a  grafana*  topic name, or add the  sns:Publish  policy
Agent can't read Grafana
Service-account token expired (30-day max)
Create a new token and update the registration

Keep it healthy

  • Rotate the Grafana service-account token before its ≤30-day expiry and update the registration.
  • Keep the webhook secret only in Secrets Manager; if it's ever exposed, rotate the webhook and update the secret's value.
  • Add a dead-letter queue to the SNS → Lambda subscription so no alert is lost if the webhook is briefly unavailable.
  • Write clear  summary  text on alert rules — that's what the agent reads.

Learn more



Have you connected other monitoring tools to the AWS DevOps Agent? Share your setup in the comments.
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article