
Automate incident investigation: connect Amazon Managed Grafana alerts to AWS DevOps Agent
Amazon Managed Grafana can't call a webhook directly, so how do you make a Grafana alert kick off an AWS DevOps Agent investigation automatically? This guide shows the supported AWS SNS + Lambda pattern, end to end, including the cross-account setup.
Monitoring tells you something broke. This guide connects that signal to an AI agent that starts investigating on its own — no human needed to kick it off.
When an alert fires in Amazon Managed Grafana, wouldn't it be great if an investigation just… started? This walkthrough connects Amazon Managed Grafana to the AWS DevOps Agent so that a breaching metric automatically opens an investigation — and the agent can read your Grafana dashboards while it works.
No prior experience with either service is assumed. Every step explains what you're doing, why, and how to check it worked.
Time to complete: ~45–60 minutes • Level: intermediate (AWS Console + a few CLI commands)
What you'll build
By the end, you'll have this automatic flow:
A metric crosses a threshold in Grafana → an alert fires → the AWS DevOps Agent automatically opens an investigation and starts digging in, even pulling data back from your Grafana dashboards.
Think of it as a "call for help" button that dials the AI agent the moment a real problem appears.
How it works (in plain English)
There are two connections between Grafana and the DevOps Agent, and you'll set up both:
- The "trigger" connection — how an alert starts an investigation.
- The "read" connection — how the agent looks at your Grafana dashboards once it's investigating.
Here's the key thing to know about the trigger connection:
Amazon Managed Grafana cannot call a webhook directly. It can, however, send a message to Amazon SNS. So we put a small AWS Lambda function in the middle: SNS receives the alert, Lambda reformats it and forwards it to the DevOps Agent. This is a normal, supported pattern — just one extra hop.
If you use self-managed Grafana, Grafana OSS, or Grafana Cloud (not the Amazon-managed version), those can call a webhook directly and you can skip SNS and Lambda. Everything else here still applies.
Architecture
AWS Secrets Manager (stores webhook URL + secret) | reads | v Grafana alert --> Amazon SNS --> AWS Lambda --> DevOps Agent webhook --> Investigation fires (mailbox) (translator) (the "call for help") starts
...and separately, the agent reads your Grafana dashboards while it investigatesWhy cross-account works with no extra networking: neither connection uses cross-account IAM roles. The trigger path is authenticated by the webhook secret over public HTTPS; the read path is authenticated by a Grafana service-account token. Both endpoints are public HTTPS.
Throughout this guide:
- "Grafana account" = the account where Amazon Managed Grafana lives.
- "Agent account" = the account where the DevOps Agent Space lives.
- Single account? They're the same — just do everything in one place.
A quick glossary
Term | What it means, simply |
|---|---|
Agent Space | Your workspace in the DevOps Agent console where investigations happen. |
Webhook | A special URL. Send it a message and it triggers something — here, an investigation. |
SNS topic | An AWS "mailbox" that receives messages and passes them on. |
Lambda | A small piece of code that runs on demand — here, it reformats and forwards the alert. |
Contact point | In Grafana, where an alert sends its notification (e.g., to SNS). |
Before you begin
Make sure you have:
- An AWS DevOps Agent Space created.
- An Amazon Managed Grafana workspace, Grafana version 9 or later, status Active.
- Permission in the Grafana account to create SNS topics, Lambda functions, IAM policies, and Secrets Manager secrets.
- The ability to log in to the Grafana web UI (to add a service account, contact point, and alert rule).
The setup at a glance
Part A — In the DevOps Agent (Agent account)
- Step 1: Let the agent read your Grafana dashboards
- Step 2: Create the webhook (your "call for help" URL)
Part B — In Amazon Managed Grafana (Grafana account)
- Step 3: Create the SNS mailbox + store the webhook secret safely
- Step 4: Deploy the Lambda translator + give it permissions
- Step 5: Connect Grafana's alerts to SNS
- Step 6: Create an alert rule and test the whole thing
Part A — In the DevOps Agent
Step 1 — Let the agent read your Grafana dashboards
Why: When the agent investigates, it's far more useful if it can look at the same dashboards and metrics you do. This gives it read-only access.
First, in the Grafana web UI, create a read-only login for the agent:
- Go to Administration → Service accounts → Add service account.
- Give it the Viewer role.
- Choose Add service account token and copy the token (starts with
glsa_).
Heads up: Amazon Managed Grafana tokens expire after 30 days maximum. Set a reminder to rotate it — if it expires, the agent quietly loses access.
Then, in the DevOps Agent console, register Grafana:
- Go to Capability Providers → Telemetry → Grafana → Register.
- Service Name: any friendly name, e.g.
my-grafana.
- Grafana URL: e.g.
https://g-xxxxxxxxxx.grafana-workspace.us-east-1.amazonaws.com.
- Service Account Access Token: paste the
glsa_...token.
- Choose Submit.
Finally, attach it to your Agent Space: open the Space → Capabilities → Telemetry → Add → select the registration → Save.
Check it worked: On your Agent Space's Capabilities tab, Grafana now appears under Telemetry.
Step 2 — Create the webhook (your "call for help" URL)
Why: This is the URL that, when sent a message, tells the DevOps Agent to start an investigation.
In the DevOps Agent console, open your Agent Space → Capabilities tab. Find the Agent Space Webhook table and choose Add webhook. Then pick how the webhook is secured:
Option | What it means | When to choose it |
|---|---|---|
HMAC (recommended) | Each message is cryptographically signed. More secure. | Best default; adds tamper-proofing and replay protection. |
API key (bearer token) | A simple secret token sent with each message. | Simpler; fine over trusted HTTPS. |
If you configure the Grafana integration in the console and it auto-creates a webhook, that one uses the API key (bearer) type.
Immediately copy and save the webhook URL and its secret. The secret is shown only once. The URL looks like:
https://event-ai.us-east-1.api.aws/webhook/generic/<WEBHOOK_ID>Lost the secret? Open the webhook → Actions → Edit → Rotate webhook. The URL stays the same; you get a new secret.
Check it worked: You have the webhook URL and secret saved safely — you'll use them in Step 3.
Reference: the webhook payload format
You won't build this by hand (the Lambda does it), but for reference the webhook accepts a JSON message like this:
{ "eventType": "incident", "incidentId": "string", "action": "created|updated|closed|resolved", "priority": "CRITICAL|HIGH|MEDIUM|LOW|MINIMAL", "title": "string", "description": "string (optional)", "timestamp": "ISO-8601 (optional)", "service": "string (optional)", "data": {}}Notes on the fields:
-
incidentIdmust be unique; repeats are ignored.
-
datais accepted, but not read by the agent.
- Only
title,description,priority, and the incident ID reach the agent — so put the useful details indescription. (The Lambda fills this from your Grafana alert'ssummarytext.)
Part B — In Amazon Managed Grafana
For all CLI commands below, replace
<ACCOUNT_ID> with your Grafana account ID and adjust the region if you're not using us-east-1 .Step 3 — Create the SNS mailbox + store the secret safely
Why: SNS is the mailbox Grafana can send to, and Secrets Manager keeps the webhook secret out of your code.
Create the SNS topic. Name it starting with
grafana — Grafana's default permissions only allow it to send to topics named grafana* :aws sns create-topic --name grafana-devops-agent-alerts --region us-east-1Store the webhook URL + secret from Step 2:
aws secretsmanager create-secret \ --name devops-agent/webhook \ --secret-string '{"url":"https://event-ai.us-east-1.api.aws/webhook/generic/<WEBHOOK_ID>","secret":"<WEBHOOK_SECRET>"}' \ --region us-east-1Check it worked: Both commands return an ARN with no errors.
Step 4 — Deploy the Lambda translator + give it permissions
Why: Grafana's alert format and the webhook's expected format differ. This Lambda translates between them, authenticates the message, and forwards it. It supports HMAC or bearer auth via the
AUTH_MODE environment variable.Save this as
lambda_function.py :import base64, hashlib, hmac, json, logging, os, urllib.requestfrom datetime import datetime, timezoneimport boto3
logger = logging.getLogger(); logger.setLevel(logging.INFO)SECRET_ID = os.environ.get("SECRET_ID", "devops-agent/webhook")AUTH_MODE = os.environ.get("AUTH_MODE", "hmac").lower() # "hmac" or "bearer"PRIORITY_MAP = {"critical":"CRITICAL","high":"HIGH","warning":"MEDIUM", "medium":"MEDIUM","low":"LOW","info":"MINIMAL","minimal":"MINIMAL"}_sm = boto3.client("secretsmanager"); _cfg = None
def _get_config(): global _cfg if _cfg is None: _cfg = json.loads(_sm.get_secret_value(SecretId=SECRET_ID)["SecretString"]) return _cfg
def _iso_now(): return datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%S.000Z")
def _build_payload(message): alert, labels, annotations, status = {}, {}, {}, "firing" if isinstance(message, dict): status = message.get("status", "firing") alerts = message.get("alerts") or [] if alerts: alert = alerts[0]; labels = alert.get("labels", {}) or {} annotations = alert.get("annotations", {}) or {} name = labels.get("alertname", "GrafanaAlert") fp = alert.get("fingerprint") or str(abs(hash(json.dumps(labels, sort_keys=True)))) sev = str(labels.get("severity", "high")).lower() desc = annotations.get("summary") or annotations.get("description") or name return { "eventType": "incident", "incidentId": f"{name}-{fp}", "action": "resolved" if status == "resolved" else "created", "priority": PRIORITY_MAP.get(sev, "HIGH"), "title": name, "description": desc, "service": labels.get("job", "amazon-managed-grafana"), "timestamp": alert.get("startsAt") or _iso_now(), "data": {"metadata": {**labels, "_source": "amazon-managed-grafana"}}, }
def _post(url, secret, payload): body = json.dumps(payload); ts = _iso_now() headers = {"Content-Type": "application/json", "x-amzn-event-timestamp": ts} if AUTH_MODE == "bearer": headers["Authorization"] = f"Bearer {secret}" else: sig = base64.b64encode(hmac.new(secret.encode(), f"{ts}:{body}".encode(), hashlib.sha256).digest()).decode() headers["x-amzn-event-signature"] = sig req = urllib.request.Request(url, data=body.encode(), method="POST", headers=headers) with urllib.request.urlopen(req, timeout=10) as resp: return resp.status, resp.read().decode(errors="replace")
def lambda_handler(event, context): cfg = _get_config(); results = [] for record in event.get("Records", []): raw = record["Sns"]["Message"] try: message = json.loads(raw) except (json.JSONDecodeError, TypeError): message = raw payload = _build_payload(message) status, text = _post(cfg["url"], cfg["secret"], payload) logger.info("Webhook responded %s for %s: %s", status, payload["incidentId"], text) results.append({"incidentId": payload["incidentId"], "status": status}) return {"processed": len(results), "results": results}Package and create the function (Python 3.12+). Set
AUTH_MODE to match the webhook type you chose in Step 2 ( hmac or bearer ):zip function.zip lambda_function.pyaws lambda create-function --function-name devops-webhook \ --runtime python3.12 --handler lambda_function.lambda_handler \ --role arn:aws:iam::<ACCOUNT_ID>:role/<lambda-role> \ --environment "Variables={SECRET_ID=devops-agent/webhook,AUTH_MODE=hmac}" \ --timeout 15 --zip-file fileb://function.zip --region us-east-1If the Lambda runs inside a VPC, ensure it has internet egress (a NAT gateway) to reach the webhook. Outside a VPC, it has access by default.
Give the Lambda permission to read the secret (attach to its execution role):
{ "Effect": "Allow", "Action": "secretsmanager:GetSecretValue", "Resource": "arn:aws:secretsmanager:us-east-1:<ACCOUNT_ID>:secret:devops-agent/webhook-*"}Let Grafana publish to your SNS topic. If you named the topic
grafana-* , the default policy already allows it. Otherwise, add this to the Grafana workspace role:{ "Effect": "Allow", "Action": "sns:Publish", "Resource": "arn:aws:sns:us-east-1:<ACCOUNT_ID>:<your-topic-name>"}Connect the SNS mailbox to the Lambda:
aws sns subscribe --protocol lambda \ --topic-arn arn:aws:sns:us-east-1:<ACCOUNT_ID>:grafana-devops-agent-alerts \ --notification-endpoint arn:aws:lambda:us-east-1:<ACCOUNT_ID>:function:devops-webhook \ --region us-east-1
aws lambda add-permission --function-name devops-webhook \ --statement-id sns-invoke --action lambda:InvokeFunction \ --principal sns.amazonaws.com \ --source-arn arn:aws:sns:us-east-1:<ACCOUNT_ID>:grafana-devops-agent-alerts \ --region us-east-1Check it worked: Publish a fake alert and confirm the Lambda forwards it:
aws sns publish --topic-arn arn:aws:sns:us-east-1:<ACCOUNT_ID>:grafana-devops-agent-alerts \ --message '{"status":"firing","alerts":[{"labels":{"alertname":"HelloTest","severity":"high","job":"web"},"annotations":{"summary":"Just testing the pipeline"},"startsAt":"2026-01-01T00:00:00Z","fingerprint":"test1"}]}' \ --region us-east-1
aws logs tail /aws/lambda/devops-webhook --region us-east-1 --since 5m --format shortYou should see a log line like
Webhook responded 200 ... {"message": "Webhook received"} . A 200 means the agent received it.Step 5 — Connect Grafana's alerts to SNS
Why: This tells Grafana where to send a notification when an alert fires.
In the Grafana web UI, go to Alerting → Contact points → Add contact point. Choose integration type Amazon SNS and fill in:
- Topic ARN:
arn:aws:sns:us-east-1:<ACCOUNT_ID>:grafana-devops-agent-alerts
- Region:
us-east-1
- Auth Provider: AWS SDK Default (leave access key / secret / assume-role empty — it uses the workspace role)
- Message format: JSON (so the Lambda can parse it)
- The Amazon SNS API URL: leave blank (defaults to the regional endpoint)
Save the contact point.
Check it worked: Use the Test button on the contact point, then re-run the
aws logs tail ... command. You should see another Webhook responded 200 .Step 6 — Create an alert rule and test end to end
Why: This is the condition that decides when to raise an alarm.
In the Grafana web UI, go to Alerting → Notification policies and route your alerts (e.g. anything labeled
severity=high ) to the Amazon SNS contact point from Step 5.Then go to Alerting → Alert rules → New alert rule and define your condition. Give it a
severity label and a summary annotation — the summary becomes the description the agent reads, so make it useful.Example: alert when an Amazon Bedrock model responds slower than 8 seconds
- Data source: CloudWatch
- Metric: Namespace
AWS/Bedrock, MetricInvocationLatency, DimensionModelId = <your-model-id>, Statistic Maximum, period 60s
- Condition:
WHEN last() OF A IS ABOVE 8000
- Evaluate: every
1m, with a shortforduration (e.g.,1m), routed to the SNS contact point.
Watch the units: this metric is in milliseconds, so "8 seconds" is 8000, not 8.
Check the whole thing worked — follow the trail:
- In Grafana, the alert rule shows state Firing.
- In the Lambda logs, you see
Webhook responded 200.
- In the DevOps Agent console, a new investigation appears in your Agent Space.
You're done
Your monitoring now automatically summons the DevOps Agent when something breaks:
Grafana alert --> Amazon SNS --> Lambda --> DevOps Agent --> Investigation ^ +-- reads webhook URL + secret from AWS Secrets ManagerTroubleshooting
What you see | Likely cause | Fix |
|---|---|---|
Webhook returns a 4xx | Lambda auth type doesn't match the webhook | Make sure AUTH_MODE ( hmac / bearer ) matches how you created the webhook |
200 but no investigation | Malformed payload, or duplicate incident ID | Duplicates are ignored — ensure each alert has a unique ID (the Lambda handles this) |
200, then investigation cancels instantly | Monthly investigation limit reached | Contact your AWS account team to raise it |
Grafana alert never reaches SNS | Grafana lacks sns:Publish on the topic | Use a grafana* topic name, or add the sns:Publish policy |
Agent can't read Grafana | Service-account token expired (30-day max) | Create a new token and update the registration |
Keep it healthy
- Rotate the Grafana service-account token before its ≤30-day expiry and update the registration.
- Keep the webhook secret only in Secrets Manager; if it's ever exposed, rotate the webhook and update the secret's value.
- Add a dead-letter queue to the SNS → Lambda subscription so no alert is lost if the webhook is briefly unavailable.
- Write clear
summarytext on alert rules — that's what the agent reads.
Learn more
Have you connected other monitoring tools to the AWS DevOps Agent? Share your setup in the comments.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article