AWS Builder Center

Integrating incident.io with AWS DevOps Agent

This post walks through an integration that connects incident.ip and DevOps Agent to close the loop between incident declaration and root cause resolution, reducing MTTR without context-switching to a separate console.

Overview

When your team uses incident.io to manage incidents, each incident automatically gets a dedicated Slack channel — a real-time war room where responders collaborate. This integration connects incident.io to AWS DevOps Agent so that when an incident is declared, DevOps Agent automatically starts investigating your AWS environment. When the investigation completes, findings and a mitigation plan are posted directly to that incident's dedicated Slack channel, with no manual routing.
This closes the loop between incident declaration and root cause resolution. Responders stay in their Slack war room and get DevOps Agent findings delivered to them — reducing MTTR without context-switching to a separate console.
No DynamoDB, no polling infrastructure, no external state store needed. The Slack channel ID is carried forward through the DevOps Agent investigation description. Lambda reads it back at completion time and routes directly.



Architecture

incident.io
│ Incident declared → dedicated Slack channel created automatically
│ Workflow fires webhook with incidentId + slackChannelId in body
▼
AWS DevOps Agent (agent space)
│ Investigates: CloudWatch, CloudTrail, ECS, RDS...
│
├── Investigation Completed ──→ Amazon EventBridge
│ │
│ ▼
│ AWS Lambda (~2s)
│ │ posts investigation summary + console link to Slack
│ │ triggers mitigation plan generation
│
├── Mitigation Completed ────→ Amazon EventBridge
│ │
│ ▼
│ AWS Lambda (~2s)
│ │ posts mitigation plan + console link to Slack
│
└───────────────────────────────────── #incident-dedicated-slack-channel
✅ Investigation findings
🔧 Mitigation plan



Prerequisites

  • AWS DevOps Agent agent space with an AWS account association
  • incident.io workspace with workflow permissions
  • Slack app with  chat:write  and  chat:write.public  bot token scopes



Step 1: Configure the DevOps Agent Webhook

DevOps Agent supports two webhook authentication types.

Option A — API Key (simpler)

  1. Agent space → Capabilities → Webhook → Configure → Generate webhook
  1. Select API key
  1. Copy the webhook URL and API key
The incident.io workflow sends:
Authorization: Bearer <your-api-key>

Option B — HMAC (more secure)

  1. Same steps, select HMAC
  1. Copy the webhook URL and signing secret
  1. Place a Lambda in front — incident.io calls the Lambda, Lambda computes the HMAC signature and forwards to DevOps Agent with the correct headers



Step 2: Configure the incident.io Workflow

Create a workflow that fires when an incident is declared and sends a webhook to DevOps Agent.
Trigger: An incident is created or changed Condition: Incident → Status → Category is one of  live ,  triage  Once for: Incident

Webhook step

URL: your DevOps Agent webhook URL Headers:
Content-Type: application/json
Authorization: Bearer <your-api-key>
Body: Type the static parts and use the Insert variable button for each dynamic field:
{"eventType":"incident","incidentId":"[Incident → Incident ID]","action":"created","priority":"HIGH","title":"[Incident → Name]","description":"incidentId=[Incident → Incident ID] slackChannel=[Incident → Slack Channel → ID]","service":"incident.io"}
Important: Use the Insert variable button for each  [...]  badge — do not type them as plain text. Only badges get resolved to actual values when the webhook fires.
When sent, DevOps Agent receives:
TITLE: my ec2 is down
DESCRIPTION: incidentId=01KWDCP9... slackChannel=C0BEEADGB2Q
The  slackChannel  in the description is how Lambda routes findings to the right channel later.



Step 3: Amazon EventBridge Rule

{
"source": ["aws.aidevops"],
"detail-type": [
"Investigation Completed", "Investigation Timed Out", "Investigation Failed",
"Mitigation Completed", "Mitigation Failed", "Mitigation Timed Out"
],
"detail": {
"metadata": {
"agent_space_id": ["<your-agent-space-id>"]
}
}
}
Set the target to your Lambda function.



Step 4: Lambda

The same Lambda handles both event types. It checks  detail-type  first and executes only the relevant path. Note: You can customize lambda as needed.
Investigation event ( handle_investigation ):
  1. Calls  GetBacklogTask  → reads  slackChannel  and  incidentId  from description
  1. Calls  ListJournalRecords  → fetches  investigation_summary_md 
  1. Posts summary + "View in DevOps Agent" button to Slack
  1. Calls  UpdateBacklogTask(PENDING_START)  to kick off mitigation
  1. Posts "Generating mitigation plan..." to Slack
  1. Exits — ~2 seconds total
Mitigation event ( handle_mitigation ):
  1. Calls  GetBacklogTask  → reads  slackChannel  from description (same task)
  1. Calls  ListJournalRecords  → fetches  mitigation_summary_md 
  1. Posts mitigation plan + "View in DevOps Agent" button to Slack
  1. Exits — ~2 seconds total

Lambda environment variables

Variable
Purpose
 AGENT_SPACE_ID 
DevOps Agent agent space ID
 SLACK_BOT_TOKEN 
Slack bot token ( xoxb-... ) with  chat:write  scope
 INCIDENTIO_API_KEY 
Optional — only if you want to call incident.io API for enrichment (e.g. update incident status, fetch severity)

What Lambda needs

  • DevOps Agent via IAM role —  GetBacklogTask ,  ListJournalRecords ,  UpdateBacklogTask 
  • Slack bot token — to call  chat.postMessage 
The incident.io API is not required in the primary flow. The Slack channel ID travels inside the DevOps Agent investigation description — no external state store needed.

What the SRE sees in Slack

✅ Investigation completed: my ec2 is down
[Root cause and findings]
[🔍 View in DevOps Agent]

⚙️ Generating mitigation plan...

🔧 Mitigation plan completed: my ec2 is down
[Step-by-step remediation actions]
[🔍 View in DevOps Agent]
The "View in DevOps Agent" link:  https://<agent-space-id>.aidevops.global.app.aws/investigation/<task-id> 



Practical Use Cases

  • Multiple concurrent incidents — findings routed to the exact right channel automatically
  • On-call engineers stay in the incident Slack channel — no console context-switching
  • Closed loop: declared → investigated → mitigated → all visible in one place
  • MTTR reduction through automated investigation and mitigation surfaced where the team already works
📢 Published by the ES-TAM-Builder community. Follow our  ES-TAM-Builder  Space for more tools, guides, and technical deep-dives from fellow TAMs
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article