Integrating incident.io with AWS DevOps Agent
This post walks through an integration that connects incident.ip and DevOps Agent to close the loop between incident declaration and root cause resolution, reducing MTTR without context-switching to a separate console.
Overview
When your team uses incident.io to manage incidents, each incident automatically gets a dedicated Slack channel — a real-time war room where responders collaborate. This integration connects incident.io to AWS DevOps Agent so that when an incident is declared, DevOps Agent automatically starts investigating your AWS environment. When the investigation completes, findings and a mitigation plan are posted directly to that incident's dedicated Slack channel, with no manual routing.
This closes the loop between incident declaration and root cause resolution. Responders stay in their Slack war room and get DevOps Agent findings delivered to them — reducing MTTR without context-switching to a separate console.
No DynamoDB, no polling infrastructure, no external state store needed. The Slack channel ID is carried forward through the DevOps Agent investigation description. Lambda reads it back at completion time and routes directly.
Architecture
incident.io │ Incident declared → dedicated Slack channel created automatically │ Workflow fires webhook with incidentId + slackChannelId in body ▼AWS DevOps Agent (agent space) │ Investigates: CloudWatch, CloudTrail, ECS, RDS... │ ├── Investigation Completed ──→ Amazon EventBridge │ │ │ ▼ │ AWS Lambda (~2s) │ │ posts investigation summary + console link to Slack │ │ triggers mitigation plan generation │ ├── Mitigation Completed ────→ Amazon EventBridge │ │ │ ▼ │ AWS Lambda (~2s) │ │ posts mitigation plan + console link to Slack │ └───────────────────────────────────── #incident-dedicated-slack-channel ✅ Investigation findings 🔧 Mitigation plan
Prerequisites
- AWS DevOps Agent agent space with an AWS account association
- incident.io workspace with workflow permissions
- Slack app with
chat:writeandchat:write.publicbot token scopes
Step 1: Configure the DevOps Agent Webhook
DevOps Agent supports two webhook authentication types.
Option A — API Key (simpler)
- Agent space → Capabilities → Webhook → Configure → Generate webhook
- Select API key
- Copy the webhook URL and API key
The incident.io workflow sends:
Authorization: Bearer <your-api-key>Option B — HMAC (more secure)
- Same steps, select HMAC
- Copy the webhook URL and signing secret
- Place a Lambda in front — incident.io calls the Lambda, Lambda computes the HMAC signature and forwards to DevOps Agent with the correct headers
Step 2: Configure the incident.io Workflow
Create a workflow that fires when an incident is declared and sends a webhook to DevOps Agent.
Trigger: An incident is created or changed
Condition: Incident → Status → Category is one of
live , triage
Once for: IncidentWebhook step
URL: your DevOps Agent webhook URL
Headers:
Content-Type: application/jsonAuthorization: Bearer <your-api-key>Body: Type the static parts and use the Insert variable button for each dynamic field:
{"eventType":"incident","incidentId":"[Incident → Incident ID]","action":"created","priority":"HIGH","title":"[Incident → Name]","description":"incidentId=[Incident → Incident ID] slackChannel=[Incident → Slack Channel → ID]","service":"incident.io"}Important: Use the Insert variable button for each[...]badge — do not type them as plain text. Only badges get resolved to actual values when the webhook fires.
When sent, DevOps Agent receives:
TITLE: my ec2 is downDESCRIPTION: incidentId=01KWDCP9... slackChannel=C0BEEADGB2QThe
slackChannel in the description is how Lambda routes findings to the right channel later.
Step 3: Amazon EventBridge Rule
{ "source": ["aws.aidevops"], "detail-type": [ "Investigation Completed", "Investigation Timed Out", "Investigation Failed", "Mitigation Completed", "Mitigation Failed", "Mitigation Timed Out" ], "detail": { "metadata": { "agent_space_id": ["<your-agent-space-id>"] } }}Set the target to your Lambda function.
Step 4: Lambda
The same Lambda handles both event types. It checks
detail-type first and executes only the relevant path. Note: You can customize lambda as needed. Investigation event (
handle_investigation ):- Calls
GetBacklogTask→ readsslackChannelandincidentIdfrom description
- Calls
ListJournalRecords→ fetchesinvestigation_summary_md
- Posts summary + "View in DevOps Agent" button to Slack
- Calls
UpdateBacklogTask(PENDING_START)to kick off mitigation
- Posts "Generating mitigation plan..." to Slack
- Exits — ~2 seconds total
Mitigation event (
handle_mitigation ):- Calls
GetBacklogTask→ readsslackChannelfrom description (same task)
- Calls
ListJournalRecords→ fetchesmitigation_summary_md
- Posts mitigation plan + "View in DevOps Agent" button to Slack
- Exits — ~2 seconds total
Lambda environment variables
Variable | Purpose |
|---|---|
AGENT_SPACE_ID | DevOps Agent agent space ID |
SLACK_BOT_TOKEN | Slack bot token ( xoxb-... ) with chat:write scope |
INCIDENTIO_API_KEY | Optional — only if you want to call incident.io API for enrichment (e.g. update incident status, fetch severity) |
What Lambda needs
- DevOps Agent via IAM role —
GetBacklogTask,ListJournalRecords,UpdateBacklogTask
- Slack bot token — to call
chat.postMessage
The incident.io API is not required in the primary flow. The Slack channel ID travels inside the DevOps Agent investigation description — no external state store needed.
What the SRE sees in Slack
✅ Investigation completed: my ec2 is down[Root cause and findings][🔍 View in DevOps Agent]
⚙️ Generating mitigation plan...
🔧 Mitigation plan completed: my ec2 is down[Step-by-step remediation actions][🔍 View in DevOps Agent]The "View in DevOps Agent" link:
https://<agent-space-id>.aidevops.global.app.aws/investigation/<task-id>
Practical Use Cases
- Multiple concurrent incidents — findings routed to the exact right channel automatically
- On-call engineers stay in the incident Slack channel — no console context-switching
- Closed loop: declared → investigated → mitigated → all visible in one place
- MTTR reduction through automated investigation and mitigation surfaced where the team already works
📢 Published by the ES-TAM-Builder community.
Follow our ES-TAM-Builder Space for more tools, guides, and technical deep-dives from fellow TAMs
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article