AWS Builder Center
Real-Time Slack Incident Alerts for Self-Healing Infrastructure on AWS — Part03

Real-Time Slack Incident Alerts for Self-Healing Infrastructure on AWS — Part03

Demo 3 (Part 03) adds a formatted Slack notification to the existing self-healing pipeline, so the moment Nginx goes down, your #aiops-alerts channel lights up with everything your team needs to know: what broke, why, what the AI decided, and whether it was fixed.

Series: AWS-Bedrock (4 articles)

  1. …
  2. 3
    Real-Time Slack Incident Alerts for Self-Healing Infrastructure on AWS — Part03 This article
Head of DevOps
This is Part 3 of the AIOps on AWS series. Read Part 1 here — self-healing infrastructure. Read Part 2 — SNS email alerting.
Disclaimer: This is a learning demo, not production-ready. Always test thoroughly in a dev or pre-prod environment before applying any automated remediation to production workloads.
Email alerts are great. But in a real operations team, nobody is watching their inbox at 2 AM.
They are watching Slack.
Demo 3 (Part 03) adds a formatted Slack notification to the existing self-healing pipeline, so the moment Nginx goes down, your  #aiops-alerts  channel lights up with everything your team needs to know: what broke, why, what the AI decided, and whether it was fixed.

What Changed

Demos 3 (Part 01) and 4 (Part 02) built the self-healing pipeline with email alerts. Demo 5 adds one parallel output, a Slack message delivered via Incoming Webhook alongside the existing SNS email:
Nginx stops
↓
Lambda → Bedrock analysis
↓ ↓
SNS → Email Slack Webhook
(plain text) (rich Block Kit)
Both run in parallel. If one fails, the other still delivers. Neither can crash the remediation pipeline.
Press enter or click to view image in full size

The Slack Message

Every incident posts a structured Block Kit message to  #aiops-alerts  with four sections:

Header — severity emoji + one-line issue summary, colored border matching severity.
Metadata — instance ID, timestamp, log group, and Bedrock model used, displayed in a two-column grid.
AI Analysis — full root cause explanation from Claude Sonnet, recommended fix command in code format, and estimated user impact.
Remediation result — the action taken and whether it succeeded, failed, or was skipped due to maintenance mode.
The color coding is immediate and unambiguous:
Severity Border color CRITICAL Red HIGH Orange MEDIUM Yellow LOW Green

Key Implementation Details

No external libraries needed
Slack’s Incoming Webhook is a simple HTTPS POST. Python’s built-in  urllib.request  handles it — no  requests  library, no Lambda Layer, no extra dependencies.

data = json.dumps(payload).encode('utf-8')
request = urllib.request.Request(
SLACK_WEBHOOK_URL,
data=data,
headers={'Content-Type': 'application/json'},
method='POST'
)
with urllib.request.urlopen(request, timeout=5) as resp:
print(f"💬 Slack alert sent | Status: {resp.status}")
Silent failure — same principle as SNS
If the Slack webhook is unreachable, the Lambda function logs a warning and moves on. The self-healing logic — Bedrock analysis, SSM remediation — completes regardless.
except Exception as e:
print(f"⚠️ Failed to send Slack alert: {str(e)}")
# Pipeline continues regardless
One environment variable to control it
SLACK_WEBHOOK_URL = https://hooks.slack.com/services/T.../B.../xxx
Remove it and Slack alerts stop silently. No code change needed.

What the Full Pipeline Looks Like Now

EC2 Nginx failure detected
↓
CloudWatch Agent → Log Group
↓
Subscription Filter → Lambda
↓
Pre-check: startup only? → exit (no cost)
Pre-check: maintenance mode? → alert only, no remediation
↓
Amazon Bedrock (Claude Sonnet)
↓
SSM Run Command → Nginx restarted
↓
SNS → Email + Slack → #aiops-alerts
Five AWS services. One Lambda function. Zero manual steps.

Lessons from This Build

Slack Block Kit is worth the extra lines. Plain text webhooks work, but Block Kit messages are scannable in seconds — color, bold labels, code blocks, and a two-column layout mean your on-call engineer sees what matters without reading a wall of text.
Test the webhook URL before wiring it into Lambda. Paste it directly into a curl command first:
curl -X POST -H 'Content-type: application/json' \
--data '{"text":"AIOps webhook test"}' \
YOUR_WEBHOOK_URL
If it returns  ok , your URL is valid. Do this before deploying.
Keep the timeout short. The Slack API call uses  timeout=5 . Lambda has a 5-minute limit, but a stuck Slack call should never eat into that. Five seconds is generous for a webhook.

Full Source Code

The complete Lambda function — self-healing, email alerts, and Slack notifications — is on GitHub:

Series: AWS-Bedrock (4 articles)

  1. …
  2. 3
    Real-Time Slack Incident Alerts for Self-Healing Infrastructure on AWS — Part03 This article
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article