This is Part 3 of the AIOps on AWS series. Read Part 1 here — self-healing infrastructure. Read Part 2 — SNS email alerting.
Disclaimer: This is a learning demo, not production-ready. Always test thoroughly in a dev or pre-prod environment before applying any automated remediation to production workloads.
Email alerts are great. But in a real operations team, nobody is watching their inbox at 2 AM.
They are watching Slack.
Demo 3 (Part 03) adds a formatted Slack notification to the existing self-healing pipeline, so the moment Nginx goes down, your #aiops-alerts channel lights up with everything your team needs to know: what broke, why, what the AI decided, and whether it was fixed.
What Changed
Demos 3 (Part 01) and 4 (Part 02) built the self-healing pipeline with email alerts. Demo 5 adds one parallel output, a Slack message delivered via Incoming Webhook alongside the existing SNS email:
Nginx stops
↓
Lambda → Bedrock analysis
↓ ↓
SNS → Email Slack Webhook
(plain text) (rich Block Kit)
Both run in parallel. If one fails, the other still delivers. Neither can crash the remediation pipeline.
Press enter or click to view image in full size
The Slack Message
Every incident posts a structured Block Kit message to #aiops-alerts with four sections:
Header — severity emoji + one-line issue summary, colored border matching severity.
Metadata — instance ID, timestamp, log group, and Bedrock model used, displayed in a two-column grid.
AI Analysis — full root cause explanation from Claude Sonnet, recommended fix command in code format, and estimated user impact.
Remediation result — the action taken and whether it succeeded, failed, or was skipped due to maintenance mode.
The color coding is immediate and unambiguous:
Severity Border color CRITICAL Red HIGH Orange MEDIUM Yellow LOW Green
Key Implementation Details
No external libraries needed
Slack’s Incoming Webhook is a simple HTTPS POST. Python’s built-in urllib.request handles it — no requests library, no Lambda Layer, no extra dependencies.
data = json.dumps(payload).encode('utf-8')
request = urllib.request.Request(
SLACK_WEBHOOK_URL,
data=data,
headers={'Content-Type': 'application/json'},
method='POST'
)
with urllib.request.urlopen(request, timeout=5) as resp:
print(f"💬 Slack alert sent | Status: {resp.status}")
Silent failure — same principle as SNS
If the Slack webhook is unreachable, the Lambda function logs a warning and moves on. The self-healing logic — Bedrock analysis, SSM remediation — completes regardless.
except Exception as e:
print(f"⚠️ Failed to send Slack alert: {str(e)}")
# Pipeline continues regardless
One environment variable to control it
SLACK_WEBHOOK_URL = https://hooks.slack.com/services/T.../B.../xxx
Remove it and Slack alerts stop silently. No code change needed.
What the Full Pipeline Looks Like Now
EC2 Nginx failure detected
↓
CloudWatch Agent → Log Group
↓
Subscription Filter → Lambda
↓
Pre-check: startup only? → exit (no cost)
Pre-check: maintenance mode? → alert only, no remediation
↓
Amazon Bedrock (Claude Sonnet)
↓
SSM Run Command → Nginx restarted
↓
SNS → Email + Slack → #aiops-alerts
Five AWS services. One Lambda function. Zero manual steps.
Lessons from This Build
Slack Block Kit is worth the extra lines. Plain text webhooks work, but Block Kit messages are scannable in seconds — color, bold labels, code blocks, and a two-column layout mean your on-call engineer sees what matters without reading a wall of text.
Test the webhook URL before wiring it into Lambda. Paste it directly into a curl command first:
curl -X POST -H 'Content-type: application/json' \
--data '{"text":"AIOps webhook test"}' \
YOUR_WEBHOOK_URL
If it returns ok , your URL is valid. Do this before deploying.
Keep the timeout short. The Slack API call uses timeout=5 . Lambda has a 5-minute limit, but a stuck Slack call should never eat into that. Five seconds is generous for a webhook.
Full Source Code
The complete Lambda function — self-healing, email alerts, and Slack notifications — is on GitHub:
Loading article