AWS Builder Center
AI-Powered Incident Alerting on AWS — Adding SNS Email Notifications to Self-Healing Infrastructure - Part02

AI-Powered Incident Alerting on AWS — Adding SNS Email Notifications to Self-Healing Infrastructure - Part02

Part 2 adds exactly that: an AI-generated incident alert delivered to your inbox the moment something goes wrong.

Series: AWS-Bedrock (4 articles)

  1. 1
    AI-Powered Incident Alerting on AWS — Adding SNS Email Notifications to Self-Healing Infrastructure - Part02 This article
Head of DevOps
This is Part 2 of the AIOps on AWS series. Read Part 1, where we built a self-healing Nginx pipeline using EC2, CloudWatch, Lambda, Bedrock, and SSM.
In Part 1, our infrastructure could detect a failure, analyze it with AI, and fix it automatically. But there was one gap: nobody knew it happened.
In production, silent automation is a risk. You need a record. You need to know what broke, what the AI decided, and whether the fix worked, even if you were asleep when it happened.
Part 2 adds exactly that: an AI-generated incident alert delivered to your inbox the moment something goes wrong.

What We Added

The self-healing pipeline from Part 1 remains unchanged. We layered one new capability on top:
[Existing] EC2 → CloudWatch → Lambda → Bedrock → SSM (auto-fix)
↓
[NEW] Amazon SNS
↓
Formatted email alert
Every time Bedrock completes an analysis, whether the service was auto-remediated, in maintenance mode, or healthy. Lambda now publishes a structured alert to an SNS topic, which delivers it to your email instantly.

Three Alert Scenarios

Scenario 1 — Auto-remediated (most common)
Nginx stops unexpectedly. Bedrock detects it, SSM restarts it. You get an email:
🔴 AIOps CRITICAL: Nginx has stopped after receiving SIGQUIT

INCIDENT SUMMARY
Severity : CRITICAL
Instance ID : i-0dcf96048cd5cfebf
Timestamp : 2026-07-04 15:41:00 UTC

AI ANALYSIS
Root Cause : Nginx received SIGQUIT from systemd, triggering graceful shutdown
Fix : sudo systemctl start nginx

REMEDIATION
Action : nginx_service_stopped
✅ Auto-remediated — service restarted successfully
Exit Code : 0
Press enter or click to view image in full size
CloudWatch Logs
Scenario 2 — Maintenance mode
You set  MAINTENANCE_MODE_SERVICES = nginx  before planned maintenance. Nginx stops, Lambda detects it but skips auto-remediation. You still get notified:
🟢 AIOps LOW: Nginx stopped during planned maintenance
🔶 Maintenance mode — auto-remediation skipped
Scenario 3 — Remediation failed
Something unexpected prevents SSM from running. You get an immediate alert so you can intervene manually:
🔴 AIOps CRITICAL: Nginx has stopped
❌ Remediation FAILED — manual intervention required

Key Implementation Details

Silent failure design
The SNS alert is wrapped in a try/except that never raises. If SNS is misconfigured or unavailable, the alert is skipped, but the remediation pipeline continues unaffected. Alerting is critical, but it must never crash the system that’s doing the actual fixing.
try:
sns.publish(TopicArn=SNS_TOPIC_ARN, Subject=subject, Message=body)
print(f"📧 Alert sent | MessageId: {response['MessageId']}")
except Exception as e:
print(f"⚠️ Failed to send SNS alert: {str(e)}")
# Pipeline continues regardless
Severity-based email subjects
Every alert subject is prefixed with a severity emoji so your inbox filters can act on it:
🔴 CRITICAL → service is down, action required
🟠 HIGH → major degradation
🟡 MEDIUM → partial failure
🟢 LOW → informational (maintenance, healthy)
Press enter or click to view image in full size
  • Received Email Notification
Press enter or click to view image in full size
  • Complete Email Notification Details
One environment variable to configure
The entire alerting feature is controlled by a single Lambda environment variable:
SNS_TOPIC_ARN = arn:aws:sns:eu-central-1:xxxx:aiops-incident-alerts
Nginx stops
↓
CloudWatch Agent ships logs
↓
Subscription Filter triggers Lambda
↓
Pre-check: startup only? → exit early
Pre-check: maintenance mode? → alert + exit
↓
Bedrock (Claude Sonnet) — AI analysis
↓
SSM Run Command — auto-remediation
↓
SNS → Email alert (severity + root cause + fix status)
Every stage is logged. Every incident is traceable.

Lessons from This Build

Confirm your SNS subscription. AWS sends a confirmation email when you subscribe. If you miss that click, no alerts will ever arrive, silently. Always verify in the SNS console that the subscription status shows Confirmed.
IAM permissions are the most common silent failure. If Lambda cannot publish to SNS, it simply logs a warning and moves on. Always check that  SNS:Publish  is explicitly allowed on the topic ARN in the Lambda execution role.
Test the failure case, not just the happy path. Most people test that alerts arrive when Nginx stops. Test that the pipeline still heals the service even if SNS is broken. Resilience means the core function works regardless of what the notification layer does.
Disclaimer: This solution is intended for learning and demonstration purposes only. Before using it in any real environment, thoroughly test it in a dev or pre-production setup and ensure it meets your organization’s security, compliance, and operational standards.

Full Source Code

The complete Lambda function for Demo 3, including SNS alerting, maintenance mode, startup pre-check, and SSM remediation — is available on GitHub:

Series: AWS-Bedrock (4 articles)

  1. 1
    AI-Powered Incident Alerting on AWS — Adding SNS Email Notifications to Self-Healing Infrastructure - Part02 This article
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article