AWS Builder Center
Autonomous CloudOps: Building an Event-Driven AI Incident Responder with Amazon Bedrock, CloudWatch & AWS Lambda

Autonomous CloudOps: Building an Event-Driven AI Incident Responder with Amazon Bedrock, CloudWatch & AWS Lambda

"Automate your 2 AM incident triage by combining CloudWatch alarm events with Amazon Bedrock (Claude 3.5 Sonnet) and AWS Lambda. Get AI-generated incident reports and exact CLI remediation steps delivered automatically to your team."

IT Student passionate about AWS cloud architecture and DevOps.
A few weeks ago, I was working on automated alerting setups for my cloud projects. Anyone who has managed workloads on AWS knows the routine: a CloudWatch alarm triggers, and to figure out what actually broke, you end up jumping between tabs, searching through CloudWatch Logs Insights, and reading through raw stack traces.
I wanted to streamline that process. So, I put together a simple serverless pipeline: when an alarm fires, an AWS Lambda function automatically pulls recent error logs, passes the context to Amazon Bedrock, and sends a structured diagnostic report with immediate remediation steps straight to SNS (Slack or Email).
Here is how I built it, along with the full code and configuration so you can deploy it in your own environment.
Architecture Flow:
  1. Event Trigger: AWS Workloads (EC2, ECS, Lambda, or EKS) push metrics and log streams to CloudWatch.
  2. Alert Routing: A CloudWatch Alarm enters the ALARM state and triggers an AWS EventBridge rule.
  3. AI Processing: EventBridge invokes an AWS Lambda function, which fetches recent error logs from CloudWatch Logs and queries Amazon Bedrock for root-cause analysis.
  4. Alert Delivery: Lambda publishes the generated summary and remediation steps to Amazon SNS, pushing alerts directly to Slack or Email. šŸ› ļø Step-by-Step Implementation

Step 1: IAM Role & Least-Privilege Policy

Create an IAM Role for your Lambda function with permissions restricted to CloudWatch logs, Amazon Bedrock model execution, and SNS publishing:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"logs:CreateLogGroup",
"logs:CreateLogStream",
"logs:PutLogEvents",
"logs:FilterLogEvents"
],
"Resource": "arn:aws:logs:::"
},
{
"Effect": "Allow",
"Action": [
"bedrock:InvokeModel"
],
"Resource": "arn:aws:bedrock:
::foundation-model/anthropic.claude-3-5-sonnet-20241022-v2:0"
},
{
"Effect": "Allow",
"Action": [
"sns:Publish"
],
"Resource": "arn:aws:sns:
:*:cloudops-incident-notifications"
}
]
}

Step 2: EventBridge Event Rule Pattern
Create an EventBridge Rule to capture alarm state changes and route the payload to Lambda:
{
"source": ["aws.cloudwatch"],
"detail-type": ["CloudWatch Alarm State Change"],
"detail": {
"state": {
"value": ["ALARM"]
}
}
}

Step 3: Complete Lambda Function (python3.12)
This function uses the modern boto3 Bedrock Converse API to perform context-aware incident analysis:
import json
import os
import boto3
from botocore.exceptions import ClientError
________________________________________
Initialize AWS SDK Clients
bedrock_runtime = boto3.client('bedrock-runtime')
logs_client = boto3.client('logs')
sns_client = boto3.client('sns')
MODEL_ID = os.environ.get('BEDROCK_MODEL_ID', 'anthropic.claude-3-5-sonnet-20241022-v2:0')
SNS_TOPIC_ARN = os.environ.get('SNS_TOPIC_ARN')
def get_recent_error_logs(log_group_name, minutes=5):
"""Fetches recent error log events from CloudWatch."""
try:
response = logs_client.filter_log_events(
logGroupName=log_group_name,
filterPattern='?ERROR ?Exception ?5xx ?Failed',
limit=15
)
events = response.get('events', [])
return "\n".join([e['message'] for e in events]) if events else "No detailed log entries found."
except Exception as e:
return f"Unable to retrieve logs: {str(e)}"
def lambda_handler(event, context):
try:
# Extract Alarm Details from EventBridge payload
detail = event.get('detail', {})
alarm_name = detail.get('alarmName', 'Unknown Alarm')
reason = detail.get('state', {}).get('reason', 'No reason provided')
________________________________________
šŸ” How to Verify and Test
  1. Trigger a Test Alarm: Manually update a CloudWatch Alarm to the ALARM state using the AWS CLI:
aws cloudwatch set-alarm-state
--alarm-name "High-Error-Rate-Backend"
--state-value ALARM
--state-reason "HTTP 500 error threshold exceeded (> 5% in 1 minute)."
  • Verify Lambda Execution: Check the Lambda log group /aws/lambda/your-function-name to ensure the Converse API returned HTTP 200.
  • Verify Notification: Check your SNS subscriber endpoint (Email or Slack webhook) for the generated RCA.
šŸ’” Key Learnings & SRE Takeaways
  • Temperature Tuning: Keeping the LLM temperature low ($0.1 - 0.2$) prevents hallucinations when diagnosing technical log strings.
  • Cost Efficiency: Running this pattern on serverless primitives means zero idle costs. Bedrock model invocation costs fractions of a cent per incident.
  • Security First: Ensure your Lambda execution role follows strict principle of least privilege, allowing bedrock:InvokeModel only for authorized foundation models.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
# Extract Log Group from environment variable
log_group_name = os.environ.get('TARGET_LOG_GROUP', '/aws/lambda/production-backend')
log_context = get_recent_error_logs(log_group_name)

# Construct System and User Prompts
system_prompts = [{"text": "You are a Senior Cloud Operations & SRE Expert. Analyze cloud infrastructure incident alerts and provide concise, high-impact root cause analysis."}]

user_message = f"""
AN INCIDENT HAS BEEN DETECTED IN THE ENVIRONMENT.

Alarm Name: {alarm_name}
Trigger Reason: {reason}

Recent Log Snippets:
---
{log_context}
---

Please provide a response in the following format:
1. 🚨 **Root Cause Summary** (Maximum 2 concise sentences)
2. šŸ” **Probable Impacted Components**
3. šŸ› ļø **Immediate Remediation Steps** (Provide exact AWS CLI commands or AWS Console actions)
"""


messages = [
{
"role": "user",
"content": [{"text": user_message}]
}
]

# Call Bedrock using the Converse API
response = bedrock_runtime.converse(
modelId=MODEL_ID,
messages=messages,
system=system_prompts,
inferenceConfig={"temperature": 0.2, "maxTokens": 600}
)

ai_analysis = response['output']['message']['content'][0]['text']

# Publish structured message to SNS
formatted_notification = (
f"🚨 AUTOMATED INCIDENT REPORT: {alarm_name}\n"
f"{'='*45}\n\n"
f"{ai_analysis}\n\n"
f"{'='*45}\n"
f"Automated by AWS Bedrock CloudOps Responder"
)

sns_client.publish(
TopicArn=SNS_TOPIC_ARN,
Subject=f"🚨 [AI Triage] {alarm_name}",
Message=formatted_notification
)

return {
'statusCode': 200,
'body': json.dumps({'status': 'Incident processed successfully', 'alarm': alarm_name})
}

except ClientError as e:
print(f"AWS Client Error: {e}")
raise e
except Exception as e:
print(f"Unexpected Error: {e}")
raise e
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article