
Autonomous CloudOps: Building an Event-Driven AI Incident Responder with Amazon Bedrock, CloudWatch & AWS Lambda
"Automate your 2 AM incident triage by combining CloudWatch alarm events with Amazon Bedrock (Claude 3.5 Sonnet) and AWS Lambda. Get AI-generated incident reports and exact CLI remediation steps delivered automatically to your team."
A few weeks ago, I was working on automated alerting setups for my cloud projects. Anyone who has managed workloads on AWS knows the routine: a CloudWatch alarm triggers, and to figure out what actually broke, you end up jumping between tabs, searching through CloudWatch Logs Insights, and reading through raw stack traces.
I wanted to streamline that process. So, I put together a simple serverless pipeline: when an alarm fires, an AWS Lambda function automatically pulls recent error logs, passes the context to Amazon Bedrock, and sends a structured diagnostic report with immediate remediation steps straight to SNS (Slack or Email).
Here is how I built it, along with the full code and configuration so you can deploy it in your own environment.
Architecture Flow:
- Event Trigger: AWS Workloads (EC2, ECS, Lambda, or EKS) push metrics and log streams to CloudWatch.
- Alert Routing: A CloudWatch Alarm enters the ALARM state and triggers an AWS EventBridge rule.
- AI Processing: EventBridge invokes an AWS Lambda function, which fetches recent error logs from CloudWatch Logs and queries Amazon Bedrock for root-cause analysis.
- Alert Delivery: Lambda publishes the generated summary and remediation steps to Amazon SNS, pushing alerts directly to Slack or Email. š ļø Step-by-Step Implementation
Step 1: IAM Role & Least-Privilege Policy
Create an IAM Role for your Lambda function with permissions restricted to CloudWatch logs, Amazon Bedrock model execution, and SNS publishing:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"logs:CreateLogGroup",
"logs:CreateLogStream",
"logs:PutLogEvents",
"logs:FilterLogEvents"
],
"Resource": "arn:aws:logs:::"
},
{
"Effect": "Allow",
"Action": [
"bedrock:InvokeModel"
],
"Resource": "arn:aws:bedrock:::foundation-model/anthropic.claude-3-5-sonnet-20241022-v2:0"
},
{
"Effect": "Allow",
"Action": [
"sns:Publish"
],
"Resource": "arn:aws:sns::*:cloudops-incident-notifications"
}
]
}
Step 2: EventBridge Event Rule Pattern
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"logs:CreateLogGroup",
"logs:CreateLogStream",
"logs:PutLogEvents",
"logs:FilterLogEvents"
],
"Resource": "arn:aws:logs:::"
},
{
"Effect": "Allow",
"Action": [
"bedrock:InvokeModel"
],
"Resource": "arn:aws:bedrock:::foundation-model/anthropic.claude-3-5-sonnet-20241022-v2:0"
},
{
"Effect": "Allow",
"Action": [
"sns:Publish"
],
"Resource": "arn:aws:sns::*:cloudops-incident-notifications"
}
]
}
Step 2: EventBridge Event Rule Pattern
Create an EventBridge Rule to capture alarm state changes and route the payload to Lambda:
{
"source": ["aws.cloudwatch"],
"detail-type": ["CloudWatch Alarm State Change"],
"detail": {
"state": {
"value": ["ALARM"]
}
}
}
Step 3: Complete Lambda Function (
"source": ["aws.cloudwatch"],
"detail-type": ["CloudWatch Alarm State Change"],
"detail": {
"state": {
"value": ["ALARM"]
}
}
}
Step 3: Complete Lambda Function (
python3.12)This function uses the modern
boto3 Bedrock Converse API to perform context-aware incident analysis:import json
import os
import boto3
from botocore.exceptions import ClientError
import os
import boto3
from botocore.exceptions import ClientError
________________________________________
Initialize AWS SDK Clients
bedrock_runtime = boto3.client('bedrock-runtime')
logs_client = boto3.client('logs')
sns_client = boto3.client('sns')
logs_client = boto3.client('logs')
sns_client = boto3.client('sns')
MODEL_ID = os.environ.get('BEDROCK_MODEL_ID', 'anthropic.claude-3-5-sonnet-20241022-v2:0')
SNS_TOPIC_ARN = os.environ.get('SNS_TOPIC_ARN')
SNS_TOPIC_ARN = os.environ.get('SNS_TOPIC_ARN')
def get_recent_error_logs(log_group_name, minutes=5):
"""Fetches recent error log events from CloudWatch."""
try:
response = logs_client.filter_log_events(
logGroupName=log_group_name,
filterPattern='?ERROR ?Exception ?5xx ?Failed',
limit=15
)
events = response.get('events', [])
return "\n".join([e['message'] for e in events]) if events else "No detailed log entries found."
except Exception as e:
return f"Unable to retrieve logs: {str(e)}"
"""Fetches recent error log events from CloudWatch."""
try:
response = logs_client.filter_log_events(
logGroupName=log_group_name,
filterPattern='?ERROR ?Exception ?5xx ?Failed',
limit=15
)
events = response.get('events', [])
return "\n".join([e['message'] for e in events]) if events else "No detailed log entries found."
except Exception as e:
return f"Unable to retrieve logs: {str(e)}"
def lambda_handler(event, context):
try:
# Extract Alarm Details from EventBridge payload
detail = event.get('detail', {})
alarm_name = detail.get('alarmName', 'Unknown Alarm')
reason = detail.get('state', {}).get('reason', 'No reason provided')
try:
# Extract Alarm Details from EventBridge payload
detail = event.get('detail', {})
alarm_name = detail.get('alarmName', 'Unknown Alarm')
reason = detail.get('state', {}).get('reason', 'No reason provided')
________________________________________
š How to Verify and Test
- Trigger a Test Alarm: Manually update a CloudWatch Alarm to the
ALARMstate using the AWS CLI:
aws cloudwatch set-alarm-state
--alarm-name "High-Error-Rate-Backend"
--state-value ALARM
--state-reason "HTTP 500 error threshold exceeded (> 5% in 1 minute)."
--alarm-name "High-Error-Rate-Backend"
--state-value ALARM
--state-reason "HTTP 500 error threshold exceeded (> 5% in 1 minute)."
- Verify Lambda Execution: Check the Lambda log group
/aws/lambda/your-function-nameto ensure the Converse API returned HTTP 200. - Verify Notification: Check your SNS subscriber endpoint (Email or Slack webhook) for the generated RCA.
š” Key Learnings & SRE Takeaways
- Temperature Tuning: Keeping the LLM
temperaturelow ($0.1 - 0.2$) prevents hallucinations when diagnosing technical log strings. - Cost Efficiency: Running this pattern on serverless primitives means zero idle costs. Bedrock model invocation costs fractions of a cent per incident.
- Security First: Ensure your Lambda execution role follows strict principle of least privilege, allowing
bedrock:InvokeModelonly for authorized foundation models.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
# Extract Log Group from environment variable
log_group_name = os.environ.get('TARGET_LOG_GROUP', '/aws/lambda/production-backend')
log_context = get_recent_error_logs(log_group_name)
# Construct System and User Prompts
system_prompts = [{"text": "You are a Senior Cloud Operations & SRE Expert. Analyze cloud infrastructure incident alerts and provide concise, high-impact root cause analysis."}]
user_message = f"""
AN INCIDENT HAS BEEN DETECTED IN THE ENVIRONMENT.
Alarm Name: {alarm_name}
Trigger Reason: {reason}
Recent Log Snippets:
---
{log_context}
---
Please provide a response in the following format:
1. šØ **Root Cause Summary** (Maximum 2 concise sentences)
2. š **Probable Impacted Components**
3. š ļø **Immediate Remediation Steps** (Provide exact AWS CLI commands or AWS Console actions)
"""
messages = [
{
"role": "user",
"content": [{"text": user_message}]
}
]
# Call Bedrock using the Converse API
response = bedrock_runtime.converse(
modelId=MODEL_ID,
messages=messages,
system=system_prompts,
inferenceConfig={"temperature": 0.2, "maxTokens": 600}
)
ai_analysis = response['output']['message']['content'][0]['text']
# Publish structured message to SNS
formatted_notification = (
f"šØ AUTOMATED INCIDENT REPORT: {alarm_name}\n"
f"{'='*45}\n\n"
f"{ai_analysis}\n\n"
f"{'='*45}\n"
f"Automated by AWS Bedrock CloudOps Responder"
)
sns_client.publish(
TopicArn=SNS_TOPIC_ARN,
Subject=f"šØ [AI Triage] {alarm_name}",
Message=formatted_notification
)
return {
'statusCode': 200,
'body': json.dumps({'status': 'Incident processed successfully', 'alarm': alarm_name})
}
except ClientError as e:
print(f"AWS Client Error: {e}")
raise e
except Exception as e:
print(f"Unexpected Error: {e}")
raise eEnjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article