AWS Builder Center

Builder Center

From Alert Noise to Action: Using Amazon Bedrock for Incident Triage on AWS

From Alert Noise to Action: Using Amazon Bedrock for Incident Triage on AWS

A hands-on guide to building an AI-assisted incident triage pipeline on AWS. Use EventBridge for routing, Lambda for enrichment, and Amazon Bedrock for intelligent summarization.

Serhii Yenhalychev

DevOps engineer

The 2 AM context problem

It's 2 AM. Your phone buzzes with a PagerDuty alert:  API-CHECKOUT-5XX-HIGH . You're awake, but you still don't know: Is this affecting customers? Which deployment broke it? What should you check first? You'll spend the next 10 minutes gathering context that should have arrived with the alert.
Modern alerting systems are excellent at detecting signals, but much less effective at providing immediate context. An alarm may tell you that something is wrong, yet still leave the responder to figure out which service is affected, whether the issue is urgent, and what to investigate first.
I wanted to reduce that gap by building a lightweight AI-assisted incident triage flow on AWS. In this article, I'll show how I used Amazon EventBridge, AWS Lambda, and Amazon Bedrock to convert raw alert events into short, structured summaries delivered to Slack for faster first response.


The goal

I needed a system that would:
  • Accept incoming alert events from multiple sources
  • Enrich them with environment and service context
  • Ask Bedrock for a concise triage summary
  • Send the result to Slack
  • Keep a human in the loop
One critical boundary: This is not an auto-remediation pipeline. It's a triage accelerator.
I wanted to compress noisy signals into clearer next steps for human responders, not replace operational judgment.


Architecture at a glance

The flow is deliberately simple:
A lightweight AWS incident triage flow using EventBridge, Lambda, Amazon Bedrock, and Slack



Routing alerts with EventBridge

EventBridge decouples event sources from processing logic. I can add new alert sources later without touching the Lambda function. More importantly, I can filter events before they reach Lambda, which reduces cost and noise.
I started with CloudWatch alarms in  ALARM  state:
JSON
{
  "source": ["aws.cloudwatch"],
  "detail-type": ["CloudWatch Alarm State Change"],
  "detail": {
    "state": {
      "value": ["ALARM"]
    }
  }
}
This pattern ensures that only actionable state changes trigger the Lambda function. Routine  OK  transitions and informational events don't create noise.
With the events now filtered, the Lambda function needed to add context before calling Bedrock.


Enriching events before calling Bedrock

The model is only useful when the prompt contains structured context. Raw CloudWatch events are technically complete but operationally sparse. They tell you what alarmed, but not why it matters or what changed recently.
I enriched each event with:
  • Environment name (prod, staging, dev)
  • Service name and ownership team
  • AWS account and region
  • Recent deployment marker (from tags or external API)
  • Severity mapping (critical, high, medium, low)
This metadata transforms a generic alarm into a situated incident.
Here's the Lambda skeleton:
Python
import json
import os
import boto3
import urllib.request

bedrock = boto3.client("bedrock-runtime", region_name=os.environ["AWS_REGION"])

SLACK_WEBHOOK_URL = os.environ["SLACK_WEBHOOK_URL"]
MODEL_ID = "anthropic.claude-3-sonnet-20240229-v1:0"

def build_prompt(event, enriched):
    return f"""You are assisting with incident triage.

Use only the information provided below.
Do not invent causes, metrics, or remediation steps.
If the signal is ambiguous, state "More investigation needed."
Keep the response under 120 words.

Return a short response with exactly these sections:
1. Summary
2. Likely impact
3. First checks

Event:
{json.dumps(event, indent=2)}

Enriched context:
{json.dumps(enriched, indent=2)}
""".strip()

def invoke_bedrock(prompt):
    body = {
        "anthropic_version": "bedrock-2023-05-31",
        "max_tokens": 400,
        "temperature": 0.2,
        "messages": [
            {"role": "user", "content": prompt}
        ]
    }
    
    response = bedrock.invoke_model(
        modelId=MODEL_ID,
        body=json.dumps(body)
    )
    
    response_body = json.loads(response['body'].read())
    return response_body['content'][0]['text']

def post_to_slack(message):
    payload = json.dumps({"text": message}).encode("utf-8")
    req = urllib.request.Request(
        SLACK_WEBHOOK_URL,
        data=payload,
        headers={"Content-Type": "application/json"}
    )
    urllib.request.urlopen(req)

def lambda_handler(event, context):
    # Enrich with metadata from tags, environment variables, or external API
    enriched = {
        "environment": "prod",
        "service": "checkout-api",
        "owner_team": "platform",
        "region": event.get("region"),
        "account": event.get("account"),
        "recent_deployment": "v2.4.1 deployed 8 minutes ago by platform-team"
    }

    prompt = build_prompt(event, enriched)
    summary = invoke_bedrock(prompt)
    post_to_slack(summary)

    return {"statusCode": 200, "body": "ok"}


Prompt design that made the output useful

The biggest improvement didn't come from adding more raw data. It came from constraining the model output.
Early versions of my prompt were too open-ended. I'd ask Bedrock to "analyze this incident," and I'd get verbose, speculative responses that mentioned possible causes the model couldn't verify. They weren't wrong, but they weren't useful in the first 60 seconds of an incident.
I refined the prompt using three principles:

1. Constraint over creativity

Bad prompt:
Plain text
Analyze this incident and provide recommendations.
Good prompt:
Plain text
Use only the provided data. Return exactly three sections: Summary, 
Likely Impact, First Checks. Maximum 120 words.
The model follows structure better when you're explicit about boundaries.

2. Explicit boundaries

I always include these instructions:
Plain text
Do not invent causes. 
Do not assume missing information. 
If unclear, state "More investigation needed."
This prevents hallucination. If the event doesn't contain deployment data, the model should say "check recent deployments" rather than inventing a deployment timeline.

3. Low temperature for consistency

I set  temperature: 0.2  to reduce variability. Incident triage is not a creative task. I wanted the same event to produce nearly identical summaries on repeated invocations.
These constraints transformed Bedrock from a general-purpose assistant into a specialized triage tool.


Before and after: what the output looks like

The value is clearest when you see the transformation.

Raw CloudWatch Alert

Plain text
AlarmName: api-checkout-5xx-high
State: ALARM
Threshold: 50
Value: 127
Namespace: AWS/ApplicationELB
This is accurate, but it doesn't tell the on-call engineer what to do.

AI-Enriched Triage (delivered to Slack)

Plain text
🚨 Incident Triage Summary

Service: checkout-api | Environment: prod | Severity: high

Summary:
5XX errors spiked to 127/min (threshold: 50). Started 3 minutes after 
deployment v2.4.1 to checkout-api.

Likely Impact:
Payment processing failures. Estimated 15-20% of checkout attempts failing.

First Checks:
1. Rollback deployment v2.4.1 (deployed by @platform-team 8 min ago)
2. Check application logs for new exceptions
3. Verify database connection pool exhaustion

[View CloudWatch Dashboard] [View Deployment]
The enriched summary includes deployment context, impact estimation, and prioritized next steps. An engineer reading this can act in seconds rather than minutes.


What could go wrong (and how I handled it)

No system survives first contact with production unchanged. Here's what broke and how I fixed it:

Problem: Bedrock throttling during alert storms

During a cascading failure, dozens of alarms fire simultaneously. Bedrock's default rate limits couldn't handle the burst.
Solution: I added exponential backoff with jitter and a dead-letter queue (DLQ) for failed invocations. Non-critical alerts could retry later. Critical alerts got priority routing.

Problem: Model hallucinating causes when event context was sparse

If an alarm lacked deployment data or service tags, the model would sometimes invent plausible-sounding explanations.
Solution: I made the prompt constraints stricter and added a "confidence" field to the output. If the model detected insufficient context, it would return:
Plain text
Confidence: Low
Summary: 5XX errors detected, but no recent deployment or service metadata available.
First Checks: Gather deployment history and service ownership before investigating further.
This prevented false certainty.

Problem: Duplicate alerts creating noise in Slack

Some alarms transition to  ALARM  state multiple times during a single incident.
Solution: I added EventBridge content-based deduplication and Lambda idempotency with DynamoDB. Each incident gets a hash based on service, alarm name, and time window. Duplicate hashes within 15 minutes are suppressed.


Security guardrails I would not skip

AI-assisted workflows introduce new risks. Here's what I consider non-negotiable:
Never log full prompts Prompts may contain sensitive event data — IP addresses, internal service names, or error messages with user identifiers.
Python
logger.info("Bedrock invoked", extra={
    "event_type": event.get("detail-type"),
    "service": enriched.get("service"),
    "model_id": MODEL_ID
})
IAM least privilege. The Lambda execution role should only invoke specific Bedrock models:
JSON
{
"Effect": "Allow",
"Action": "bedrock:InvokeModel",
"Resource": "arn:aws:bedrock:::foundation-model/anthropic.claude-"
}
Secrets Manager for Slack webhook Never hardcode the webhook URL. Store it in AWS Secrets Manager and retrieve it at runtime.
VPC endpoints for Bedrock (if handling sensitive data) If your events contain regulated or sensitive information, use VPC endpoints to keep traffic off the public internet.
Human approval gate This system summarizes and suggests. It never executes remediation automatically. Human responders remain the decision-makers.


What worked in practice

After running this in production for three months, here's what I learned:
Worked well
Needed adjustment
Structured prompts
Large raw payloads reduced clarity
Small, enriched context
Duplicate alerts created noise initially
Low temperature (0.2)
Some alerts still lacked enough metadata
Slack-friendly formatting
Severity mapping needed per-service tuning
Deployment timestamp enrichment
API latency spikes required timeout tuning
The most valuable insight: Bedrock worked best as a context compressor, not as an incident decision-maker.
It excelled at taking 50 lines of JSON and producing 4 lines of actionable guidance. It struggled when asked to diagnose root cause or predict incident duration.

Lessons learned

1. Concise metadata mattered more than huge payloads

I initially passed entire CloudWatch event payloads to Bedrock. The responses became verbose and unfocused. When I reduced the prompt to just the essential fields (alarm name, metric value, threshold, recent deployment), the summaries improved immediately.

2. Event filtering before Lambda reduced cost and noise

Early versions processed every CloudWatch event. EventBridge filtering cut Lambda invocations by 60% and eliminated alerts that didn't need triage.

3. Response structure made Slack alerts more usable

Freeform summaries were hard to scan. Enforcing the three-section format (Summary, Impact, First Checks) made alerts skimmable at a glance.

4. Human-in-the-loop remains essential

I never trusted the AI output enough to trigger automated remediation. The system's value came from accelerating human judgment, not replacing it.

What I'd tell another builder

Start small. Pick your noisiest alert source, add a Lambda function, and let Bedrock compress the context. You don't need perfect enrichment on day one — even basic summaries make alerts more actionable.
The code in this article is enough to deploy a working prototype in an afternoon. If you're drowning in alert noise, this pattern might be the fastest path to breathing room.
I found that Bedrock added the most value not by replacing operational judgment, but by helping compress noisy signals into clearer next steps for human responders.
Try it: Start with the EventBridge pattern and Lambda skeleton above. Add your own enrichment logic based on what context your team actually needs. Deploy to a non-critical environment first, tune the prompt, and iterate based on real feedback from your on-call rotation.
The gap between "alert fired" and "engineer knows what to check" is where incidents get expensive. This pattern closes that gap.
# amazon-bedrock# aws-lambda# aws-cloudwatch# amazon-eventbridge# amazon-dynamodb
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article