
Building an AI SRE That Investigates Its Own Incidents — on AWS Bedrock AgentCore
Incident Response Copilot: an always-on agent that diagnoses production incidents autonomously in 30 seconds, remembers every one it has ever seen, and can safely apply a fix behind two independent gates — with real code and the bugs that only showed up in production.
THE PROBLEM
When something breaks in production, whoever is on call does the same boring
first steps every time:
- Open CloudWatch to see which number went bad.
- Search the logs for the real error.
- Check if someone changed a setting recently.
- Look at what was deployed just before it broke.
- Try to remember: have we seen this before? What fixed it last time?
This is SLOW (it eats the first 15–45 minutes of every incident), it's
REPETITIVE (the steps barely change), and it's FORGETFUL (the "what fixed it
last time" knowledge leaves when a teammate leaves).
So I asked a simple question: can an agent do this first pass for me, and just
hand me the answer?
That's the Incident Response Copilot, built on Amazon Bedrock AgentCore. It
doesn't replace the engineer — it does the boring first 30 minutes and never
forgets anything.
WHAT IT GIVES YOU
When an alarm fires, the agent investigates on its own and emails a report like
this. Nobody triggered it, and nobody wrote it:
[SEV2] checkout-api-p99-latency — Increased worker concurrency 10->40 likely caused it
Alarm: checkout-api-p99-latency
Severity: high
Confidence: medium
Loop in: priya-dev
LIKELY ROOT CAUSE
Increased worker concurrency from 10 to 40 likely caused the p99 latency increase.
TIMELINE
11:17 — Deployment increased worker concurrency from 10 to 40.
11:24 — Alarm breached.
SEEN BEFORE
This alarm has been diagnosed before: 3 times on record (recurrence count 3).
Fix recorded last time: Review and adjust worker concurrency settings.
RECOMMENDED ACTION
Review and adjust worker concurrency settings.
That's a lot more than a raw alarm: the likely cause, a timeline linking a
deploy to the breach, a confidence level, who to loop in, and — the best part —
a memory of past times and the fix that worked.
THE AGENT'S INSTRUCTIONS
The agent runs on its own. Nobody is there to answer its questions. That one
fact shapes everything, because a normal chatbot loves to do two things you
can't allow here: ask for permission, and give up when something fails.
So the system prompt is blunt about it:
CRITICAL OPERATING CONSTRAINTS — you run UNATTENDED:
- There is NO human available. Never ask a question, never wait for input.
- If a tool returns an error or nothing, that is normal. Record it as a gap
and CONTINUE. A missing metric or log NEVER ends the investigation.
- You MUST always finish by sending a report, even at low confidence.
An honest "here's what I checked, cause unclear" is useful. Silence is not.
- You are bounded: at most 20 tool calls and 300 seconds. When the budget is
spent, stop gathering and report with what you have.Why this matters: early on, the model would stop halfway and ask "Should I keep
going?" — to an empty room. The "you run UNATTENDED" rule fixed that.
After the rules, the prompt gives it a simple 7-step method: check memory ->
understand the alarm -> gather evidence -> analyse the numbers -> form a theory
-> report -> remember.
THE TOOLS, AND THE ONE RULE THEY ALL FOLLOW
The agent can't see anything by itself. It uses six read-only tools:
- describe_alarm — what broke
- get_metric_data — the shape of the problem (the graph)
- search_logs — the actual error text
- get_recent_config_changes — recent infrastructure changes
- get_recent_deployments — what was deployed just before
- get_service_health — is the service starved or crash-looping?
Every tool follows one rule: it never crashes — it returns an error instead. If
one tool blew up the whole run, a single missing log group would ruin an
investigation that could have succeeded on the other five signals. So a broken
tool just becomes a "gap" the agent works around:
@tool
def get_metric_data(namespace, metric_name, ...):
try:
response = retry_aws("cloudwatch:GetMetricData", cw.get_metric_data, ...)
return {"timestamps": [...], "values": [...], "summary": {...}}
except Exception as exc: # fail soft — hand the agent an error, don't crash
return failure_fields("get_metric_data", exc)That one line — return an error instead of crashing — is what keeps the whole
thing robust. (It's also why a nasty bug hid for a long time.
THE PART THAT MAKES IT LEARN: MEMORY
Here's the difference between this and a normal alarm tool. A normal tool treats
the 1st and the 50th time a problem happens exactly the same — it has no memory.
A team has memory, but it's fragile and leaves with people.
The Copilot saves every incident to AgentCore Memory, and the first thing it
does on any new alarm is ask, "have I seen this before?"
@tool
def recall_similar_incidents(alarm_name, symptom_summary="", top_k=3):
"""ALWAYS call this first."""
if MEMORY_ID:
matches = _agentcore_recall(query, top_k) # deployed: AgentCore Memory
else:
matches = _local_similarity_search(query) # local dev: a JSON file
return _publish_recall(alarm_name, {
"match_count": len(matches),
"past_incidents": matches,
"guidance": "A past incident matched — try its recorded fix first."
if matches else "First time seeing this. Investigate fresh.",
})Memory is per team (one team never sees another's history) and writes are safe
to retry (a retry can't accidentally count the same incident twice). That
"SEEN BEFORE" line in the email is this memory at work — and in testing, the
recurrence count really did climb each time the same problem came back.
HOW THE PIECES FIT
The whole thing is serverless and event-driven — nothing polls, everything
reacts.
CloudWatch Alarm --(SNS)--> Trigger Lambda (dedup + invoke)
|
v
AgentCore Runtime (Strands agent + Nova Pro)
| | |
recall/write | | (MCP) | (SES)
v v v
AgentCore Memory AgentCore Gateway On-call email
|
v
6 read-only tools
(optional) Runtime .... propose fix ....> Approval + Cedar policy gateFollowing an incident through it:
- TRIGGER: a CloudWatch alarm sends a message to SNS, which runs a trigger Lambda. The Lambda takes a DynamoDB lock first, so ten rapid alarms create ONE investigation, not ten. An EventBridge Scheduler also does regular health checks.
- AGENT: a Strands agent runs on AgentCore Runtime using Nova Pro. Each run is capped at 20 tool calls and 300 seconds.
- TOOLS: served over AgentCore Gateway, with an in-process fallback so a gateway outage never stops a run.
- REPORTING: the report goes out by SES. If email fails, it's saved to S3 so the diagnosis is never lost.
- REMEDIATION (off by default): an optional path with a human approval step and a policy check.

THE HARD PART: MAKING IT SAFE TO TOUCH PRODUCTION
Diagnosing is safe — the worst case is a wrong email. Changing production is
not. An AI confidently making a wrong change is worse than the original problem.
So this part starts from distrust:
- It's off by default.
- It can only do three reversible things: scale out a service, restart a task, or raise a Lambda's concurrency. No deletes. Never a database.
- When it's on, two separate gates stand in front of every action.
GATE 1 — A HUMAN MUST APPROVE. The agent never acts on its own. It emails an
approve/reject link with a one-time token: single-use, tied to one incident and
one action, and it expires. The token is stored only as a scrambled fingerprint,
never the real thing. And every rejection looks identical, so nobody can poke the
endpoint to learn which tokens are real:
decision = tokens.validate_and_consume(request.token, expected_decision)
if not decision.authorized:
# One identical 403 for unknown, expired, reused, wrong-incident, malformed.
return _refuse(request, decision.reason)This gate protects against the agent being WRONG — a human checks the fix first.
GATE 2 — A POLICY CHECK (CEDAR). Even after a human approves, a separate policy
engine checks the action independently. This protects against the agent being
TRICKED (say, by a poisoned log line). The rules are written so anything
unexpected is denied:
// No action without a real, approved human decision.
@id("forbid-without-recorded-approval")
forbid (principal, action, resource)
unless {
context has approval &&
context.approval.decision == "approved"
};
// Never touch a database — blocked by service name.
@id("forbid-database-service-namespaces")
forbid (principal, action, resource)
when {
["rds","dynamodb","elasticache","memorydb","docdb",
"neptune","redshift","timestream","qldb"].contains(resource.aws_service)
};If the policy engine can't be reached, the answer is automatically NO — never a
silent yes. And every decision is written to an S3 audit log that can't be edited
(write-only permission), so there's always an honest record.
In short: a change happens only if a human approved it AND the policy allowed it.
Either one can say no.
One honest trade-off: turning remediation on creates a PUBLIC approval link (so
you can approve from your phone), protected only by that one-time token. That's
on purpose — and it's why remediation ships turned off.
A FEW PRINCIPLES THAT HELD UP
CODE ENFORCES WHAT MUST BE TRUE, NOT THE PROMPT. The model writes the words,
but the confidence cap, the recurrence count, and secret-hiding are enforced
in code — because a model has been caught ignoring an instruction in a prompt:
Never claim "high" confidence when evidence is missing — enforced here.
if gaps and _rank(level) > _rank("medium"):
level = "medium"- NO SECRET EVER REACHES A REPORT OR A LOG. Emails, logs, and traces are all scrubbed of things that look like keys or tokens, and tests prove it.
- TEST WITH GENERATED INPUTS, NOT HAND-PICKED ONES. Instead of a few examples, thousands of random inputs check that the rules always hold:
@given(tool_arguments()) # Hypothesis generates lots of inputs
def test_no_tool_ever_raises(args):
assert isinstance(tool(**args), dict) # always a dict, never a crashWHAT IT IS — AND WHAT IT ISN'T
It's a solid monitoring-and-diagnosis agent: point any alarm at it and it
investigates and reports on its own, day and night, getting smarter about repeat
problems.
It is NOT a magic "self-healing infrastructure" system. It only sees what has an
alarm. It can only fix a few reversible things, and even those need approval.
That narrowness is the point — it's the safety design. In one line:
An always-on junior SRE that does the first 30 minutes of every incident for
you, remembers every incident it has seen, and can safely apply a few fixes
with your one-click approval.
WHAT'S NEXT
The natural next step is safe auto-remediation: a second agent that acts on the
diagnosis — scaling a service up under real load, watching it, and scaling back
down when the surge passes. It keeps the policy check, swaps the human click for
a tightly-bounded auto-approve, and undoes the change if it doesn't help.
The key idea is a growing list of known, reversible fixes, each added carefully —
never one agent with open-ended power told to "fix whatever's broken." And it
escalates to a human whenever it isn't sure. "I don't know how to fix this
safely" is a good answer for an agent to give.
CLOSING THOUGHT
The hard part of building an agent that touches production isn't making it
capable — models are already capable. It's making it trustworthy: honest about
what it knows, unable to leak a secret, unable to act without both a human and a
policy agreeing, and easy to audit afterward.
Capability is the easy part. Safety and honesty are what decide whether anyone
would actually let an agent near their systems.
Built on AWS Bedrock AgentCore.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article