I Built an SRE That Fixes AWS Incidents While You Sleep — On Bedrock AgentCore Harness

An autonomous SRE agent on AWS Bedrock AgentCore Harness that reads real CloudWatch and CloudTrail data, classifies incident risk with property-tested logic, and executes safe remediation on EC2, RDS, and ECS gating every risky action through Slack human-in-the-loop approval. Powered by Amazon Nova with mid-session model switching (Micro for triage, Pro for deep reasoning)Built end-to-end with the AgentCore CLI, 17 MCP tools, and 350 property-based tests.

S Logesh

AI Engineer | AWS Community Builder – AI Engineering | AWS User Group Madurai Leader

Published Jun 16, 2026

Last modified Jun 17, 2026

An autonomous SRE agent (sre-harness-agentcore) that monitors, diagnoses, and remediates AWS infrastructure — with deterministic safety guarantees, human-in-the-loop approval over Slack, mid-session model switching, and authorization policies that evolve from incidents. Built spec-first, with 350 property-based tests, and deployed on AWS Bedrock AgentCore Harness with Amazon Nova.

TL;DR

I set out to build an SRE agent that responds to infrastructure incidents in seconds but can never cause production damage. The result runs on AWS Bedrock AgentCore Harness, reasons with Amazon Nova (Micro for cheap triage, Pro for deep analysis — switched mid-session), and executes real remediation on EC2/RDS/ECS through 17 tools hosted in a Lambda behind an AgentCore Gateway.

Every risky action is gated by a human approving in Slack. Every decision is backed by property-tested logic, not LLM guesswork. The agent even proposes Cedar authorization policies to prevent incidents from recurring.

It's live. It started a real EC2 instance through the full safety chain. Here's how I built it — and the platform gaps I had to engineer around.

The Problem

Production SRE faces four demands that pull against each other:

Speed — incidents need sub-minute response
Safety — a wrong remediation cascades into a bigger outage
Cost — 24/7 human on-call is expensive and burns people out
Learning — the same incidents recur because fixes never get codified into guardrails

Most "AI ops" demos solve speed and ignore the other three. I wanted all four: an agent that acts instantly, only within proven boundaries, at low cost, and gets smarter after every incident.

Why Spec-First (and Property-Based Testing)

Before writing code, I wrote a formal spec: 15 requirements, 100+ acceptance criteria in EARS notation (WHEN/IF/WHILE/THE…SHALL). For an agent that can touch production infrastructure, "it seems to work" is not good enough — I needed executable correctness guarantees.

So the core logic is validated by property-based tests (Hypothesis). Instead of a handful of examples, each property asserts an invariant across thousands of generated inputs. Example — risk classification must be total (return exactly one tier for any input):

1
2
3
4
5
6
7
8
9
10
11
12
13
# src/agent/remediation/risk.py — deterministic, total risk classifier

def classify_risk_tier(scope_count: int, reversible: bool,
                       data_loss_potential: DataLossPotential) -\> RiskTier:
    """Pure, total. Returns exactly one tier for every possible input."""
    score = {"none": 0, "low": 2, "high": 4}[data_loss_potential.value]
    score += 0 if reversible else 2
    score += 0 if scope_count \<= 1 else (1 if scope_count \<= 5
              else (2 if scope_count \<= 20 else 3))
    if score \<= 0: return RiskTier.LOW
    if score \<= 2: return RiskTier.MEDIUM
    if score \<= 4: return RiskTier.HIGH
    return RiskTier.CRITICAL

1
2
3
4
5
# tests/property/test_p04_risk_tier_totality.py
@given(scope=st.integers(), reversible=st.booleans(),
       dlp=st.sampled_from(DataLossPotential))
def test_risk_tier_is_total(scope, reversible, dlp):
    assert classify_risk_tier(scope, reversible, dlp) in RiskTier

The build came together as 51 tasks across 10 dependency waves, ending green: 350 tests passing, mypy --strict clean, ruff clean, 13 property tests at ≥100 examples each. Key invariants proven this way:

Risk classification is total (Property 4)
Anomaly classification is exhaustive — every observation is exactly NORMAL or ANOMALY (Properties 1–2)
Cedar parse→print→parse round-trips to identical authorization decisions (Property 11)
Knowledge-graph adds are idempotent and confluent — retries and reordering never change state (Properties 10, 15)
Audit sequence numbers increase by exactly 1 with no gaps (state-machine property)

Architecture

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
        EventBridge (every 5 min)
                 │
                 ▼
          Poller Lambda ──────────► AgentCore Harness
                                    (Nova Micro / Lite / Pro)
                                    • persistent memory
                                    • browser + code interpreter
                                            │
                                    AgentCore Gateway (toolsgw)
                                            │  SigV4
                                            ▼
                                  selfheal-tools Lambda
                            ┌───────────────────────────────┐
                            │ 17 tools, backed by the tested │
                            │ `agent` package (350 tests):   │
                            │  Detect   → classify_anomaly,   │
                            │             get_cloudwatch_*    │
                            │  Diagnose → lookup_cloudtrail,  │
                            │             describe ec2/rds/ecs│
                            │  Decide   → assign_risk,        │
                            │             check_blast_radius  │
                            │  Approve  → request_approval,   │
                            │             send_slack_approval │
                            │  Act      → execute_remediation │
                            │  Learn    → propose/promote     │
                            │             cedar policies      │
                            │  Govern   → get_budget_status   │
                            └───────────────────────────────┘
                                            │
                  ┌─────────────────────────┼─────────────────────────┐
                  ▼                         ▼                         ▼
            Real AWS APIs            Slack (2-way HITL)        Cedar policy engine
         (EC2, RDS, ECS,            #agent channel via          LOG_ONLY → ENFORCE
          CloudWatch, CloudTrail)   webhook + Events API

The Self-Healing Loop

Detect — EventBridge wakes the agent every 5 minutes; it pulls real CloudWatch metrics and classifies them against per-metric thresholds.
Diagnose — correlates the anomaly with recent CloudTrail changes and EC2/RDS/ECS topology, plus memory of past incidents.
Decide — proposes a remediation, calls assign_risk (scope × reversibility × data-loss → LOW/MEDIUM/HIGH/CRITICAL), and check_blast_radius.
Gate — HIGH/CRITICAL → send_slack_approval; an operator approves or denies in Slack.
Act — execute_remediation runs only safe, reversible actions (reboot/stop/start) within the blast radius.
Learn — proposes a Cedar policy (in LOG_ONLY) to prevent recurrence; an operator later promotes it to ENFORCE.
Remember — the resolved incident is written to memory for future correlation.

Six Design Decisions That Mattered

1. AgentCore Harness over a DIY agent loop. The harness gives stateful microVM sessions, persistent memory, a built-in browser + code interpreter, and — crucially — per-invocation model override. I didn't have to build an agent runtime; I configured one.

2. Amazon Nova, not Claude. My AWS account is a channel-program account, which is currently blocked from Anthropic Claude on Bedrock. Nova (Micro/Lite/Pro) is first-party and works — and it turned out to be the right cost/quality ladder anyway.

3. Deterministic tools over LLM judgment. Risk tiers, blast-radius checks, and Cedar validation are pure, property-tested functions. The model decides what to do; the tested code decides whether it's allowed. The LLM never "estimates" risk.

4. Lambda as the tool host. This was forced by a platform gap (below) — but it turned out clean: the same tested agent package runs in the Lambda, and the gateway authenticates into Lambda natively with SigV4.

5. Cedar for evolving guardrails. Cedar is a formal authorization language with deterministic evaluation and a human-readable syntax. New policies start in LOG_ONLY (evaluated, not enforced) and only flip to ENFORCE after a human approves — safe, reviewable evolution.

6. Slack as the operator UI. SREs don't sit in a custom dashboard waiting for alerts. They live in Slack. So Slack is the frontend: the agent posts structured approval cards there, and operators reply in-channel.

The Hard Part: Deploying on a Preview Platform

The build was the easy half. Getting it live on the AgentCore Harness preview meant engineering around real platform gaps. These war stories are the most useful part of this write-up.

Gap 1 — Claude is blocked on channel-program accounts

First invoke failed with a model-access error. Channel-program/reseller AWS accounts are currently blocked from Anthropic models on Bedrock. Fix: pin Amazon Nova with the us. inference-profile prefix (us.amazon.nova-pro-v1:0).

Gap 2 — CDK refuses root, and a stale bootstrap bucket

The CLI deploy (CDK under the hood) won't run as root, then failed on a stale CDKToolkit stack whose asset bucket had been deleted. Fix: an IAM admin user (not root) + cdk bootstrap after clearing the stale stack.

Gap 3 — The harness couldn't authenticate into my MCP tools (403)

I first shipped the tools as their own AgentCore Runtime MCP server. The harness's remote_mcp tool type sends only static headers — it can't SigV4-sign requests to an IAM-protected runtime. Result: 403 Forbidden, twice.

> SigV4 is AWS's machine-to-machine authentication: the caller signs the request with its secret key; the receiver verifies the signature. remote_mcp has no way to produce that signature.

Gap 4 — The gateway "passthrough" exposed no tools

I put a gateway in front of the runtime using --runtimes. But a runtime passthrough target forwards raw invocations without running tools/list — so the model saw a connected gateway with zero callable tools and fabricated answers instead. And the CLI's mcp-server gateway target only offered OAuth/API-key outbound auth — no SigV4 option, no IAM credential type. Dead end for an IAM-protected MCP runtime.

The fix: a Lambda gateway target

The one target the gateway authenticates into natively is AWS Lambda. So I moved the exact same tested code into a Lambda and registered it as a gateway Lambda target with a tool schema. The gateway routes each tool call to the Lambda, stripping its prefix:

1
2
3
4
5
# lambda/handler.py — gateway → Lambda tool dispatcher
def lambda_handler(event, context):
    raw = context.client_context.custom["bedrockAgentCoreToolName"]
    tool = raw.split("___")[-1]          # strip "targetname___" prefix
    return _TOOLS[tool](event)           # 17 tools, backed by `agent` pkg

CloudWatch logs confirmed the first real tool call: a 2 ms execution, the harness reasoning over deterministic output instead of guessing.

Gap 5 — EventBridge can't target the harness directly

bedrock-agentcore isn't a supported EventBridge Scheduler target. Fix: EventBridge → a thin poller Lambda → invoke the harness. Same outcome, serverless, cheap.

Gap 6 — Two-way Slack, and a blocked Function URL

Outbound Slack was easy (Incoming Webhook). For the operator's reply to come back, Slack's Events API needs a public HTTPS endpoint. A Lambda Function URL kept returning 403 Forbidden — the account blocks public Function URLs (channel-program restriction again). Fix: API Gateway in front of the receiver Lambda. Slack's URL verification passed, and typing "approve" in the channel now flows back to the agent.

Mid-Session Model Switching (Cost Control That Works)

The harness lets you override the model per invocation while keeping the same session — context carries over. That makes a true triage→reasoning escalation ladder:

1
2
3
4
5
6
7
8
9
10
11
export SID=$(uuidgen)

# Turn 1 — cheap triage with Nova Micro
agentcore invoke --harness Harnessinfra \
  --model-id us.amazon.nova-micro-v1:0 --session-id "$SID" \
  "Triage: is instance i-xxx healthy? Rate confidence 0-1."

# Turn 2 — escalate to Nova Pro on the SAME session; context preserved
agentcore invoke --harness Harnessinfra \
  --model-id us.amazon.nova-pro-v1:0 --session-id "$SID" \
  "Confidence was low. Full investigation and remediation plan."

Micro handles the 90% of cycles where everything is fine; Pro is reserved for the incidents that actually need deep reasoning. All three tiers (Micro/Lite/Pro) are verified working on the same harness.

Human-in-the-Loop Over Slack

Outbound — the agent posts a structured approval card:

1
2
3
4
5
6
7
8
9
10
def send_slack_approval(event):
    blocks = [
        {"type": "header", "text": {"type": "plain_text",
         "text": f"HITL Approval Required - {event['risk_tier']}"}},
        {"type": "section", "text": {"type": "mrkdwn",
         "text": f"*Action:* {event['action']}\n"
                 f"*Targets:* {', '.join(event['targets'])}\n"
                 f"*Diagnosis:* {event['diagnosis']}"}},
    ]
    post_to_webhook({"blocks": blocks})

Inbound — API Gateway → Lambda receives the operator's reply and closes the loop:

1
2
3
4
5
6
7
8
9
def lambda_handler(event, context):
    payload = json.loads(event["body"])
    if payload.get("type") == "url_verification":   # Slack handshake
        return {"statusCode": 200, "body": payload["challenge"]}
    text = payload["event"]["text"].lower()
    if "approve" in text:
        notify_slack(":white_check_mark: Approved. Remediation will proceed.")
    elif "deny" in text:
        notify_slack(":x: Denied. Remediation blocked.")

HIGH/CRITICAL actions block until a human approves; LOW/MEDIUM auto-approve only if every target is inside the configured allow-list — the safe default.

Cedar Policies That Evolve From Incidents

After resolving an incident, the agent proposes a Cedar policy to prevent recurrence — validated by the tested parser before it's accepted, created in LOG_ONLY with a 7-day review window:

1
2
3
4
5
6
7
8
\> Propose a Cedar policy: forbid(principal, action, resource)
  when { resource.tag == "prod" }; for incident INC-001.

→ accepted: true
  policy_id: policy-63da4777e49f
  mode: LOG_ONLY
  review_expiry: 2026-06-23   (7 days)
  status: pending_review

An operator promotes it to ENFORCE (or rejects it) before expiry. Authorization boundaries get tighter over time, automatically, but never without human sign-off.

Proof It's Real

The agent started a real EC2 instance through the full safety chain:

1
2
3
4
5
6
7
8
9
10
Prompt: "Start instance i-09a78bc685540a484. Auto-approve prefix
         arn:aws:ec2:us-east-1:\<account\>."

Agent: check_blast_radius → within scope
       assign_risk → LOW (1 instance, reversible, no data loss)
       request_approval → auto_approved (target matches prefix)
       execute_remediation → success

$ aws ec2 describe-instance-status --instance-ids i-09a78bc685540a484
→ running

It read real account activity:

1
2
3
\> What changed in the account in the last 60 minutes?
Agent → lookup_cloudtrail_events → AssumeRole, GetAgentRuntime, Decrypt
Agent → describe_instances → i-09a78bc685540a484 (t3.micro, us-east-1b, vpc-…)

It posted a real Slack approval and received the operator's "approve" back through the API Gateway receiver.

What's Live

Requirement	Status
1 — Continuous monitoring	EventBridge → poller Lambda → harness, every 5 min
2 — Anomaly detection	`classify_anomaly` + real CloudWatch metrics
3 — Mid-session model escalation	Nova Micro → Pro, same session, context preserved
4 — Root cause analysis	CloudTrail + topology + memory correlation
5 — HITL remediation	Full chain proven on a live instance
6 — Cedar policy evolution	propose → LOG_ONLY → promote → ENFORCE
7 — Knowledge graph / topology	EC2 / RDS / ECS describe tools
8 — Long-term memory	AgentCore Memory (episodic, semantic, summarization)
9 — Gateway tool access	CloudWatch, CloudTrail, EC2, RDS, ECS, Health + Slack
10 — Browser + code interpreter	enabled on the harness
11 — Cost controls	`get_budget_status` invocation tracking
12 — Blast radius	`check_blast_radius` enforces account + region
13 — Audit trail	immutable, gap-free sequence numbers (property-tested)
14 — Config parse/print	round-trip safe (property-tested)
15 — Idempotent remediation + KG	idempotence + confluence (property-tested)

Cost

Component	Monthly (light usage)
Harness invocations (Nova Pro)	~$5–15
Lambda (17 tools, <100 ms each)	~$0.50
EventBridge (5-min cadence)	~$0.10
API Gateway (Slack events)	~$0.01
CloudWatch / CloudTrail reads	free tier
Total	~$6–16/month

What I Learned

Spec-first + property tests pay off when the agent can touch production. "Total," "idempotent," and "round-trips" are guarantees, not hopes.
The preview CLI has real gaps — no SigV4 on MCP targets, no IAM credential type, no public Function URLs, no direct EventBridge target. Lambda + API Gateway are the bridges.
Mid-session model switching is underrated. Triage cheap, escalate rarely — big cost lever, zero quality loss on hard incidents.
Deterministic tools beat LLM judgment for anything safety-critical. Let the model orchestrate; let tested code decide.
Meet operators where they are. Slack is the frontend. Don't build a dashboard nobody opens.

Try It

1
2
3
4
5
6
7
git clone https://github.com/logesh4v/sre-harness-agentcore.git
cd sre-harness-agentcore
pip install -e ".[dev]"
pytest                         # 350 tests pass

bash scripts/deploy_tools_lambda.sh    # deploy the 17-tool Lambda
# then follow the README for AgentCore harness + gateway wiring

Repo: https://github.com/logesh4v/sre-harness-agentcore

# agentic-ai # agents # agentic-ide # agent-to-agent # amazon-ec2

Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.

Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article

Builder Center

I Built an SRE That Fixes AWS Incidents While You Sleep — On Bedrock AgentCore Harness

S Logesh

TL;DR

The Problem

Why Spec-First (and Property-Based Testing)

Architecture

The Self-Healing Loop

Six Design Decisions That Mattered

The Hard Part: Deploying on a Preview Platform

Gap 1 — Claude is blocked on channel-program accounts

Gap 2 — CDK refuses root, and a stale bootstrap bucket

Gap 3 — The harness couldn't authenticate into my MCP tools (403)

Gap 4 — The gateway "passthrough" exposed no tools

The fix: a Lambda gateway target

Gap 5 — EventBridge can't target the harness directly

Gap 6 — Two-way Slack, and a blocked Function URL

Mid-Session Model Switching (Cost Control That Works)

Human-in-the-Loop Over Slack

Cedar Policies That Evolve From Incidents

Proof It's Real

What's Live

Cost

What I Learned

Try It