I Built an Agentic SRE on AWS DevOps Agent — Here's What the Official Guide Left Out

I built an autonomous SRE agent using AWS DevOps Agent that investigates Lambda incidents without waking me up — correlating CloudWatch metrics, logs, and GitHub deployments, then posting root cause analysis to Slack. This post covers the full CDK implementation in a single AWS account (no Splunk, no 3-account setup), the webhook handler with HMAC signing, and the one thing the official docs don't mention: the SKILL.md frontmatter requirement that causes silent upload failures.

Mohammed Anes

Published May 23, 2026

Last modified May 23, 2026

When a CloudWatch alarm fires at 2 AM, my agent wakes up, investigates the Lambda function, correlates logs and metrics, posts the root cause to Slack, and goes back to sleep. I don't get paged until there's actually something for me to do.

That's what I built this week. Here's exactly how — including the parts the AWS documentation doesn't cover.

The Problem With On-Call

Every engineer who has been on-call knows the drill. An alarm fires. You open your laptop half-asleep. You check CloudWatch. You check logs. You check recent deployments. You check metrics. Forty-five minutes later you find out someone deployed bad code two hours ago and the fix is a one-line rollback.

AWS DevOps Agent can do that entire loop autonomously. The moment an alarm fires, it starts investigating — correlating telemetry, checking deployment history, reading your runbooks, and posting findings to Slack. You only get involved when there's a decision to make.

I built a working end-to-end demo of this. Let me walk you through every step.

What I Built

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
CloudWatch Alarm (error rate / duration / throttling)
        │
        ▼
    SNS Topic
        │
        ▼
  Webhook Handler Lambda
  (HMAC-signed payload, secrets in Secrets Manager)
        │
        ▼
  AWS DevOps Agent Space (agentic-sre-space)
        │
        ├── Reads CloudWatch metrics + logs directly
        ├── Checks GitHub for recent deployments
        ├── Follows your custom Skill (SRE runbook)
        │
        ▼
  Root cause + Mitigation Plan
        │
        ├── Posts findings to Slack (#agent-sre-incidents)

Key design decision: single-account, no Splunk.

The AWS official blog uses a 3-account setup with Splunk. That is an enterprise reference architecture. Most developers do not have Splunk or three AWS accounts lying around. My version runs entirely in one account and costs under $1/month in infrastructure (excluding DevOps Agent itself, which has a 2-month free trial).

The Official Guide vs This Build

Topic	AWS Official Blog	This Build
Architecture	3 accounts + Splunk	Single account, no Splunk
IaC	Manual console steps	Full CDK (Python)
Webhook handler	Node.js snippet	Complete Python Lambda with caching
Custom Skills	Mentioned briefly	Full runbook with frontmatter format
Demo triggers	Not provided	One-liner trigger script
Skill zip format	Not documented	Required frontmatter discovered the hard way

That last row is the most important. Keep reading.

Part 1 — The Infrastructure (CDK Python)

Three CDK stacks that deploy in order.

Stack 1: Demo App

A Lambda function with controllable failure modes — the "production app" being monitored.

python

1
2
3
4
5
6
7
8
9
10
self.lambda_fn = _lambda.Function(
    self,
    "DemoAppFunction",
    function_name="agentic-sre-demo-app",
    runtime=_lambda.Runtime.PYTHON_3_12,
    handler="index.handler",
    code=_lambda.Code.from_asset("demo_app"),
    timeout=Duration.seconds(30),
    memory_size=128,
)

The Lambda supports four invocation modes:

python

1
2
3
4
5
6
7
8
9
10
11
def handler(event, context):
    simulate = event.get("simulate", "normal")

    if simulate == "error":
        raise Exception("DB connection timeout: could not reach postgres://demo-db:5432")

    if simulate == "slow":
        time.sleep(random.uniform(25, 28))  # near the 30s timeout
        return {"statusCode": 200, "body": "ok"}

    return {"statusCode": 200, "body": "ok"}

This lets you trigger specific failure modes on demand without needing a real broken dependency.

Stack 2: CloudWatch Alarms

Three alarms covering the most common Lambda failure categories:

python

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
# Error rate > 20% over 5 minutes
error_rate_alarm = cw.Alarm(
    self, "ErrorRateAlarm",
    alarm_name="agentic-sre-high-error-rate",
    metric=cw.MathExpression(
        expression="errors / invocations * 100",
        using_metrics={
            "errors": lambda_fn.metric_errors(period=Duration.minutes(5), statistic="Sum"),
            "invocations": lambda_fn.metric_invocations(period=Duration.minutes(5), statistic="Sum"),
        },
    ),
    threshold=20,
    evaluation_periods=1,
)

# P99 duration > 24s (80% of 30s timeout)
duration_alarm = cw.Alarm(
    self, "DurationAlarm",
    alarm_name="agentic-sre-high-duration",
    metric=lambda_fn.metric_duration(period=Duration.minutes(5), statistic="p99"),
    threshold=24000,
    evaluation_periods=2,
)

# Any throttling in 5 minutes
throttle_alarm = cw.Alarm(
    self, "ThrottleAlarm",
    alarm_name="agentic-sre-throttling",
    metric=lambda_fn.metric_throttles(period=Duration.minutes(5), statistic="Sum"),
    threshold=1,
    evaluation_periods=1,
)

All three alarms publish both ALARM and OK state changes to a shared SNS topic. The OK state matters — it lets the agent post a resolution to Slack when the incident clears.

Stack 3: Webhook Handler

This Lambda bridges CloudWatch alarms and DevOps Agent. It receives SNS notifications, formats them to the DevOps Agent webhook schema, signs with HMAC-SHA256, and forwards to the agent.

Secrets Manager over environment variables — the webhook URL and secret are never stored in Lambda environment variables. The handler caches the secret at module level so it fetches from Secrets Manager once per cold start, not on every invocation:

python

1
2
3
4
5
6
7
8
9
10
_secret_cache = None

def get_webhook_config():
    global _secret_cache
    if _secret_cache:
        return _secret_cache
    client = boto3.client("secretsmanager")
    response = client.get_secret_value(SecretId=SECRET_NAME)
    _secret_cache = json.loads(response["SecretString"])
    return _secret_cache

HMAC signing in the exact format DevOps Agent expects:

python

1
2
3
4
5
6
7
8
def sign_payload(payload_str: str, secret: str, timestamp: str) -> str:
    message = f"{timestamp}:{payload_str}"
    sig = hmac.new(
        secret.encode("utf-8"),
        message.encode("utf-8"),
        hashlib.sha256,
    ).digest()
    return base64.b64encode(sig).decode("utf-8")

Part 2 — Setting Up the Agent Space

Ten console steps. I'll cover only what the official docs skip.

IAM Role Shortcut

The console has an Auto-create option for both the Agent Space role and the Operator App role. Use it. It creates DevOpsAgentRole-AgentSpace and DevOpsAgentRole-WebappAdmin with the correct trust policies automatically.

Webhook Setup — 3-Step Wizard

Step 1 — Verify data schema. The console shows you the exact payload format. Our handler already matches it.

Step 2 — HMAC signing format. Our handler already implements this.

Step 3 — Generate URL and secret key. Copy the secret immediately — it is only shown once.

Save both to Secrets Manager right away:

bash

1
2
3
4
aws secretsmanager create-secret \
  --name agentic-sre/devops-agent-webhook \
  --secret-string '{"url":"YOUR_WEBHOOK_URL","secret":"YOUR_SECRET"}' \
  --region us-east-1

GitHub Connection

GitHub must be connected via the console UI first (OAuth flow). The CLI cannot register GitHub directly. Go to: Agent Space → Features tab → CI/CD Pipeline → GitHub → Install and Authenticate.

Part 3 — The Custom Skill (The Part Nobody Documents)

This is the most underappreciated feature of DevOps Agent and the least documented.

A Skill is a Markdown file containing your SRE runbook. You upload it to the Agent Space and the agent reads it at the start of every investigation. Instead of reasoning from scratch each time, it follows your documented investigation sequence.

When I triggered a test incident, I could see this in the Operator Console timeline:

1
Read skill: /aidevops/skills/user/understanding-agent-space/SKILL.md

The agent literally reads your runbook before it starts investigating. The quality of your investigation scales directly with the quality of your documentation.

The Frontmatter Requirement Nobody Mentions

The console says "upload a zip file containing a SKILL.md file." What it does not say is that the SKILL.md must have a YAML frontmatter block or the upload fails silently.

I found this out the hard way after my first upload appeared to succeed but the skill never loaded.

markdown

1
2
3
4
5
6
7
8
9
10
11
12
---
name: Lambda Incident Investigation Runbook
description: Use this skill when investigating AWS Lambda function errors,
  timeouts, high duration, or throttling incidents. Guides the agent through
  a 5-step investigation sequence covering alarm confirmation, metric analysis,
  log inspection, deployment correlation, and root cause identification.
---

# Lambda Incident Investigation Runbook

## Investigation Sequence
...

The zip structure must be exactly:

1
2
your-skill.zip
└── SKILL.md   ← root level, must have frontmatter

My runbook encodes a 5-step investigation sequence:

Confirm the alarm and check state history
Pull Lambda metrics (Errors, Invocations, Duration p50/p95/p99, Throttles)
Query CloudWatch Logs for errors and timeout messages
Correlate with recent GitHub deployments
Map symptoms to root cause categories

Part 4 — Triggering the Demo

Deploy everything:

bash

1
2
3
4
cd agentic-sre
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
cdk deploy --all

Trigger an investigation:

bash

1
python scripts/trigger.py error

Or trigger naturally with real invocations (recommended):

bash

1
python scripts/trigger.py burst 15

This sends 15 async error invocations, which trips the CloudWatch alarm after the 5-minute evaluation window and gives the agent actual logs to investigate.

Part 5 — What Actually Happened

In Slack (#agent-sre-incidents):

The agent posted investigation updates in real time. Multiple alarms were grouped automatically — the agent identified follow-up alarms as continuations of the same incident and linked them to the primary investigation rather than opening duplicate tickets.

In the Operator Console:

The investigation timeline showed every step the agent took:

Read the SKILL.md runbook
Discovered the account topology (1,200+ resource relationships mapped)
Checked CloudWatch metrics and logs
Queried GitHub for recent deployments
Generated root cause analysis
Produced a structured mitigation plan

The Topology tab showed a visual graph of resource relationships — the agent understands how your Lambda connects to other services before it starts investigating.

The Honest Part — Investigation Gaps

The agent also surfaced two gaps worth sharing:

"Cannot directly query Bedrock AgentCore API to verify runtime operational status — GetAgentRuntime, ListAgentRuntimes, InvokeRuntime APIs are not available in the current toolset."

"No information about what triggered this investigation — the investigation was created with N/A for both description and starting point."

The second gap happened because I triggered the alarm manually via set-alarm-state rather than through actual Lambda invocations. The fix is to use burst mode so real invocation data exists.

These gaps are valuable. They show the agent being honest about the limits of its investigation rather than fabricating a root cause. That is exactly the behavior you want from an autonomous system.

What This Cost

Service	Monthly Cost
Lambda (demo app + webhook handler)	$0 (free tier)
CloudWatch Alarms (3)	$0.30
SNS	$0 (free tier)
Secrets Manager	$0.40
Total infra	~$0.70/month
AWS DevOps Agent	2-month free trial, then usage-based

Key Learnings

1. Skills are your biggest leverage point. The agent follows your SKILL.md exactly. A well-written runbook is the difference between a generic investigation and a useful one. Invest time here.

2. Secrets Manager over environment variables. Webhook URL and HMAC secret should never be in Lambda environment variables. Module-level caching in Secrets Manager means one fetch per cold start.

3. Single-account is enough to start. The official 3-account setup is for production enterprise environments. One account is perfectly sufficient for learning and building.

4. Alarm grouping works out of the box. When multiple alarms fire for the same incident, the agent automatically identifies them as related. Genuinely useful behavior you get for free.

5. Trigger with real invocations, not manual alarm overrides. Use burst mode. The agent needs actual logs and metrics to investigate, not just an alarm state change.

What I'm Building Next

Adding a human approval gate before any remediation action — a Step Functions workflow that pauses the agent before it executes anything destructive and sends a Slack approval request. Essentially the MCP Governance pattern applied to DevOps Agent.

Full Source Code

CDK stacks, webhook handler, demo app, Skill zip, and trigger scripts:

👉 https://github.com/MohammedAnes/agentic-sre

Resources

AWS Community Builder — AI/ML. Building on AgentCore and DevOps Agent. Let's connect if you are too.

# amazon-bedrock-agentcore # aws-cdk # amazon-cloudwatch # sre # devops

Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.

Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article

Builder Center

I Built an Agentic SRE on AWS DevOps Agent — Here's What the Official Guide Left Out

Mohammed Anes

The Problem With On-Call

What I Built

The Official Guide vs This Build

Part 1 — The Infrastructure (CDK Python)

Stack 1: Demo App

Stack 2: CloudWatch Alarms

Stack 3: Webhook Handler

Part 2 — Setting Up the Agent Space

IAM Role Shortcut

Webhook Setup — 3-Step Wizard

GitHub Connection

Part 3 — The Custom Skill (The Part Nobody Documents)

The Frontmatter Requirement Nobody Mentions

Part 4 — Triggering the Demo

Part 5 — What Actually Happened

The Honest Part — Investigation Gaps

What This Cost

Key Learnings

What I'm Building Next

Full Source Code

Resources