Simulating and Investigating Your First Incident (Part 4)

In this part, we will: Set up the Agent Space Webhook to enable autonomous, alarm-triggered investigations Wire up CloudWatch → SNS → Lambda → DevOps Agent into a fully automated incident pipeline Intentionally spike CPU on the EC2 instance to trigger the alarm Watch the agent automatically start an investigation — no human intervention Review the investigation results: root cause, affected resources, and remediation suggestions

Series: Setting Up AWS DevOps Agent from Scratch (7 articles)

1
What is AWS DevOps Agent? A Beginner’s Guide to Autonomous Incident Response — PART 1
…
4
Simulating and Investigating Your First Incident (Part 4) This article
5
Cross-Cloud Integration with Azure (Part 5)

Fawaz c p

Published Jul 16, 2026

Last modified Jul 16, 2026

Let’s trigger our first incident.

The Goal: True Autonomous Incident Response

Before we start, it’s worth being clear on what we’re building.

Most monitoring setups alert a human, who then logs into a dashboard, clicks around, and tries to figure out what’s wrong. The DevOps Agent changes this flow completely:

1
2
3
4
5
6
7
8
9
10
11
CloudWatch Alarm fires
        ↓
   SNS Topic notified
        ↓
Lambda function transforms + signs the payload
        ↓
Agent Space Webhook receives the event
        ↓
DevOps Agent automatically starts an investigation
        ↓
Root cause analysis complete — results waiting in the dashboard

By the end of this part, an alarm firing will trigger a full investigation automatically, and you’ll have results ready to read by the time you open the dashboard.

Step 1: Generate the Agent Space Webhook

The Agent Space Webhook is the entry point that lets external events trigger investigations. We’ll generate a URL and secret key that our Lambda function will use to securely call the agent.

1.1 — Open the Webhook Wizard

1. In the AWS Management Console, go to AWS DevOps Agent → Agent Spaces and open your Agent Space.
2. On the Capabilities tab, scroll down to the Webhooks section.
3. Under Agent Space Webhook, click Add.

The wizard has three steps:

1.2 — Step 1: Understand the Data Schema

1
2
3
4
5
6
7
8
9
10
11
{
  "eventType": "incident",
  "incidentId": "string",
  "action": "created" | "updated" | "closed" | "resolved",
  "priority": "CRITICAL" | "HIGH" | "MEDIUM" | "LOW" | "MINIMAL",
  "title": "string",
  "description": "string",       // optional
  "timestamp": "string",         // optional — ISO 8601 format
  "service": "string",           // optional
  "data": {}                     // optional — attach the original alarm payload here
}

Keep this in mind — our Lambda function will transform CloudWatch’s alarm notification format into this schema. Click Next.

1.3 — Step 2: Understand HMAC Signing

The webhook uses HMAC-SHA256 to verify that requests are legitimate and haven’t been tampered with. Every request to the webhook URL must include two custom headers:

1
2
3
4
| Header                   | Value                             |
|--------------------------|-----------------------------------|
| `x-amzn-event-timestamp` | ISO 8601 timestamp of the request |
| `x-amzn-event-signature` | HMAC-SHA256 of `"<timestamp>:<JSON payload>"` using your secret, base64-encoded |

Our Lambda function will handle this signing automatically. Click Next.

1.4 — Step 3: Generate the URL and Secret

Click Generate URL and secret key
Immediately click Download .csv file— the secret key is only shown once and cannot be retrieved later.
Check ”I’ve saved and stored my URL and secret key”.
Click Add.

The webhook now appears under Agent Space Webhook with status Connected.

Press enter or click to view image in full size

Step 2: Store the Webhook Credentials Securely

Never hardcode webhook URLs or secrets in Lambda code. Store them in AWS Secrets Manager.

Open AWS Secrets Manager → Store a new secret.
Select Other type of secret.
Add two key-value pairs from the CSV you downloaded:
— Key: `webhookUrl` → Value: your webhook URL
— Key: `webhookSecret` → Value: your secret key
Name the secret: `devops-agent/webhook-credentials`
Click Next through the remaining steps and click Store.

Press enter or click to view image in full size

Step 3: Create an SNS Topic

CloudWatch alarms notify SNS topics, which then invoke Lambda. We need a topic to bridge the two.

Open the SNS Console → Topics → Create topic.
Type: Standard
Name: `devops-agent-alarm-topic`
Click Create topic and copy the Topic ARN — you’ll need it in the next step.

Press enter or click to view image in full size

Step 4: Update the CloudWatch Alarm to Notify SNS

Now connect the alarm we created in Part 3 to the SNS topic.

Open CloudWatch → Alarms → All alarms.
Select `devops-agent-demo-cpu-alarm` → Actions → Edit.
On the Configure actions page, click Add notification:
— Alarm state trigger: In alarm
— Send notification to: `devops-agent-alarm-topic`
Optionally add a second notification for OK state — this will send a `resolved` event to the agent when the alarm clears.
Click Update alarm.

Press enter or click to view image in full size

Step 5: Create the Lambda Bridge Function

This is the critical piece. The Lambda function receives the CloudWatch notification from SNS, transforms it into the webhook schema, signs it with HMAC-SHA256, and POSTs it to the Agent Space Webhook.

5.1 — Create the Function

Open Lambda → Create function
Function name:`devops-agent-alarm-bridge `
Runtime: Python 3.12
Architecture: x86_64
Click Create function.

5.2 — Grant Permission to Read the Secret

The Lambda function needs access to Secrets Manager. Add an inline policy to the Lambda’s execution role:

Go to Configuration → Permissions → click the execution role name to open IAM.
Click Add permissions → Create inline policy.
Use the following JSON:

1
2
3
4
5
6
7
8
9
10
  {
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "secretsmanager:GetSecretValue",
      "Resource": "arn:aws:secretsmanager:us-east-1:YOUR_ACCOUNT_ID:secret:devops-agent/webhook-credentials*"
    }
  ]
}

Replace `YOUR_ACCOUNT_ID` with your actual account ID. Click Create policy.

5.3 — Deploy the Function Code

Back in the Lambda function, replace the default code with the following:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
import json
import hmac
import hashlib
import base64
import urllib.request
import boto3
from datetime import datetime, timezone

def get_credentials():
    client = boto3.client("secretsmanager", region_name="us-east-1")
    response = client.get_secret_value(SecretId="devops-agent/webhook-credentials")
    return json.loads(response["SecretString"])

def sign_request(secret, timestamp, payload_str):
    message = f"{timestamp}:{payload_str}"
    sig = hmac.new(secret.encode("utf-8"), message.encode("utf-8"), hashlib.sha256)
    return base64.b64encode(sig.digest()).decode("utf-8")

def map_priority(alarm_name):
    name = alarm_name.lower()
    if "critical" in name:
        return "CRITICAL"
    if "high" in name or "cpu" in name or "memory" in name:
        return "HIGH"
    return "MEDIUM"

def map_action(state):
    return {
        "ALARM": "created",
        "OK": "resolved",
        "INSUFFICIENT_DATA": "updated"
    }.get(state, "created")

def lambda_handler(event, context):
    credentials = get_credentials()
    webhook_url = credentials["webhookUrl"]
    webhook_secret = credentials["webhookSecret"]

    # Parse the CloudWatch alarm notification from SNS
    sns_message = json.loads(event["Records"][0]["Sns"]["Message"])

    alarm_name = sns_message.get("AlarmName", "Unknown Alarm")
    alarm_description = sns_message.get("AlarmDescription", "")
    new_state = sns_message.get("NewStateValue", "ALARM")
    alarm_arn = sns_message.get("AlarmArn", alarm_name)

    payload = {
        "eventType": "incident",
        "incidentId": alarm_arn,
        "action": map_action(new_state),
        "priority": map_priority(alarm_name),
        "title": alarm_name,
        "description": alarm_description or f"CloudWatch alarm entered {new_state} state",
        "timestamp": datetime.now(timezone.utc).isoformat(),
        "service": "EC2",
        "data": sns_message
    }

    payload_str = json.dumps(payload)
    timestamp = datetime.now(timezone.utc).isoformat()
    signature = sign_request(webhook_secret, timestamp, payload_str)

    req = urllib.request.Request(
        webhook_url,
        data=payload_str.encode("utf-8"),
        headers={
            "Content-Type": "application/json",
            "x-amzn-event-timestamp": timestamp,
            "x-amzn-event-signature": signature,
        },
        method="POST"
    )

    with urllib.request.urlopen(req) as response:
        body = response.read().decode()
        print(f"Webhook response: {response.status} - {body}")

    return {"statusCode": 200}

Click Deploy.

Press enter or click to view image in full size

5.4 — Add SNS as a Trigger

In the Lambda function, click + Add trigger.
Select SNS as the source.
Choose `devops-agent-alarm-topic` from the dropdown.
Click Add.

The trigger now appears in the Lambda function diagram.

Press enter or click to view image in full size

The autonomous pipeline is now complete. Let’s test it.

Step 6: Simulate the Incident — Spike the CPU

We’ll SSH into the EC2 instance and use the `stress` tool to push CPU utilisation above 80%, which will trigger the CloudWatch alarm.

6.1 — Connect to the Instance

SSH into the instance using your key pair:

1
ssh -i your-key.pem ec2-user@YOUR_INSTANCE_PUBLIC_IP

6.2 — Install and Run the Stress Tool

1
2
3
4
5
# Install the stress tool
sudo yum install -y stress

# Spike all CPU cores to 100% for 5 minutes
stress --cpu $(nproc) --timeout 300

Leave this running and switch back to the AWS console.

6.3 — Watch the Alarm Fire

Open CloudWatch → Alarms.
Within 1–2 minutes you should see `devops-agent-demo-cpu-alarm` transition from OK to In alarm.

When the alarm fires, the following happens automatically in the background:

CloudWatch publishes to the SNS topic
SNS invokes the Lambda function
Lambda transforms the alarm data and POSTs to the Agent Space Webhook
The DevOps Agent receives the event and starts an investigation

Press enter or click to view image in full size

Step 7: Watch the Agent Investigate

7.1 — Open the Incident Response Dashboard

Open the DevOps Agent web app (via Operator access on the Agent Space → Web app tab).
Click Incident Response in the left sidebar.
On the right panel, you’ll see the Investigations table.

Within a minute or two of the alarm firing, a new investigation will appear — created automatically by the webhook event. No manual action was needed.

Press enter or click to view image in full size

7.2 — Follow the Investigation in Real Time

Click on the investigation to open it. You’ll see the agent working through its analysis in real time:

Triage: The agent reads the alarm, identifies the affected resource (your EC2 instance), and determines severity.
Topology traversal: It walks the resource graph — instance → security group → subnet → VPC — to understand the blast radius.
Evidence collection: It checks CloudWatch metrics, recent alarm history, and instance state.
Root cause analysis: It correlates the evidence to identify the cause.
Remediation suggestions: It surfaces actionable next steps.

The left chat panel shows the agent’s reasoning as it works through each stage. You can ask follow-up questions mid-investigation.

Press enter or click to view image in full size

7.3 — Review the Investigation Results

Once complete, the investigation will show:

Root cause: High CPU utilisation on `devops-agent-demo-app` (t3 instance), detected via `CPUUtilization` metric exceeding 80% threshold
Affected resources: The EC2 instance, its associated CloudWatch alarm, and any connected networking resources
Remediation suggestions: May include scaling recommendations, instance type changes, or runbook actions depending on the context the agent has about your workload

Press enter or click to view image in full size

Step 8: See What the Agent Learned

Every investigation makes the agent smarter. Here’s what to check after the investigation completes.

8.1 — Topology Graph Updated

In the web app, click Topology.
The graph is now richer than before — the agent added relationships it discovered while traversing your infrastructure during the investigation.

Resources that were previously unconnected nodes may now have edges drawn between them, reflecting the real dependency chain the agent traced.

Press enter or click to view image in full size

8.2 — Improvements Surfaced

Click Improvements in the left sidebar.
After an investigation, the agent may surface proactive recommendations here — for example:
— Adding memory utilisation monitoring (if only CPU is currently alarmed)
— Setting up composite alarms for multi-signal incidents
— Rightsizing suggestions based on the observed utilisation pattern

These are generated by the agent based on what it observed during the investigation, not generic advice.

8.3 — Managed Skills Unlocking

Go to Knowledge → Skills in the web app.
Under Managed skills, you’ll see progress against unlockable skills:
— `tool-use-best-practices` — unlocks after 10 completed investigations
— `chat-tool-use-best-practices` — unlocks after 10 chat sessions

Each investigation brings the agent closer to unlocking these skills, which improve its reasoning and efficiency in future investigations.

8.4 — Stop the CPU Stress Test

Back on the EC2 instance, let the stress command finish (it runs for 5 minutes) or stop it manually:

1
2
# If still running, press Ctrl+C or kill the process
sudo killall stress

Within a few minutes, `CPUUtilization` will drop below 80% and the alarm will return to OK state. If you wired up an OK notification in Step 4, the Lambda will fire again and send a `resolved` event to the agent, automatically closing the investigation.

What We Accomplished

In this part, we:

Generated an Agent Space Webhook and securely stored the credentials in Secrets Manager
Created an SNS topic and connected the CloudWatch alarm to notify it when triggered
Built a Lambda function that bridges CloudWatch alarm notifications → DevOps Agent’s webhook schema, including HMAC-SHA256 signing
Simulated a real incident by spiking CPU on the EC2 instance
Watched the DevOps Agent automatically start an investigation, traverse the resource topology, perform root cause analysis, and surface remediation suggestions — all without human intervention
Observed the Topology graph grow, Improvements populate, and Managed Skills progress after the investigation

The agent is now fully operational as an autonomous incident responder for your EC2 workload.

What’s Next — Part 5

In Part 5 — Cross-Cloud Integration with Azure, we’ll extend the agent beyond AWS:

Register Azure as a secondary cloud source in the Agent Space
Connect Azure Monitor and resource groups so the agent can see Azure workloads
Run a cross-cloud investigation that spans both AWS and Azure resources
See how the topology graph expands to include Azure nodes and their relationships to AWS services

Full article link: https://medium.com/@fawazcp/setting-up-aws-devops-agent-from-scratch-simulating-and-investigating-your-first-incident-part-ea665f56f7d4?sharedUserId=fawazcp

If you found this helpful, follow along for Part 5. Questions or issues? Drop them in the comments below.

Series: Setting Up AWS DevOps Agent from Scratch (7 articles)

1
What is AWS DevOps Agent? A Beginner’s Guide to Autonomous Incident Response — PART 1
…
4
Simulating and Investigating Your First Incident (Part 4) This article
5
Cross-Cloud Integration with Azure (Part 5)

Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.

Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article

Builder Center

Simulating and Investigating Your First Incident (Part 4)

Series: Setting Up AWS DevOps Agent from Scratch (7 articles)

Fawaz c p

The Goal: True Autonomous Incident Response

Step 1: Generate the Agent Space Webhook

1.1 — Open the Webhook Wizard

1.2 — Step 1: Understand the Data Schema

1.3 — Step 2: Understand HMAC Signing

1.4 — Step 3: Generate the URL and Secret

Step 2: Store the Webhook Credentials Securely

Step 3: Create an SNS Topic

Step 4: Update the CloudWatch Alarm to Notify SNS

Step 5: Create the Lambda Bridge Function

5.1 — Create the Function

5.2 — Grant Permission to Read the Secret

5.3 — Deploy the Function Code

5.4 — Add SNS as a Trigger

Step 6: Simulate the Incident — Spike the CPU

6.1 — Connect to the Instance

6.2 — Install and Run the Stress Tool

6.3 — Watch the Alarm Fire

Step 7: Watch the Agent Investigate

7.1 — Open the Incident Response Dashboard

7.2 — Follow the Investigation in Real Time

7.3 — Review the Investigation Results

Step 8: See What the Agent Learned

8.1 — Topology Graph Updated

8.2 — Improvements Surfaced

8.3 — Managed Skills Unlocking

8.4 — Stop the CPU Stress Test

What We Accomplished

What’s Next — Part 5

Series: Setting Up AWS DevOps Agent from Scratch (7 articles)