How AWS DevOps Agent Gave a 6-Person Cloud Team Their Nights Back

A 6-person cloud ops team was drowning in 2 AM pages for 15+ dev teams. I helped them deploy AWS DevOps Agent to automate root cause analysis, cut MTTR by 70%, and shift dev teams from bystanders to self-service investigators.

Sandeep Nalam

Published Jun 18, 2026

Last modified Jun 18, 2026

The Problem We Were Solving

Here's a scenario I see often as a Technical Account Manager: a small, highly capable cloud engineering team responsible for the operational health of an entire organization's AWS footprint. In this case, a 6-person cloud operations team supporting over 15 development teams — monitoring business-critical applications 24/7, fielding every incident, and still expected to deliver infrastructure improvements.

Every on-call rotation looked the same: an alert fires at 2 AM, one of six cloud engineers starts hopping between CloudWatch, Splunk, X-Ray, deployment history, and Slack threads. Meanwhile, the dev team that owns the service is asleep — because the cloud team is the only group with the operational tooling and context to investigate.

I wanted to help them break that pattern. What if every dev team could get autonomous root cause analysis the moment an incident was created — without needing deep operational expertise? What if the cloud team could shift from reactive firefighting to proactive architecture improvements?

That's what we built with AWS DevOps Agent. Let me walk you through how.

The Real-World Incident That Changed the Conversation

The 2 AM Page That Started It All

Let me tell you about the incident that crystallized the problem for this customer.

It was a Wednesday at 2:14 AM. The cloud team's on-call engineer received a PagerDuty alert: "5xx error rate exceeded 15% on the order processing API — P1 severity". He opened his laptop, still half asleep, and started the familiar ritual.

First, CloudWatch. Error rate was spiking — 5xx responses climbing from baseline 0.2% to 18% in under three minutes. Then CloudWatch Logs, scanning for exceptions. Then X-Ray to trace the failing requests. Then the deployment history to see what changed. Then Slack to ping the dev team that owns the order service — who were all asleep.

45 minutes in, the cloud engineer still didn't have root cause. And he didn't own this service's codebase. He was cross-referencing infrastructure metrics against application behavior he wasn't deeply familiar with.

After an hour and twenty minutes, he finally traced it: a deployment two hours earlier by the order service dev team had changed a database connection pool configuration — reducing the maximum connections from 50 to 20 as part of a "resource optimization" effort. Under normal load, this was fine. But when the nightly batch reconciliation job kicked off at 2 AM (something the dev team didn't account for), the reduced pool couldn't handle concurrent request volume. Connections queued, timeouts cascaded, and 5xx errors spread across three downstream microservices.

The root cause was a well-intentioned config change by a dev team + a timing coincidence that only the cloud team's operational context could have surfaced. No single dashboard showed this. The cloud engineer had to mentally stitch together: deployment history (2 hours prior) + RDS connection metrics (pool saturation) + application logs (timeout errors) + the fact that batch reconciliation runs at 2 AM nightly.

Total time to resolution: 1 hour 52 minutes.

And here's the kicker — the cloud team had seen a nearly identical pattern six weeks earlier with a different dev team's service. But at 2 AM, operating on a service they don't own, that institutional knowledge didn't surface.

Running the Same Incident Through DevOps Agent

After we set up DevOps Agent with Agent Spaces scoped per application, we replayed a similar scenario — this time with the dev team receiving the investigation results directly. Here's what happened differently:

0:00 — CloudWatch alarm fires. EventBridge routes it to the webhook Lambda.

0:02 — DevOps Agent receives the incident, begins autonomous investigation.

0:04 — Agent identifies the error spike and starts pulling metrics from CloudWatch, logs from Splunk, and traces from X-Ray simultaneously. Not sequentially like a single engineer would — simultaneously.

0:07 — Agent checks GitHub deployment history. Finds the connection pool config change from 2 hours earlier. Flags it as a candidate root cause.

0:09 — Agent correlates the timing: deployment + nightly batch reconciliation schedule + connection pool saturation pattern. Identifies the causal chain.

0:11 — Root cause posted to the dev team's #order-service-alerts Slack channel:

1
2
3
4
5
6
7
8
9
10
🔍 Investigation Complete

Root cause: Config change in deployment v3.8.2 (commit 7a2f1e9) reduced
max DB connections from 50 → 20. Under normal load: no impact. Combined
with nightly batch reconciliation (started 02:00 UTC), concurrent request
volume exceeds pool capacity → connection queuing → timeout cascade →
5xx errors across order-service, payment-service, notification-service.

This matches a similar pattern from 6 weeks ago (Investigation #INV-0019)
involving inventory-service connection pool changes.

11 minutes. Not 1 hour 52 minutes. The dev team woke up to a root cause and a fix plan — not an active fire. And the cloud team's on-call engineer? Never paged.

What AWS DevOps Agent Actually Is

AWS DevOps Agent is an autonomous frontier agent — not a chatbot, not a runbook runner. It acts as an always-available SRE teammate that:

Investigates incidents autonomously — triggered from ticketing systems, alarms, or chat
Correlates across your entire stack — observability tools, code repositories, CI/CD pipelines, and infrastructure
Identifies root cause — across system changes, input anomalies, resource limits, component failures, and dependency issues
Generates mitigation plans — structured 4-phase plans (Prepare → Pre-Validate → Apply → Post-Validate)
Produces agent-ready specs — instructions a coding agent like Kiro can implement directly
Learns from past incidents — creates learned investigation skills that make future investigations faster
Empowers dev teams — gives application owners the operational insight to investigate their own services without deep infrastructure expertise

It supports AWS, multicloud, and on-premises environments.

Architecture: The End-to-End Pipeline

Here's the architecture we implemented. Three components working together:

The flow of the events for the Incidents and RCA

Flow summary:

Incident Source — CloudWatch Alarm fires or ticketing system creates an incident
Event Routing — EventBridge matches the event, triggers a Lambda function that signs the payload with HMAC and POSTs to the DevOps Agent webhook
DevOps Agent Space — The agent investigates autonomously, pulling from CloudWatch metrics/logs, Splunk (via MCP), GitHub deployments, and X-Ray traces simultaneously
Agent Pipeline — Investigate → Root Cause Analysis → Mitigation Plan → Agent-Ready Spec
Outputs — RCA posted to the dev team's Slack channel, ticket updated with structured findings, and fix spec handed to Kiro for implementation

📁 Full CloudFormation + CDK templates: github.com/NSandeep12/aws-devops-agent

Step 1: Create a DevOps Agent Space

A Space defines what the agent has access to — your AWS accounts, third-party integrations, and permissions. Think of it as a scoped identity for the agent.

1
2
3
4
# Via AWS CLI
aws devopsagent create-agent-space \
  --name "OrderService-Agent" \
  --description "Autonomous SRE for the order processing service"

Or through the console: navigate to AWS DevOps Agent → Create Agent Space.

Naming convention we used: {service-name}-Agent — one Space per dev team's application, so each team gets scoped visibility into their own services.

📖 Detailed guide: Creating an Agent Space

Step 2: Configure the Webhook (HMAC-Secured)

The webhook is how incidents trigger investigations. AWS DevOps Agent uses HMAC (Hash-based Message Authentication Code) for webhook security — not basic auth, not API keys.

Webhook payload schema:

1
2
3
4
5
6
7
8
9
10
11
{
  eventType: 'incident';
  incidentId: string;
  action: 'created' | 'updated' | 'closed' | 'resolved';
  priority: 'CRITICAL' | 'HIGH' | 'MEDIUM' | 'LOW' | 'MINIMAL';
  title: string;
  description?: string;
  timestamp?: string;
  service?: string;
  data?: object;  // Original event attached here
}

HMAC signature implementation (Node.js):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
import { createHmac } from "node:crypto";

function sendEventToWebhook(webhookUrl, secret, incidentPayload) {
  const payload = {
    eventType: "incident",
    incidentId: incidentPayload.ticketKey,
    action: "created",
    priority: mapPriority(incidentPayload.priority),
    title: incidentPayload.summary,
    description: incidentPayload.description,
    timestamp: new Date().toISOString(),
    service: incidentPayload.affectedService,
    data: incidentPayload.rawEvent
  };

  const timestamp = new Date().toISOString();
  const hmac = createHmac("sha256", secret);
  hmac.update(`${timestamp}:${JSON.stringify(payload)}`, "utf8");
  const signature = hmac.digest("base64");

  return fetch(webhookUrl, {
    method: "POST",
    headers: {
      "Content-Type": "application/json",
      "x-amzn-event-timestamp": timestamp,
      "x-amzn-event-signature": signature,
    },
    body: JSON.stringify(payload),
  });
}

Critical: When you create the webhook in the console, download and save the URL and secret immediately. You won't be able to see the secret again after leaving that screen.

📁 Webhook handler code: src/webhook-handler/ in the GitHub repo

Step 3: Build the Auto-Trigger Pipeline

I used CloudWatch Alarm → EventBridge → Lambda → DevOps Agent Webhook. This is the exact pattern from the official architecture.

Lambda function (Python 3.12):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
import json
import hmac
import hashlib
import base64
import os
from datetime import datetime
import urllib3

WEBHOOK_URL = os.environ['DEVOPS_AGENT_WEBHOOK_URL']
WEBHOOK_SECRET = os.environ['DEVOPS_AGENT_WEBHOOK_SECRET']

http = urllib3.PoolManager()

def handler(event, context):
    """Triggered by EventBridge rule matching alarm state change."""

    detail = event.get('detail', {})

    payload = {
        "eventType": "incident",
        "incidentId": detail.get('issue_key', f"INC-{context.aws_request_id[:8]}"),
        "action": "created",
        "priority": map_priority(detail.get('priority', 'HIGH')),
        "title": detail.get('summary', 'Automated investigation triggered'),
        "description": detail.get('description', ''),
        "timestamp": datetime.utcnow().isoformat() + 'Z',
        "service": detail.get('affected_service'),
        "data": detail
    }

    # HMAC signature
    timestamp = datetime.utcnow().isoformat() + 'Z'
    message = f"{timestamp}:{json.dumps(payload)}"
    signature = base64.b64encode(
        hmac.new(
            WEBHOOK_SECRET.encode('utf-8'),
            message.encode('utf-8'),
            hashlib.sha256
        ).digest()
    ).decode('utf-8')

    response = http.request(
        'POST',
        WEBHOOK_URL,
        body=json.dumps(payload),
        headers={
            'Content-Type': 'application/json',
            'x-amzn-event-timestamp': timestamp,
            'x-amzn-event-signature': signature,
        }
    )

    return {
        'statusCode': response.status,
        'investigationTriggered': response.status == 200
    }

def map_priority(priority):
    mapping = {
        'P1': 'CRITICAL', 'Highest': 'CRITICAL',
        'P2': 'HIGH', 'High': 'HIGH',
        'P3': 'MEDIUM', 'Medium': 'MEDIUM',
        'P4': 'LOW', 'Low': 'LOW',
    }
    return mapping.get(priority, 'MEDIUM')

EventBridge Rule (CloudFormation):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
Resources:
  IncidentTriggerRule:
    Type: AWS::Events::Rule
    Properties:
      Name: devops-agent-incident-trigger
      EventBusName: !Ref IncidentEventBus
      EventPattern:
        source:
          - "custom.ticketing"
          - "aws.cloudwatch"
        detail-type:
          - "Incident Created"
          - "CloudWatch Alarm State Change"
        detail:
          state:
            value:
              - "ALARM"
      Targets:
        - Id: InvestigationLambda
          Arn: !GetAtt WebhookLambda.Arn

  WebhookLambda:
    Type: AWS::Lambda::Function
    Properties:
      FunctionName: devops-agent-webhook-trigger
      Runtime: python3.12
      Handler: index.handler
      Timeout: 30
      Role: !GetAtt WebhookLambdaRole.Arn
      Environment:
        Variables:
          DEVOPS_AGENT_WEBHOOK_URL: !Sub "{{resolve:secretsmanager:devops-agent-webhook:SecretString:url}}"
          DEVOPS_AGENT_WEBHOOK_SECRET: !Sub "{{resolve:secretsmanager:devops-agent-webhook:SecretString:secret}}"

  WebhookLambdaRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: "2012-10-17"
        Statement:
          - Effect: Allow
            Principal:
              Service: lambda.amazonaws.com
            Action: sts:AssumeRole
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
      Policies:
        - PolicyName: SecretsAccess
          PolicyDocument:
            Version: "2012-10-17"
            Statement:
              - Effect: Allow
                Action:
                  - secretsmanager:GetSecretValue
                Resource: !Ref WebhookSecret

  WebhookSecret:
    Type: AWS::SecretsManager::Secret
    Properties:
      Name: devops-agent-webhook
      Description: "DevOps Agent webhook URL and HMAC secret"

📁 Full stack: infra/trigger-stack.yaml in the GitHub repo

Step 4: Connect Your Telemetry Sources

This is where the agent's power comes from — it correlates across multiple data sources simultaneously.

Built-in integrations I connected:

Integration	What It Provides	Setup
Amazon CloudWatch	Metrics, alarms, Log Insights	Native — just add AWS account to Space
Splunk	Centralized logs via MCP Server	Splunk MCP app + bearer token
GitHub	Commit history, deployments, code context	OAuth app + repo selection
Slack	Real-time notifications to SRE channel	Workspace-level app install

Splunk MCP Integration:

Splunk integration requires enabling the Splunk MCP Server app:

Install Splunk MCP Server from Splunkbase
Create a role called mcp_user (no capabilities needed)
Generate a bearer token with audience set to mcp
Note the MCP endpoint URL (port 8089)
Add the MCP config to your DevOps Agent Space

GitHub Integration:

Two-step process:

Register at account level: Agent Space → Capabilities → Pipeline → GitHub → OAuth flow
Select specific repositories the agent can access (read-only + deployment events)

The agent uses GitHub to correlate code changes with operational incidents — temporal relationship between a deployment and the start of errors.

Step 5: Define Investigation Skills

This is the part that makes the agent smarter over time. Skills are Markdown-based instructions that guide how the agent investigates. They follow the open Agent Skills specification .

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
# Production App Investigation Skill

## Telemetry Routing
- For application errors → check Splunk logs first (index: production-app)
- For infrastructure metrics → check CloudWatch (namespace: ProductionApp)
- For serverless function logs → check CloudWatch Logs (/aws/lambda/*)

## Known Failure Patterns
- Connection pool exhaustion: check RDS connection count + app error logs
- Memory pressure: check container memory utilization + OOM killer events
- Deployment correlation: always check GitHub deployments in the 2-hour window
  before incident

## Escalation Criteria
- If root cause involves IAM or networking → flag for human review
- If blast radius > 3 services → escalate priority to CRITICAL

Upload via console (Skills → Add skill → Create skill) or as a directory structure:

1
2
3
4
5
6
7
my-investigation-skill/
├── SKILL.md              # Main instructions
├── runbooks/
│   ├── database.md       # DB-specific investigation steps
│   └── networking.md     # Network failure patterns
└── architecture/
    └── service-map.md    # How services connect

The agent also creates learned investigation skills automatically — it reviews past investigations and learns how to triage events faster next time. You don't teach it. It teaches itself.

Step 6: What the Agent Produces

When the investigation completes, the agent generates three outputs:

Root Cause Analysis

Posted to Slack and available in the operator console:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
🔍 ROOT CAUSE IDENTIFIED

Summary: Deployment v2.3.1 (commit abc123f) introduced a connection pool
exhaustion bug. The new database query in getUserPreferences() opens
connections without properly releasing them under timeout conditions.

Evidence:
- RDS ActiveConnections spiked from 45 → 200 at 01:47 UTC
- Error rate increased from 0.1% → 12.3% starting 01:49 UTC
- Deployment v2.3.1 completed at 01:45 UTC (2 min before spike)
- Splunk logs show "pool timeout" errors correlating with the spike

Causal chain:
  Deploy v2.3.1 → connection leak under timeout → pool exhaustion
  → request queuing → 5xx errors → CloudWatch alarm

Mitigation Plan (4 Phases)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
MITIGATION PLAN

Phase 1: PREPARE
  - Verify rollback target: v2.3.0 (last known good)
  - Confirm deployment pipeline access
  - Notify on-call team in #sre-incidents

Phase 2: PRE-VALIDATE
  - Check current error rate (expected: elevated)
  - Confirm v2.3.0 artifact exists in CodeDeploy
  - Verify no other deployments in progress

Phase 3: APPLY
  - Execute rollback to v2.3.0 via CodeDeploy
  - Monitor deployment progress (target: < 5 min)

Phase 4: POST-VALIDATE
  - Verify error rate returns to baseline (< 0.5%) within 10 min
  - Confirm RDS ActiveConnections drops below 60
  - Verify no new alarms triggered
  - If validation fails → escalate to on-call lead

Agent-Ready Spec (for Coding Agent Handoff)

The agent produces a spec that a coding agent like Kiro can implement directly:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
## Fix: Connection Pool Leak in getUserPreferences

### Problem
The getUserPreferences() function opens a database connection but does
not release it when the query times out.

### Required Change
Wrap the query in a try/finally block that always calls connection.release().

### Files to Modify
- src/services/user.ts (line 142-158)

### Success Criteria
- All existing tests pass
- New test: verify connection released on timeout
- Connection pool usage stays under 50 during load test

### Rollback
Revert to commit previous to abc123f if tests fail.

You hand this spec directly to Kiro, and the fix gets implemented without manually translating mitigation steps into code changes.

Step 7: Close the Loop — Update Your Ticket

Once findings are ready, a Lambda function posts the RCA back to the ticketing system:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
def post_findings_to_ticket(investigation_id, ticket_key):
    """
    Retrieve investigation results and post to ticketing system.
    CLI alternative:
    - aws devopsagent create-backlog-task (trigger investigation)
    - aws devopsagent update-backlog-task (approve mitigation plans)
    """
    client = boto3.client('devopsagent')

    investigation = client.get_investigation(
        investigationId=investigation_id
    )

    root_cause = investigation.get('rootCause', {})
    mitigation = investigation.get('mitigationPlan', {})

    comment = format_rca_comment(root_cause, mitigation)

    ticket_client.add_comment(ticket_key, comment)
    ticket_client.transition(ticket_key, status="RCA Complete")

    return {'updated': True, 'ticket': ticket_key}

If you use Slack as your coordination point, the agent posts directly to your channel — on-call engineers wake up to a root cause, not an active incident.

How This Changed Their Operations

The investigation output didn't just solve incidents — it transformed how the entire organization operates.

Before: Cloud Team as Bottleneck

The operational model looked like this:

1
2
3
Alert fires → Cloud team investigates → Cloud team finds root cause
→ Cloud team tells dev team what to fix → Dev team fixes
→ Cloud team validates

Six cloud engineers. Fifteen dev teams. Every incident funneled through the same bottleneck. The cloud team spent 60-70% of their time on reactive incident work, leaving little capacity for infrastructure improvements, cost optimization, or architecture evolution.

Their runbook for "5xx spike" was a static checklist:

1
2
3
4
5
1. Check CloudWatch error metrics
2. Check recent deployments
3. Check application logs for exceptions
4. Check downstream dependencies
5. Escalate if not resolved in 30 min

Five vague steps that assumed deep cross-service context. Useful for the cloud team on a good day. Useless for a dev team trying to self-serve at 2 AM.

After: Dev Teams Self-Serve, Cloud Team Focuses on Strategy

Instead of writing better runbooks for humans to follow manually, the cloud team encodes their operational knowledge as Investigation Skills that the agent uses autonomously — and shares results directly with dev teams:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
# Production API Investigation Skill

## Known Failure Patterns

### Pattern: Connection Pool Exhaustion
Trigger: 5xx spike + RDS ActiveConnections > 150
Correlation window: Check deployments in prior 4 hours
Timing factor: Batch processing runs 02:00-03:00 UTC daily
Historical: See Investigation #INV-0034, #INV-0052

### Pattern: Downstream Dependency Timeout
Trigger: P99 latency > 2000ms + upstream retry storms
Check: X-Ray service map for fault segments
Common cause: Third-party API degradation or DNS resolution delays

### Pattern: Memory Pressure Cascade
Trigger: Container restarts + OOMKilled events
Check: ECS task memory utilization in 15-min window before restart
Common cause: Memory leak in long-running connections after deployment

Now when a similar incident happens:

The agent already knows to check the batch reconciliation schedule
It already knows to correlate deployments in a 4-hour window (not just 30 minutes)
It references past investigations with similar patterns
It gets faster every time through learned investigation skills
Dev teams get the RCA directly — they don't wait for the cloud team to investigate

The Compound Effect

Week	What Improved
Week 1-2	Basic setup. Agent investigates, cloud team validates. Still the final check.
Week 3-4	Cloud team started writing Skills based on patterns the agent surfaced. Investigations got more targeted.
Month 2	Agent created its first learned skill — it reviewed 8 past investigations involving deployment rollbacks and learned to check canary metrics first. Nobody taught it that.
Month 3	Cloud team on-call pages dropped by 55%. Not because incidents stopped — because the agent resolved them and routed findings to dev teams before they escalated to the cloud team.
Month 4	Three dev teams started writing their own Investigation Skills for service-specific patterns. The cloud team finally had bandwidth for their deferred infrastructure modernization work.

Results

Metric	Before	After
Time to first context	30-45 min (cloud team investigates)	< 3 min (agent posts to dev team Slack)
Manual tool switching	4-6 consoles	0 (agent correlates everything)
Incidents with documented RCA	~40% (only when cloud team had bandwidth)	100% (auto-generated)
MTTR (P1 incidents)	1 hr 50 min avg	35 min
Cloud team on-call pages	~18/week	~8/week (55% reduction)
Dev team self-service resolution	~10% of incidents	~65% of incidents
Cloud team time on reactive work	60-70%	~25%
Fix implementation time	Hours (manual handoff)	Minutes (agent-ready spec → Kiro)
Runbook accuracy	Static, went stale	Self-improving via learned skills

The Full Repository

🔗 GitHub: github.com/NSandeep12/aws-devops-agent

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
├── infra/
│   ├── trigger-stack.yaml          # EventBridge + Lambda + Secrets
│   ├── agent-space-cdk/            # CDK construct for Agent Space
│   └── iam-policies/               # Minimum IAM for webhook Lambda
├── src/
│   ├── webhook-handler/            # Lambda: formats + signs + sends
│   ├── findings-handler/           # Lambda: posts RCA back to ticket
│   └── priority-mapper/            # Maps ticket priority → DevOps Agent priority
├── skills/
│   ├── production-app/             # Investigation skill for main app
│   └── data-pipeline/              # Investigation skill for ETL jobs
├── scripts/
│   ├── create-webhook.sh           # CLI script to create webhook + save creds
│   ├── test-webhook.sh             # Force an alarm state and verify
│   └── setup-splunk-mcp.sh         # Splunk MCP server setup
├── docs/
│   ├── architecture.md             # Full architecture walkthrough
│   ├── troubleshooting.md          # Common setup issues
│   └── customizing-skills.md       # How to write investigation skills
└── examples/
    ├── sample-webhook-payloads/    # Test payloads for each trigger type
    ├── sample-mitigation-plan.md   # What the agent outputs
    └── sample-agent-ready-spec.md  # Spec handed to Kiro

My Advice If You're Starting

Start with your most common incident type. Don't try to cover everything. Pick the alert that pages your team most often.
Scope Agent Spaces per application. Give each dev team their own Space with access to their services. The cloud team maintains shared Skills across Spaces.
Write Skills from real incidents. After every investigation, ask: "What did the cloud team know that the agent didn't? How do we encode that?" Add it to a Skill.
Let learned skills accumulate. After 10-15 investigations, the agent starts recognizing patterns nobody explicitly taught it. That's when the compound effect kicks in.
Trust but verify. For the first month, have the cloud team validate every RCA the agent produces. Once you see consistent accuracy, let it post directly to dev team channels without review.
Encode timing and context, not just steps. The difference between a good Skill and a great one: "check deployments" vs. "check deployments in a 4-hour window, cross-reference with the batch reconciliation schedule that runs 02:00-03:00 UTC daily."
Frame it as empowerment, not replacement. Dev teams aren't being asked to do more ops work — they're getting better tools to own their services. The cloud team isn't being made redundant — they're being freed to do higher-value work.

Key Resources

What's Next

We're building on this with:

Learned investigation skills — letting the agent review its own past investigations to get faster automatically
Agent-ready spec → Kiro pipeline — auto-creating PRs from mitigation plans, closing the loop from incident to code fix without cloud team involvement
Multi-account Spaces — separate Agent Spaces per dev team with shared Skills maintained by the cloud team
Proactive prevention — using the agent's recommendation engine to identify observability gaps and infrastructure improvements before they cause incidents
Onboarding new dev teams — rolling DevOps Agent out to the remaining dev teams, with the first three teams writing their own Skills as templates

Sandeep Nalam is a Technical Account Manager at AWS Enterprise Support who works closely with customers to accelerate their growth, strengthen their operations, and advocate for their success through every engagement.

# devops # ai-agents # sre # amazon-cloudwatch # incident-response

Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.

Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article

Builder Center

How AWS DevOps Agent Gave a 6-Person Cloud Team Their Nights Back

Sandeep Nalam

The Problem We Were Solving

The Real-World Incident That Changed the Conversation

The 2 AM Page That Started It All

Running the Same Incident Through DevOps Agent

What AWS DevOps Agent Actually Is

Architecture: The End-to-End Pipeline

Step 1: Create a DevOps Agent Space

Step 2: Configure the Webhook (HMAC-Secured)

Step 3: Build the Auto-Trigger Pipeline

Step 4: Connect Your Telemetry Sources

Step 5: Define Investigation Skills

Step 6: What the Agent Produces

Root Cause Analysis

Mitigation Plan (4 Phases)

Agent-Ready Spec (for Coding Agent Handoff)

Step 7: Close the Loop — Update Your Ticket

How This Changed Their Operations

Before: Cloud Team as Bottleneck

After: Dev Teams Self-Serve, Cloud Team Focuses on Strategy

The Compound Effect

Results

The Full Repository

My Advice If You're Starting

Key Resources

What's Next