AWS Builder Center
Builders Skill Sprint 2026 : Breaking things on purpose with AWS DevOps Agent

Builders Skill Sprint 2026 : Breaking things on purpose with AWS DevOps Agent

My complete experience completing all 5 challenges of the AWS User Group Madurai Builders Skill Sprint May 2026 — from a broken Lambda to a runaway EC2 process, with AWS DevOps Agent as the investigator.

Breaking AWS on Purpose: What I Learned from the Builders Skill Sprint — May 2026

Author: KANISKASRI T S P Community: AWS User Group Madurai Event: Builders Skill Sprint — May 2026 (DevOps Month) Tags: AWS DevOps Agent, CloudFormation, Lambda, EC2, IAM, CloudWatch



Why I Broke My Own Infrastructure

Most AWS tools tell you what is wrong — a red alarm, a failed Lambda, a spiked CPU metric. Very few tell you why, and almost none tell you what to do about it.
That's the gap AWS DevOps Agent is built to close. And the best way to evaluate it? Break things on purpose and see if it can find the root cause.
The Builders Skill Sprint — May 2026, organized by AWS User Group Madurai, did exactly that. Five challenges. Five deliberately broken CloudFormation stacks. One AI agent to investigate them all — using plain English.
Here's what I built, how I broke it, and what the agent found.



The Setup (Do This Once)

Before any challenge, you need an Agent Space — a read-only connection between AWS DevOps Agent and your AWS account.
  1. Enable AWS DevOps Agent (2-month free trial — not free tier)
  1. Create an Agent Space named  bss-may-2026  with read-only access
  1. Set a $5 Budget Alert as a safety net
  1. Make sure everything deploys to us-east-1 — that's where the agent runs
Two screens matter throughout:
  • Admin Console — where you manage the Agent Space
  • DevOps Agent Web App — where you actually chat and investigate



Challenge 1 ⭐ — Meet Your Agent

Goal: Prove the wiring works before breaking anything.
What I asked:
  • "What resources do I have in this account?"
  • "Is anything unhealthy right now?"
  • "Give me a health summary of my environment."
What the agent found: 27 tracked resources across us-east-1 and ap-south-1. It correctly separated the real workload (one Lambda function) from AWS-managed defaults (VPC, subnets, IGW, S3 template bucket). It also flagged a subtle point: the CloudWatch alarm was showing OK not because the function was healthy, but because it had never been invoked yet — "no data" and "known good" are not the same thing.
Key insight: A static dashboard would have shown green. The agent explained why it was green.



Challenge 2 ⭐⭐ — First Investigation (Lambda NameError)

The scenario: A Lambda function called  challenge2-broken-fn  fails on every single invocation. Nobody knows why.
The broken code:
def handler(event, context):
return {"result": config["value"]} # 'config' is never defined

What the agent found: It pulled the CloudWatch Logs error trace and immediately identified  NameError: name 'config' is not defined . It confirmed there were no environment variables, no imports, nothing in the event —  config  simply didn't exist anywhere.
The fix:
def handler(event, context):
config = {"value": "Hello from Challenge 2!"}
return {"result": config["value"]}

Deploy → Test → Alarm cleared ✅
Key insight: The agent found the same answer you'd get reading logs manually — but without leaving the chat.



Challenge 3 ⭐⭐⭐ — Stress & Diagnose (Runaway EC2 Processes)

The scenario: An EC2 server called  challenge3-stress  is painfully slow. The CPU alarm is red. Nobody knows what's running on it.
What was actually happening (UserData script):
#!/bin/bash
for i in $(seq $(nproc)); do
setsid bash -c 'while true; do :; done' >/dev/null 2>&1 < /dev/null &
done

One infinite busy-loop per CPU thread, launched on boot. On a t3.micro (2 vCPUs), this pinned both cores to 100% immediately — no application code involved at all.
What the agent found: Boot-time CPU saturation consistent with a UserData-launched process. It pointed straight at the cause without me having to guess.
The fix: Connected via Session Manager (no SSH keys needed — the instance role had  AmazonSSMManagedInstanceCore ) and ran:
pkill -f 'while true'

CPU dropped from 100% to ~0% instantly. Alarm cleared ✅
Key insight: Rebooting the instance would have "fixed" the alarm temporarily — but the loops would restart on next boot. The agent's diagnosis led to the right fix, not just the quick one.



Challenge 4 ⭐⭐⭐⭐ — Bad Deploy Detective (Missing IAM Permission)

The scenario: An app ( challenge4-app-fn ) that reads from DynamoDB used to work fine. After a recent deploy, every request fails — but the code looks completely correct.
The code (genuinely correct):
def handler(event, context):
resp = ddb.get_item(TableName=TABLE, Key={"id": {"S": "1"}})
return {"statusCode": 200, "body": json.dumps({"item": resp.get("Item", {})})}

The actual bug (in the IAM role, not the code): The "bad deploy" attached a role with only  AWSLambdaBasicExecutionRole  — CloudWatch Logs access only. No DynamoDB permissions. Every  get_item  call threw  AccessDeniedException  before touching the table.
What the agent found: It reasoned across the resource graph — function → role → policy → target table — and explicitly ruled out a code change. The root cause was a missing IAM policy, three hops away from the function itself.
The fix: IAM → Roles →  challenge-4-AppRole  → Attach  AmazonDynamoDBReadOnlyAccess 
No code changes. Test returned:
{"statusCode": 200, "body": "{\"item\": {\"product\": \"Builders Hoodie\", \"price\": \"49\"}}"}

Alarm cleared ✅
Key insight: This is the highest-signal test for any agentic ops tool. If it only summarizes the nearest log stream, it fails here. The agent passed.



Challenge 5 🚀 — Build Your Own (Timeout Trap)

What I built: A Lambda ( challenge5-timeout-trap ) simulating a slow external API call with  time.sleep(60)  — but configured with a 5-second function timeout.
Why it's interesting: Every invocation dies at exactly  Duration: 5000ms  with "Task timed out." This looks like a hang or infinite loop if you only read the error message. But the fixed duration is the tell — it's a configuration mismatch, not a logic bug.
What the agent found: It read the 5000ms-every-time pattern as a timeout/configuration mismatch rather than a code error, and recommended:
  1. Increase timeout to at least 2x expected call duration
  1. Add client-side timeouts for graceful error handling
  1. Add logging around slow calls for observability
The fix:
  • Increased timeout: 5 sec → 30 sec
  • Fixed sleep:  time.sleep(60)  →  time.sleep(5) 
Function completed in ~5 seconds. Alarm cleared ✅



What the Pattern Looks Like Across All Challenges

Challenge
Alarm that fired
Actual root cause
Where the fix lives
2
Lambda Errors
NameError in code
Application code
3
EC2 CPU > 70%
Runaway UserData processes
Runtime process
4
Lambda Errors
Missing IAM permission
IAM policy
5
Task timed out
Timeout too short
Lambda configuration
The throughline: the alarm that fires is rarely where the fix belongs.
An Errors alarm can mean a code bug (Ch. 2) or a permissions gap three hops away (Ch. 4). A timeout can mean an infinite loop or a misconfigured setting (Ch. 5). A CPU alarm can mean undersized hardware or a rogue process that resizing would never fix (Ch. 3).
Distinguishing these requires correlating logs, metrics, IAM, and deploy history — exactly what a chat-based agent is built to do faster than manually pivoting between Lambda Console, CloudWatch Logs Insights, and IAM.



Tips for Running This Yourself

  • Always deploy to us-east-1 — that's where DevOps Agent runs
  • Set a $5 budget alert first — before touching anything else
  • Delete each stack immediately after finishing — especially Challenge 3 (EC2 bills by the hour)
  • Give the agent a starting point — mention the alarm name in your first question
  • The "code looks fine" scenario is the best test — Challenge 4 alone tells you if an agent is doing real cross-resource reasoning



Final Thoughts

Running deliberately broken infrastructure through an AI investigator is one of the most effective ways to evaluate it — not because the breaks are exotic, but because each one lives in a different layer: code, IAM, runtime process, configuration. That spread is what exercises the agent's actual capability.
The sprint format worked well precisely because it forced the investigation workflow, not just the fix. Deploy → break → investigate → fix → verify. That loop is what incident response actually looks like, and practicing it with an AI agent before a real incident is worth the 2-hour time investment.


Submitted as part of the Builders Skill Sprint — May 2026, organized by AWS User Group Madurai (Most Active User Group, Q1 2026). All five challenges completed.
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article