
Builders Skill Sprint 2026 : Breaking things on purpose with AWS DevOps Agent
My complete experience completing all 5 challenges of the AWS User Group Madurai Builders Skill Sprint May 2026 — from a broken Lambda to a runaway EC2 process, with AWS DevOps Agent as the investigator.
Breaking AWS on Purpose: What I Learned from the Builders Skill Sprint — May 2026
Author: KANISKASRI T S P Community: AWS User Group Madurai Event: Builders Skill Sprint — May 2026 (DevOps Month) Tags: AWS DevOps Agent, CloudFormation, Lambda, EC2, IAM, CloudWatch
Why I Broke My Own Infrastructure
Most AWS tools tell you what is wrong — a red alarm, a failed Lambda, a spiked CPU metric. Very few tell you why, and almost none tell you what to do about it.
That's the gap AWS DevOps Agent is built to close. And the best way to evaluate it? Break things on purpose and see if it can find the root cause.
The Builders Skill Sprint — May 2026, organized by AWS User Group Madurai, did exactly that. Five challenges. Five deliberately broken CloudFormation stacks. One AI agent to investigate them all — using plain English.
Here's what I built, how I broke it, and what the agent found.
The Setup (Do This Once)
Before any challenge, you need an Agent Space — a read-only connection between AWS DevOps Agent and your AWS account.
- Enable AWS DevOps Agent (2-month free trial — not free tier)
- Create an Agent Space named
bss-may-2026with read-only access
- Set a $5 Budget Alert as a safety net
- Make sure everything deploys to us-east-1 — that's where the agent runs
Two screens matter throughout:
- Admin Console — where you manage the Agent Space
- DevOps Agent Web App — where you actually chat and investigate
Challenge 1 ⭐ — Meet Your Agent
Goal: Prove the wiring works before breaking anything.
What I asked:
- "What resources do I have in this account?"
- "Is anything unhealthy right now?"
- "Give me a health summary of my environment."
What the agent found: 27 tracked resources across us-east-1 and ap-south-1. It correctly separated the real workload (one Lambda function) from AWS-managed defaults (VPC, subnets, IGW, S3 template bucket). It also flagged a subtle point: the CloudWatch alarm was showing OK not because the function was healthy, but because it had never been invoked yet — "no data" and "known good" are not the same thing.
Key insight: A static dashboard would have shown green. The agent explained why it was green.
Challenge 2 ⭐⭐ — First Investigation (Lambda NameError)
The scenario: A Lambda function called
challenge2-broken-fn fails on every single invocation. Nobody knows why.The broken code:
def handler(event, context): return {"result": config["value"]} # 'config' is never defined
What the agent found: It pulled the CloudWatch Logs error trace and immediately identified
NameError: name 'config' is not defined . It confirmed there were no environment variables, no imports, nothing in the event — config simply didn't exist anywhere.The fix:
def handler(event, context): config = {"value": "Hello from Challenge 2!"} return {"result": config["value"]}
Deploy → Test → Alarm cleared ✅
Key insight: The agent found the same answer you'd get reading logs manually — but without leaving the chat.
Challenge 3 ⭐⭐⭐ — Stress & Diagnose (Runaway EC2 Processes)
The scenario: An EC2 server called
challenge3-stress is painfully slow. The CPU alarm is red. Nobody knows what's running on it.What was actually happening (UserData script):
#!/bin/bashfor i in $(seq $(nproc)); do setsid bash -c 'while true; do :; done' >/dev/null 2>&1 < /dev/null &done
One infinite busy-loop per CPU thread, launched on boot. On a t3.micro (2 vCPUs), this pinned both cores to 100% immediately — no application code involved at all.
What the agent found: Boot-time CPU saturation consistent with a UserData-launched process. It pointed straight at the cause without me having to guess.
The fix: Connected via Session Manager (no SSH keys needed — the instance role had
AmazonSSMManagedInstanceCore ) and ran:pkill -f 'while true'
CPU dropped from 100% to ~0% instantly. Alarm cleared ✅
Key insight: Rebooting the instance would have "fixed" the alarm temporarily — but the loops would restart on next boot. The agent's diagnosis led to the right fix, not just the quick one.
Challenge 4 ⭐⭐⭐⭐ — Bad Deploy Detective (Missing IAM Permission)
The scenario: An app (
challenge4-app-fn ) that reads from DynamoDB used to work fine. After a recent deploy, every request fails — but the code looks completely correct.The code (genuinely correct):
def handler(event, context): resp = ddb.get_item(TableName=TABLE, Key={"id": {"S": "1"}}) return {"statusCode": 200, "body": json.dumps({"item": resp.get("Item", {})})}
The actual bug (in the IAM role, not the code): The "bad deploy" attached a role with only
AWSLambdaBasicExecutionRole — CloudWatch Logs access only. No DynamoDB permissions. Every get_item call threw AccessDeniedException before touching the table.What the agent found: It reasoned across the resource graph — function → role → policy → target table — and explicitly ruled out a code change. The root cause was a missing IAM policy, three hops away from the function itself.
The fix: IAM → Roles →
challenge-4-AppRole → Attach AmazonDynamoDBReadOnlyAccess No code changes. Test returned:
{"statusCode": 200, "body": "{\"item\": {\"product\": \"Builders Hoodie\", \"price\": \"49\"}}"}
Alarm cleared ✅
Key insight: This is the highest-signal test for any agentic ops tool. If it only summarizes the nearest log stream, it fails here. The agent passed.
Challenge 5 🚀 — Build Your Own (Timeout Trap)
What I built: A Lambda (
challenge5-timeout-trap ) simulating a slow external API call with time.sleep(60) — but configured with a 5-second function timeout.Why it's interesting: Every invocation dies at exactly
Duration: 5000ms with "Task timed out." This looks like a hang or infinite loop if you only read the error message. But the fixed duration is the tell — it's a configuration mismatch, not a logic bug.What the agent found: It read the 5000ms-every-time pattern as a timeout/configuration mismatch rather than a code error, and recommended:
- Increase timeout to at least 2x expected call duration
- Add client-side timeouts for graceful error handling
- Add logging around slow calls for observability
The fix:
- Increased timeout: 5 sec → 30 sec
- Fixed sleep:
time.sleep(60)→time.sleep(5)
Function completed in ~5 seconds. Alarm cleared ✅
What the Pattern Looks Like Across All Challenges
Challenge | Alarm that fired | Actual root cause | Where the fix lives |
|---|---|---|---|
2 | Lambda Errors | NameError in code | Application code |
3 | EC2 CPU > 70% | Runaway UserData processes | Runtime process |
4 | Lambda Errors | Missing IAM permission | IAM policy |
5 | Task timed out | Timeout too short | Lambda configuration |
The throughline: the alarm that fires is rarely where the fix belongs.
An Errors alarm can mean a code bug (Ch. 2) or a permissions gap three hops away (Ch. 4). A timeout can mean an infinite loop or a misconfigured setting (Ch. 5). A CPU alarm can mean undersized hardware or a rogue process that resizing would never fix (Ch. 3).
Distinguishing these requires correlating logs, metrics, IAM, and deploy history — exactly what a chat-based agent is built to do faster than manually pivoting between Lambda Console, CloudWatch Logs Insights, and IAM.
Tips for Running This Yourself
- Always deploy to us-east-1 — that's where DevOps Agent runs
- Set a $5 budget alert first — before touching anything else
- Delete each stack immediately after finishing — especially Challenge 3 (EC2 bills by the hour)
- Give the agent a starting point — mention the alarm name in your first question
- The "code looks fine" scenario is the best test — Challenge 4 alone tells you if an agent is doing real cross-resource reasoning
Final Thoughts
Running deliberately broken infrastructure through an AI investigator is one of the most effective ways to evaluate it — not because the breaks are exotic, but because each one lives in a different layer: code, IAM, runtime process, configuration. That spread is what exercises the agent's actual capability.
The sprint format worked well precisely because it forced the investigation workflow, not just the fix. Deploy → break → investigate → fix → verify. That loop is what incident response actually looks like, and practicing it with an AI agent before a real incident is worth the 2-hour time investment.
Submitted as part of the Builders Skill Sprint — May 2026, organized by AWS User Group Madurai (Most Active User Group, Q1 2026). All five challenges completed.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article