AWS Builder Center
Building Production-Ready Skills for AWS DevOps Agent

Building Production-Ready Skills for AWS DevOps Agent

In this post, we walk through what skills are, how to convert your existing runbooks into them, how to test them systematically, and how to get them into the agent today. The approach is practical and grounded in real results. 

 This Post was co-authored by  Yeremy  and  Nirmal Rajan 

A production database is showing connection exhaustion, and your on-call engineer has twenty minutes before customers start to feel it. The runbook exists, buried four clicks deep in Confluence, but reading it under pressure is the difference between a brief blip and an outage. What if the agent investigating the alarm already knew the runbook, executed it consistently every time, and got faster with every incident? That is what  skills  do for AWS DevOps Agent. 

Agent skills  

A skill lives in a directory with a single required file, SKILL.md, and optional references/ and assets/ subdirectories for architecture patterns, procedures, and diagrams. The SKILL.md file opens with YAML frontmatter that tells the agent what the skill is for and when to use it: 
my-skill/
├── SKILL.md # Main instructions (required)
├── references/ # Optional: architecture patterns, procedures
└── assets/ # Optional: diagrams, data files
--- 
name: rds-performance-investigation 
description: Investigation procedures for RDS performance issues including 
  connection exhaustion, slow queries, and replication lag. Use when 
  investigating database latency, connection errors, or read/write 
  performance degradation. 
--- 
 The description field is the single most important line in the entire skill. It is how the agent decides whether to load your skill in the first place. A vague description like "RDS skill" will get skipped. A specific one that names the scenarios, error types, and symptoms will get picked up consistently. Treat the description the way you would treat a well-written Jira title: if someone reads only this one line, they should know exactly when to reach for it. 

Converting your runbooks into skills 

Most operations teams already have runbooks or procedures for handling incidents, performing maintenance, or investigating issues — scattered across wikis, Confluence pages, and Docs. These are the key inputs for your first skills. 
The difference between a runbook and a skill is the audience. A runbook is written for a human who can interpret ambiguity, skip steps that are obviously irrelevant, and fill in missing context from memory. A skill is written for an agent that needs explicit triggers, explicit tool invocations, and explicit decision logic, because it cannot read between the lines. 

Step 1: Audit before you convert 

Not every runbook deserves to become a skill. Prioritise the procedures that: 
  • Fire frequently — if your team reached for a runbook 23 times last quarter, that is a high-value target 
  • Are well-documented with clear success criteria 
  • Touch AWS services the agent can reach 
  • Have measurable outcomes (e.g., connection count drops, error rate returns to baseline) 
A runbook used once a year for a scheduled DR test is not a good first candidate. 

Step 2: Restructure for agent consumption 

eg : Runbook for RDS connection count investigation 
 RDS High Connection Count When you get paged for high database connections:  
1. Check CloudWatch — look at the DatabaseConnections metric  
2. If it's near the limit, check for connection leaks  
3. Look at recent deployments — did we change pool settings?  
4. Check application logs for connection errors  
5. If needed, increase max_connectionsre is a typical runbook for high RDS connection counts 
It reads fine to an engineer on call, but every step is vague to an agent. "Near the limit" is a judgement call. "If needed" is a decision with no threshold. "Check for connection leaks" is a whole subroutine implied in four words. 

Four Changes that turn a runbook into an effective skill: 

Change
What it Means
Example
Add thresholds 
Replace vague judgements with explicit numbers 
"Near the limit" → "> 80% of max_connections"
Name the tools 
Tell the agent exactly which API or metric to query 
"Check CloudWatch" → "Query CloudWatch DatabaseConnections metric, Maximum statistic, 2-hour window" 
Add decision points 
Make branching logic explicit with thresholds and actions 
"If needed" → "< 70%: investigate other causes | 70–90%: connection leak analysis | > 90%: immediate mitigation" 
Define outputs 
Specify what the agent should produce at each step 


RDS Connection Exhaustion Investigation skill
---
name: rds-connection-exhaustion
description: Systematic investigation for RDS connection exhaustion.
Use when DatabaseConnections alarm triggers or when investigating
"too many connections" errors, connection timeouts, or database
performance degradation during traffic spikes.
---
# RDS Connection Exhaustion Investigation
## When to use this skill
- CloudWatch alarm: DatabaseConnections > 80% of max_connections
- Application errors: "too many connections" or connection timeouts
- Database performance degradation during traffic spikes
## Step 1: Retrieve current connection metrics
Query CloudWatch for the past 2 hours:
- Metric: DatabaseConnections
- Statistic: Maximum
- Compare against the max_connections parameter
Decision point:
- < 70% of max_connections → investigate other causes
- 70–90% → proceed to Step 2 (connection leak analysis)
- > 90% → proceed to Step 3 (immediate mitigation)
## Step 2: Analyze connection patterns
Check recent deployments in the past 24 hours for commits affecting
database connection code or pool configuration.
## Step 3: Identify root cause
Common patterns: pool misconfiguration, connection leaks,
traffic spikes, batch jobs.
## Step 4: Recommend mitigation
Immediate (< 5 min): restart app to reset connections.
Short-term (< 1 hr): increase max_connections or add RDS Proxy.
Long-term: implement connection pool monitoring.

Step 3: Extend with integrations 

Skills become even more powerful when they reference  custom MCP  (Model Context Protocol) servers. These let the agent call internal tools — deployment trackers, change management systems, custom observability platforms — as part of an investigation. 
Two common integration patterns: 
  • Deployment tracker: Query your CI/CD system (e.g., CodePipeline, Jenkins, Spinnaker) to correlate incidents with recent deployments. The skill can instruct the agent to check for deployments in the last N hours and flag any that touched the affected service. 
  • Custom observability: If your team uses Dynatrace, Datadog, or a custom metrics platform alongside CloudWatch, an MCP server lets the agent pull metrics from both sources and correlate them in a single investigation. 

Creating skills from existing investigations 

 Converting runbooks manually is the foundational approach, but there is a faster path: let the agent generate skills from its own investigations. 
When DevOps Agent investigates an incident, it builds a detailed journal of every step it took — what it queried, what it found, what it concluded. If the agent handled the incident well, you can ask it to analyse that investigation for repeatable patterns and produce a skill automatically. 

The workflow

  1. Identify a successful investigation — one where the agent found the root cause and the resolution was correct. 
  1. Ask the agent to analyse the investigation for common patterns. It will identify the repeatable workflow: which metrics it queried, which tools it called, what decision logic it applied. 


Figure 1: Asking the agent to analyse an investigation for common patterns 
3.Ask the agent to create the skill. It will generate the SKILL.md with frontmatter, step-by-step instructions, and decision points based on what it actually did. 


Figure 2: The agent creates a reusable skill from the investigation 
4.Assign to the right agent type. New skills default to the Generic Agent type. For targeted impact, assign to a specific agent type (see Deploying Skills below). 
5.Activate the Skill. The agent will ask for confirmation to activate the skill, once the skill is ready, just confirm with the agent to activate it. The skill can be activated manually in the Skills page using the toggle option. 


Figure 3: Activating the generated skill 

Testing and evaluating your skills 

Skills impact agent behaviour the same way prompts impact LLM outputs: sometimes dramatically, sometimes subtly, and often in ways you did not expect. You can spend an afternoon crafting a detailed skill only to discover that,
  • The agent never invokes it because the description was too vague 
  • It invokes the skill but gets confused by ambiguous instructions 
  • Investigations actually run slower because the skill loaded too much irrelevant content 
  • The skill works brilliantly for three scenarios and falls apart on the fourth 
The antidote is a systematic evaluation approach that validates whether your skill is genuinely improving mean time to resolution. 

Define clear scenarios 

Start with scenarios grounded in real failures you have actually seen. Keep them constrained: specify the setup, the symptoms, and what "solved" looks like. A good scenario reads a concise incident report with an answer key. 
Example: RDS connection-exhaustion scenario 
Setup: db.m5.large at 950 of 1,000 max connections.  
A connection-pool configuration change was deployed two hours ago.  
Application logs show "too many connections" errors.  
Success criteria: • Agent invokes the rds-connection-exhaustion skill in its first few steps • Queries CloudWatch for DatabaseConnections • Correlates with the recent deployment • Identifies the connection-pool misconfiguration • Proposes a mitigation in under 10 minutes 

Establish metrics you can measure 

 AWS DevOps Agent surfaces skill invocation in two places: the investigation journal shows which skills were loaded during an autonomous investigation, and the chat interface shows when a skill is used during a conversational session. 
To turn that into a metric, run each scenario multiple times and track the following 
Metric
What it measures
How to capture it
Invocation rate
Did the skill load when it should have?
Count skill appearances in investigation journal across N runs
Effectiveness
Did the investigation succeed, and how long did it take?
Track success rate and MTTR per scenario
Baseline comparison
Is the skill better than no skill at all?
Run the same scenario with no skills attached as a control

Run comparative tests and iterate 

 Run your scenarios against three configurations: no skill (baseline), first draft, and refined version. The iteration pattern is almost always the same: 
Configuration
Success rate
Avg MTTR
Notes
No skill (baseline)
~20%
~45 min
Agent relies on general knowledge only
First draft
~40%
~28 min
Description needs tightening; missing decision points
Refined version
>80%
<12 min
Tightened description, explicit decision logic
 The iteration loop is what gets you there: run the tests, identify where the agent hesitates or goes off track, and refine the specific section of the SKILL.md responsible for that step. Markdown headers make A/B testing individual sections easy. 


Use the on-demand chat interface 

You can use the on-demand chat to analyze skill invocation details in real time. Ask the agent to walk through an investigation step by step and observe where the skill is loaded, what guidance it provides, and where the agent deviates. This is the fastest feedback loop for skill refinement. 


 Figure 5: Asking the on-demand chat to analyse skill invocation details 

Figure 6: Detailed skill invocation analysis 

Figure 7: Investigation step-by-step with skill guidance 


Getting skills into DevOps Agent 

Once your skill is written and tested, deployment is straightforward. You can attach skills through the DevOps Agent console or via the API. 

Targeting the right agent type 

DevOps Agent runs several specialized agent types, each suited to different operational phases. Assigning your skill to the right type ensures it loads in the right context: 
Agent type
Best for
Example skill
Incident Triage
Deep root-cause investigation during active incidents
RDS connection exhaustion investigation
Incident Mitigation
Quick remediation actions to restore service
Auto-scale Lambda concurrency, restart unhealthy ECS tasks
On-demand Chat
Conversational queries, ad-hoc reports, exploratory analysis
Cost anomaly summary, service dependency map
Evaluation
Proactive analysis and prevention
Weekly deployment risk score, canary health assessment

Activation checklist 

  • Review the generated SKILL.md: If the skill was auto-generated from an investigation, verify that the steps, thresholds, and decision points are correct. 
  • Refine the description: This is your last chance to ensure the agent will pick up the skill in the right context. 
  • Activate the skill: Skills are inactive by default when created. Ask the agent on chat to activate when ready. 
  • Assign to the appropriate agent type: Default is Generic. Narrow to a specific type for better precision. 
  • Monitor in production: Check the investigation journal to confirm the skill is being invoked in real incidents. 

Production best practices 

Lessons from teams who have deployed skills across real production environments: 
1. One skill per scenario. A skill that tries to handle RDS connection exhaustion, slow queries, and replication lag in one file will confuse the agent. Split them. Each skill should have a single, well-defined trigger condition. 
2. Keep skills under 4,000 tokens. Longer skills dilute signal. If your skill is getting long, split investigation steps from remediation steps, or move reference data into the references/ subdirectory. 
3. Use the Markdown structure deliberately. H2 headers create natural sections the agent can navigate. Decision points under clear headers are easier for the agent to follow than decision trees buried in prose. 
4. Include "when NOT to use this skill" guidance. Negative triggers are just as valuable as positive ones. If your RDS skill should not fire for Aurora Serverless auto-scaling events, say so explicitly. 
5. Version and iterate. Treat skills as living documents. After every major incident where a skill was invoked, review the investigation journal and ask: did the skill help or hinder? Refine accordingly. 
6. Compose skills with MCP servers. Skills that reference custom MCP servers can pull data from your internal tools — deployment trackers, change management systems, custom dashboards — making investigations richer and more context-aware. 

Measuring success 

The ultimate measure of a skill is whether it improves your operational metrics. Track these over time: 
  • MTTR (Mean Time to Resolution): The primary metric. Compare resolution times for incidents where the skill was invoked vs. incidents without. 
  • Skill invocation rate: Is the skill being loaded in the right context? A low invocation rate despite relevant incidents means the description needs work. 
  • Investigation success rate: Of the investigations where the skill loaded, how many reached the correct root cause? 
  • On-call engineer satisfaction: Qualitative but real. Are your engineers trusting the agent more? Are they escalating less? 

Getting started today 

You do not need a large-scale rollout to start seeing value. Here is a one-afternoon challenge: 
  1. Pick your most-paged runbook — the one your team reached for most often last quarter. 
  1. Convert it into a skill using the four transformation moves: add thresholds, name the tools, add decision points, define outputs. 
  1. Run three test scenarios against the skill and compare against a no-skill baseline. 
  1. Iterate once based on what you learn from the test runs. 
  1. Deploy and monitor — you will have a working, production-ready skill by the end of day.
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article