Encoding Kubernetes Runbooks as AWS DevOps Agent Skills: Faster EKS Incident Resolution
Learn how to encode your team's proven Kubernetes investigation procedures as AWS DevOps Agent Skills - so the agent automatically follows your battle-tested runbooks for OOMKilled pods, CrashLoopBackOff, CoreDNS failures, and node pressure events, across regions.
The Problem: Your Best Runbooks Live in People, Not Systems
When a pod enters
CrashLoopBackOff at 3 AM, your most experienced SRE knows exactly what to check. The engineer who joined three months ago doesn’t. And neither does an AI agent that hasn’t been taught your environment.AWS DevOps Agent automates incident investigation — but by default it reasons from general knowledge. Skills change that. A Skill is a structured instruction set you upload once to your Agent Space. From that point, every investigation that matches the Skill’s description follows your team’s exact procedure automatically, at any hour, across any region.
This post walks you through four Kubernetes-specific Skills that cover the most common EKS incident patterns, and how to deploy them across
us-east-1 and eu-west-1 .
What You Will Build
Four AWS DevOps Agent Skills targeting:
- OOMKilled pod investigation
- CrashLoopBackOff root cause analysis
- CoreDNS resolution failures
- Node pressure and pod eviction events
Each Skill tells the agent exactly which CloudWatch metrics to pull, which Logs Insights queries to run, and how to interpret the results — the same way your senior SRE would.
Prerequisites
- An AWS account with an Agent Space created in
us-east-1,us-west-2,eu-west-1, oreu-central-1
- An EKS cluster (v1.29+) with CloudWatch Container Insights enabled
- AWS CLI v2
-
kubectlv1.29+ configured against your cluster
Region note: AWS DevOps Agent is GA in 6 regions including US East (N. Virginia), US West (Oregon), Europe (Ireland), and Europe (Frankfurt). One Agent Space covers clusters in both US and EU — no separate Agent Spaces needed per region. See Supported Regions .
How Skills Work
A Skill is a directory containing a
SKILL.md file with Markdown instructions. The agent reads the description field in the frontmatter to decide whether the Skill is relevant to the current incident. If it matches, the agent loads the full Skill and follows the steps.The frontmatter structure:
---name: oomkilled-investigationdescription: Use this skill when a Kubernetes pod has been terminated due to OOMKilled. Covers memory metric analysis, container limits review, and remediation steps.---
The body contains step-by-step instructions in plain Markdown — no code required. You are writing a runbook that the agent executes on your behalf.
Skill 1: OOMKilled Pod Investigation
Create
skills/oomkilled-investigation/SKILL.md :---name: oomkilled-investigationdescription: Use this skill when a Kubernetes pod has been OOMKilled. Investigates memory usage, container limits, and node-level memory pressure.---
## OOMKilled Investigation
### Step 1: Confirm the OOMKilled Event
Run the following to identify affected pods:
```bashkubectl get pods -A | grep -i oomkilledkubectl describe pod <pod-name> -n <namespace> | grep -A5 "Last State"
Look for
OOMKilled in the Reason field of the last terminated container state.Step 2: Pull Memory Metrics from CloudWatch
Query Container Insights for the pod’s memory usage over the past hour:
- Namespace:
ContainerInsights
- Metric:
pod_memory_working_set
- Dimensions:
PodName,Namespace,ClusterName
- Statistic: Maximum, Period: 300s
Compare against the container’s configured memory limit.
Step 3: Check Container Memory Limits
kubectl get pod <pod-name> -n <namespace> \ -o jsonpath='{.spec.containers[*].resources}'
If
limits.memory is absent or lower than peak usage, that is your root cause.Step 4: Review Node Memory Pressure
kubectl describe node <node-name> | grep -A5 "Conditions"
If
MemoryPressure is True , the node itself is starved — consider scaling the node group.Remediation
- Increase
resources.limits.memoryandresources.requests.memoryin the pod spec.
- For JVM workloads, set
-XX:MaxRAMPercentageinstead of hardcoded heap flags.
- Enable Vertical Pod Autoscaler (VPA) in recommendation mode to right-size over time.
---
## Skill 2: CrashLoopBackOff Root Cause Analysis
Create `skills/crashloopbackoff-analysis/SKILL.md`:
```markdown---name: crashloopbackoff-analysisdescription: Use this skill when a pod is in CrashLoopBackOff state. Identifies whether the crash is caused by application errors, misconfiguration, missing secrets, or dependency failures.---
## CrashLoopBackOff Root Cause Analysis
### Step 1: Identify the Crash Pattern
```bashkubectl describe pod <pod-name> -n <namespace>kubectl logs <pod-name> -n <namespace> --previous
The
--previous flag retrieves logs from the last terminated container — critical when the pod restarts too fast to capture live logs.Step 2: Decode the Exit Code
Exit Code | Signal | Meaning |
|---|---|---|
1 | - | Application error - check application logs |
137 | SIGKILL | OOMKilled, failed liveness probe, or forced pod deletion (kubectl delete --force) |
139 | SIGSEGV | Segmentation fault - likely a native library or memory corruption bug |
143 | SIGTERM | SIGTERM not handled - graceful shutdown issue |
If exit code is 137, disambiguate before switching Skills:
kubectl get pod <pod-name> -n <namespace> \ -o jsonpath='{.status.containerStatuses[0].lastState.terminated.reason}'- If the reason is OOMKilled → switch to the OOMKilled Investigation Skill.
- If the reason is Error and the pod has a
livenessProbedefined → the liveness probe killed the container. Check probe timing in Step 5.
- If the reason is Error and no liveness probe exists → the container was force-deleted externally. Check
kubectl get eventsfor context.
Step 3: Query CloudWatch Logs Insights
Run this against the
/aws/containerinsights/<cluster>/application log group:fields @timestamp, log, kubernetes.pod_name, kubernetes.namespace_name| filter kubernetes.pod_name like "<pod-name>"| sort @timestamp desc| limit 50
Step 4: Check for Missing Secrets or ConfigMaps
kubectl get events -n <namespace> --sort-by='.lastTimestamp' | grep <pod-name>
CreateContainerConfigError events indicate a missing Secret or ConfigMap reference.Step 5: Verify Liveness and Readiness Probes
An overly aggressive liveness probe kills a slow-starting container before it is ready.
kubectl get pod <pod-name> -n <namespace> \ -o jsonpath='{.spec.containers[*].livenessProbe}'
Increase
initialDelaySeconds if startup time exceeds the probe threshold.Remediation
- Fix application-level errors surfaced in logs.
- Add or correct Secret/ConfigMap references.
- Tune probe timing for slow-starting containers.
- Use
startupProbefor applications with variable initialization time.
---
## Skill 3: CoreDNS Resolution Failures
Create `skills/coredns-resolution-failure/SKILL.md`:
```markdown---name: coredns-resolution-failuredescription: Use this skill when pods cannot resolve internal Kubernetes DNS names or external hostnames. Covers CoreDNS health, configuration, and upstream resolver issues.---
## CoreDNS Resolution Failure Investigation
### Step 1: Confirm DNS Failure from a Pod
```bashkubectl run dns-test --image=busybox:1.36 \ --restart=Never -it --rm -- nslookup kubernetes.default
If this fails, DNS resolution is broken cluster-wide. If it succeeds, the issue is pod-specific.
Step 2: Check CoreDNS Pod Health
kubectl get pods -n kube-system -l k8s-app=kube-dnskubectl logs -n kube-system -l k8s-app=kube-dns --tail=50
Look for
SERVFAIL , i/o timeout , or connection refused in logs.Step 3: Query CloudWatch Logs Insights for CoreDNS Errors
fields @timestamp, log| filter kubernetes.namespace_name = "kube-system"| filter kubernetes.container_name = "coredns"| filter log like /SERVFAIL|error|timeout/| sort @timestamp desc| limit 100
Step 4: Inspect the CoreDNS ConfigMap
kubectl get configmap coredns -n kube-system -o yaml
Verify the
forward plugin points to a valid upstream — typically the VPC DNS resolver at the .2 address of your VPC CIDR.Step 5: Check CoreDNS CPU via CloudWatch
High CoreDNS CPU indicates DNS query saturation, common in clusters with many short-lived pods.
- Namespace:
ContainerInsights
- Metric:
pod_cpu_utilization
- Dimensions:
PodName=coredns,Namespace=kube-system
Remediation
- Scale CoreDNS:
kubectl scale deployment coredns -n kube-system --replicas=3
- Enable the
autopathplugin to reduce DNS lookup round trips for in-cluster traffic.
- For external DNS failures, verify
AmazonProvidedDNSis enabled in your VPC DHCP options set.
- Consider
NodeLocal DNSCachefor high-query-rate clusters.
---
## Skill 4: Node Pressure and Pod Eviction Events
Create `skills/node-pressure-eviction/SKILL.md`:
```markdown---name: node-pressure-evictiondescription: Use this skill when pods are being evicted or nodes are reporting memory, disk, or PID pressure. Covers node condition analysis, eviction thresholds, and remediation.---
## Node Pressure and Pod Eviction Investigation
### Step 1: Identify Evicted Pods
```bashkubectl get pods -A | grep Evictedkubectl get events -A | grep Evict
Step 2: Identify the Pressure Type
kubectl describe nodes | grep -A8 "Conditions:"
Condition | Meaning when True |
|---|---|
MemoryPressure | Node is running low on memory |
DiskPressure | Root volume or image storage is filling up |
PIDPressure | Too many processes running on the node |
Step 3: Pull Node Metrics from CloudWatch
For memory pressure:
- Namespace:
ContainerInsights, Metric:node_memory_utilization
- Dimensions:
NodeName,ClusterName, Statistic: Average, Period: 300s
For disk pressure:
- Namespace:
ContainerInsights, Metric:node_filesystem_utilization
Step 4: Identify Top Consumers
kubectl top pods -A --sort-by=memory | head -20kubectl top nodes
Step 5: Review Eviction Thresholds
EKS uses kubelet eviction thresholds defined in the node configuration. The default soft eviction triggers at 10% memory available. Frequent evictions indicate pod memory requests do not reflect actual usage.
Remediation
- Memory pressure: right-size pod requests, enable VPA, or add nodes.
- Disk pressure: increase EBS volume size, prune unused images with
crictl rmi --prune, or configure log rotation.
- PID pressure: check for runaway processes; enforce PID limits via
LimitRange.
- Add a dedicated node group for memory-intensive workloads and isolate them with taints and tolerations.
---
## Uploading Skills to Your Agent Space
Skills are managed as **assets** through the AWS DevOps Agent Asset API. You have two options.
### Option A: Operator Web App (recommended for first-time setup)
Open the AWS Console, navigate to **DevOps Agent → Agent Spaces → your space → Assets**, and use the **Create asset** button. Paste your `SKILL.md` content directly into the editor and set the asset type to `SKILL`. This is the fastest path if you are uploading a handful of Skills manually.
### Option B: AWS CLI (recommended for automation and CI/CD)
Use `create-asset` to upload each Skill. The `--content` parameter accepts the Markdown content of your `SKILL.md` as a JSON-encoded string:
```bashAGENT_SPACE_ID="<your-agent-space-id>"REGION="us-east-1"
for skill_dir in skills/*/; do skill_name=$(basename "$skill_dir") skill_content=$(cat "${skill_dir}SKILL.md")
aws devops-agent create-asset \ --agent-space-id "$AGENT_SPACE_ID" \ --name "$skill_name" \ --asset-type SKILL \ --content "{\"text\": $(echo "$skill_content" | python3 -c 'import json,sys; print(json.dumps(sys.stdin.read()))')}" \ --region "$REGION"
echo "Uploaded: $skill_name"done
Because a single Agent Space covers multiple regions, you do not need to repeat the upload for
eu-west-1 .Verify the assets were created:
aws devops-agent list-assets \ --agent-space-id "$AGENT_SPACE_ID" \ --asset-type SKILL \ --region "$REGION"
Each Skill should appear in the response with an
assetId and ACTIVE status. If you need to update a Skill later, use update-asset with the same --asset-id instead of creating a duplicate.
Testing the Skills
Start a test investigation to confirm the agent selects the right Skill:
aws devops-agent start-investigation \ --agent-space-id <your-agent-space-id> \ --description "Pod nginx-deployment-abc123 is in CrashLoopBackOff in the production namespace" \ --region us-east-1
The agent matches the description against your Skills, loads
crashloopbackoff-analysis , and follows your runbook. Stream the results in the AWS Console under DevOps Agent → Investigations.
Keeping Skills Up to Date
Skills are living documents. Treat them like code:
- Store them in your repository alongside cluster configuration.
- Update the Skill whenever your runbook changes — new metrics, new tools, changed thresholds.
- Use a
versionfield in frontmatter to track changes:
---name: oomkilled-investigationversion: 1.2.0description: ...---
Run
update-asset after any change and the agent will use the latest version on the next investigation:aws devops-agent update-asset \ --agent-space-id "$AGENT_SPACE_ID" \ --asset-id "<asset-id-from-list-assets>" \ --content "{\"text\": $(cat skills/oomkilled-investigation/SKILL.md | python3 -c 'import json,sys; print(json.dumps(sys.stdin.read()))')}" \ --region us-east-1
What’s Next
You now have four Skills that encode your team’s Kubernetes expertise into AWS DevOps Agent. The agent can investigate OOM kills, crash loops, DNS failures, and node pressure at 3 AM — following your exact procedure, not a generic one.
From here, consider:
- Adding a Skill for HPA failures — when pods aren’t scaling despite high CPU utilization.
- Integrating with Taskei to automatically create a task when an investigation surfaces a recurring issue.
- Sharing Skills across teams by storing them in a shared package that multiple Agent Spaces can reference.
The institutional knowledge that used to live in one person’s head is now part of your infrastructure.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article