AWS Builder Center

Encoding Kubernetes Runbooks as AWS DevOps Agent Skills: Faster EKS Incident Resolution

Learn how to encode your team's proven Kubernetes investigation procedures as AWS DevOps Agent Skills - so the agent automatically follows your battle-tested runbooks for OOMKilled pods, CrashLoopBackOff, CoreDNS failures, and node pressure events, across regions.

Technical Account Manager

The Problem: Your Best Runbooks Live in People, Not Systems

When a pod enters  CrashLoopBackOff  at 3 AM, your most experienced SRE knows exactly what to check. The engineer who joined three months ago doesn’t. And neither does an AI agent that hasn’t been taught your environment.
AWS DevOps Agent automates incident investigation — but by default it reasons from general knowledge. Skills change that. A Skill is a structured instruction set you upload once to your Agent Space. From that point, every investigation that matches the Skill’s description follows your team’s exact procedure automatically, at any hour, across any region.
This post walks you through four Kubernetes-specific Skills that cover the most common EKS incident patterns, and how to deploy them across  us-east-1  and  eu-west-1 .



What You Will Build

Four AWS DevOps Agent Skills targeting:
  • OOMKilled pod investigation
  • CrashLoopBackOff root cause analysis
  • CoreDNS resolution failures
  • Node pressure and pod eviction events
Each Skill tells the agent exactly which CloudWatch metrics to pull, which Logs Insights queries to run, and how to interpret the results — the same way your senior SRE would.



Prerequisites

  • An AWS account with an Agent Space created in  us-east-1 ,  us-west-2 ,  eu-west-1 , or  eu-central-1 
  • An EKS cluster (v1.29+) with CloudWatch Container Insights enabled
  • AWS CLI v2
  •  kubectl  v1.29+ configured against your cluster
Region note: AWS DevOps Agent is GA in 6 regions including US East (N. Virginia), US West (Oregon), Europe (Ireland), and Europe (Frankfurt). One Agent Space covers clusters in both US and EU — no separate Agent Spaces needed per region. See  Supported Regions .



How Skills Work

A Skill is a directory containing a  SKILL.md  file with Markdown instructions. The agent reads the  description  field in the frontmatter to decide whether the Skill is relevant to the current incident. If it matches, the agent loads the full Skill and follows the steps.
The frontmatter structure:
---
name: oomkilled-investigation
description: Use this skill when a Kubernetes pod has been terminated due to OOMKilled. Covers memory metric analysis, container limits review, and remediation steps.
---

The body contains step-by-step instructions in plain Markdown — no code required. You are writing a runbook that the agent executes on your behalf.



Skill 1: OOMKilled Pod Investigation

Create  skills/oomkilled-investigation/SKILL.md :
---
name: oomkilled-investigation
description: Use this skill when a Kubernetes pod has been OOMKilled. Investigates memory usage, container limits, and node-level memory pressure.
---

## OOMKilled Investigation

### Step 1: Confirm the OOMKilled Event

Run the following to identify affected pods:

```bash
kubectl get pods -A | grep -i oomkilled
kubectl describe pod <pod-name> -n <namespace> | grep -A5 "Last State"

Look for  OOMKilled  in the  Reason  field of the last terminated container state.

Step 2: Pull Memory Metrics from CloudWatch

Query Container Insights for the pod’s memory usage over the past hour:
  • Namespace:  ContainerInsights 
  • Metric:  pod_memory_working_set 
  • Dimensions:  PodName ,  Namespace ,  ClusterName 
  • Statistic: Maximum, Period: 300s
Compare against the container’s configured memory limit.

Step 3: Check Container Memory Limits

kubectl get pod <pod-name> -n <namespace> \
-o jsonpath='{.spec.containers[*].resources}'

If  limits.memory  is absent or lower than peak usage, that is your root cause.

Step 4: Review Node Memory Pressure

kubectl describe node <node-name> | grep -A5 "Conditions"

If  MemoryPressure  is  True , the node itself is starved — consider scaling the node group.

Remediation

  • Increase  resources.limits.memory  and  resources.requests.memory  in the pod spec.
  • For JVM workloads, set  -XX:MaxRAMPercentage  instead of hardcoded heap flags.
  • Enable Vertical Pod Autoscaler (VPA) in recommendation mode to right-size over time.

---

## Skill 2: CrashLoopBackOff Root Cause Analysis

Create `skills/crashloopbackoff-analysis/SKILL.md`:

```markdown
---
name: crashloopbackoff-analysis
description: Use this skill when a pod is in CrashLoopBackOff state. Identifies whether the crash is caused by application errors, misconfiguration, missing secrets, or dependency failures.
---

## CrashLoopBackOff Root Cause Analysis

### Step 1: Identify the Crash Pattern

```bash
kubectl describe pod <pod-name> -n <namespace>
kubectl logs <pod-name> -n <namespace> --previous

The  --previous  flag retrieves logs from the last terminated container — critical when the pod restarts too fast to capture live logs.

Step 2: Decode the Exit Code

Exit Code
Signal
Meaning
1
-
Application error - check application logs
137
SIGKILL
OOMKilled, failed liveness probe, or forced pod deletion (kubectl delete --force)
139
SIGSEGV
Segmentation fault - likely a native library or memory corruption bug
143
SIGTERM
SIGTERM not handled - graceful shutdown issue
If exit code is 137, disambiguate before switching Skills:
kubectl get pod <pod-name> -n <namespace> \
-o jsonpath='{.status.containerStatuses[0].lastState.terminated.reason}'
  • If the reason is OOMKilled → switch to the OOMKilled Investigation Skill.
  • If the reason is Error and the pod has a  livenessProbe  defined → the liveness probe killed the container. Check probe timing in Step 5.
  • If the reason is Error and no liveness probe exists → the container was force-deleted externally. Check  kubectl get events  for context.

Step 3: Query CloudWatch Logs Insights

Run this against the  /aws/containerinsights/<cluster>/application  log group:
fields @timestamp, log, kubernetes.pod_name, kubernetes.namespace_name
| filter kubernetes.pod_name like "<pod-name>"
| sort @timestamp desc
| limit 50


Step 4: Check for Missing Secrets or ConfigMaps

kubectl get events -n <namespace> --sort-by='.lastTimestamp' | grep <pod-name>

 CreateContainerConfigError  events indicate a missing Secret or ConfigMap reference.

Step 5: Verify Liveness and Readiness Probes

An overly aggressive liveness probe kills a slow-starting container before it is ready.
kubectl get pod <pod-name> -n <namespace> \
-o jsonpath='{.spec.containers[*].livenessProbe}'

Increase  initialDelaySeconds  if startup time exceeds the probe threshold.

Remediation

  • Fix application-level errors surfaced in logs.
  • Add or correct Secret/ConfigMap references.
  • Tune probe timing for slow-starting containers.
  • Use  startupProbe  for applications with variable initialization time.

---

## Skill 3: CoreDNS Resolution Failures

Create `skills/coredns-resolution-failure/SKILL.md`:

```markdown
---
name: coredns-resolution-failure
description: Use this skill when pods cannot resolve internal Kubernetes DNS names or external hostnames. Covers CoreDNS health, configuration, and upstream resolver issues.
---

## CoreDNS Resolution Failure Investigation

### Step 1: Confirm DNS Failure from a Pod

```bash
kubectl run dns-test --image=busybox:1.36 \
--restart=Never -it --rm -- nslookup kubernetes.default

If this fails, DNS resolution is broken cluster-wide. If it succeeds, the issue is pod-specific.

Step 2: Check CoreDNS Pod Health

kubectl get pods -n kube-system -l k8s-app=kube-dns
kubectl logs -n kube-system -l k8s-app=kube-dns --tail=50

Look for  SERVFAIL ,  i/o timeout , or  connection refused  in logs.

Step 3: Query CloudWatch Logs Insights for CoreDNS Errors

fields @timestamp, log
| filter kubernetes.namespace_name = "kube-system"
| filter kubernetes.container_name = "coredns"
| filter log like /SERVFAIL|error|timeout/
| sort @timestamp desc
| limit 100


Step 4: Inspect the CoreDNS ConfigMap

kubectl get configmap coredns -n kube-system -o yaml

Verify the  forward  plugin points to a valid upstream — typically the VPC DNS resolver at the  .2  address of your VPC CIDR.

Step 5: Check CoreDNS CPU via CloudWatch

High CoreDNS CPU indicates DNS query saturation, common in clusters with many short-lived pods.
  • Namespace:  ContainerInsights 
  • Metric:  pod_cpu_utilization 
  • Dimensions:  PodName=coredns ,  Namespace=kube-system 

Remediation

  • Scale CoreDNS:  kubectl scale deployment coredns -n kube-system --replicas=3 
  • Enable the  autopath  plugin to reduce DNS lookup round trips for in-cluster traffic.
  • For external DNS failures, verify  AmazonProvidedDNS  is enabled in your VPC DHCP options set.
  • Consider  NodeLocal DNSCache  for high-query-rate clusters.

---

## Skill 4: Node Pressure and Pod Eviction Events

Create `skills/node-pressure-eviction/SKILL.md`:

```markdown
---
name: node-pressure-eviction
description: Use this skill when pods are being evicted or nodes are reporting memory, disk, or PID pressure. Covers node condition analysis, eviction thresholds, and remediation.
---

## Node Pressure and Pod Eviction Investigation

### Step 1: Identify Evicted Pods

```bash
kubectl get pods -A | grep Evicted
kubectl get events -A | grep Evict


Step 2: Identify the Pressure Type

kubectl describe nodes | grep -A8 "Conditions:"

Condition
Meaning when True
MemoryPressure
Node is running low on memory
DiskPressure
Root volume or image storage is filling up
PIDPressure
Too many processes running on the node

Step 3: Pull Node Metrics from CloudWatch

For memory pressure:
  • Namespace:  ContainerInsights , Metric:  node_memory_utilization 
  • Dimensions:  NodeName ,  ClusterName , Statistic: Average, Period: 300s
For disk pressure:
  • Namespace:  ContainerInsights , Metric:  node_filesystem_utilization 

Step 4: Identify Top Consumers

kubectl top pods -A --sort-by=memory | head -20
kubectl top nodes


Step 5: Review Eviction Thresholds

EKS uses kubelet eviction thresholds defined in the node configuration. The default soft eviction triggers at 10% memory available. Frequent evictions indicate pod memory requests do not reflect actual usage.

Remediation

  • Memory pressure: right-size pod requests, enable VPA, or add nodes.
  • Disk pressure: increase EBS volume size, prune unused images with  crictl rmi --prune , or configure log rotation.
  • PID pressure: check for runaway processes; enforce PID limits via  LimitRange .
  • Add a dedicated node group for memory-intensive workloads and isolate them with taints and tolerations.

---

## Uploading Skills to Your Agent Space

Skills are managed as **assets** through the AWS DevOps Agent Asset API. You have two options.

### Option A: Operator Web App (recommended for first-time setup)

Open the AWS Console, navigate to **DevOps Agent → Agent Spaces → your space → Assets**, and use the **Create asset** button. Paste your `SKILL.md` content directly into the editor and set the asset type to `SKILL`. This is the fastest path if you are uploading a handful of Skills manually.

### Option B: AWS CLI (recommended for automation and CI/CD)

Use `create-asset` to upload each Skill. The `--content` parameter accepts the Markdown content of your `SKILL.md` as a JSON-encoded string:

```bash
AGENT_SPACE_ID="<your-agent-space-id>"
REGION="us-east-1"

for skill_dir in skills/*/; do
skill_name=$(basename "$skill_dir")
skill_content=$(cat "${skill_dir}SKILL.md")

aws devops-agent create-asset \
--agent-space-id "$AGENT_SPACE_ID" \
--name "$skill_name" \
--asset-type SKILL \
--content "{\"text\": $(echo "$skill_content" | python3 -c 'import json,sys; print(json.dumps(sys.stdin.read()))')}" \
--region "$REGION"

echo "Uploaded: $skill_name"
done

Because a single Agent Space covers multiple regions, you do not need to repeat the upload for  eu-west-1 .
Verify the assets were created:
aws devops-agent list-assets \
--agent-space-id "$AGENT_SPACE_ID" \
--asset-type SKILL \
--region "$REGION"

Each Skill should appear in the response with an  assetId  and  ACTIVE  status. If you need to update a Skill later, use  update-asset  with the same  --asset-id  instead of creating a duplicate.



Testing the Skills

Start a test investigation to confirm the agent selects the right Skill:
aws devops-agent start-investigation \
--agent-space-id <your-agent-space-id> \
--description "Pod nginx-deployment-abc123 is in CrashLoopBackOff in the production namespace" \
--region us-east-1

The agent matches the description against your Skills, loads  crashloopbackoff-analysis , and follows your runbook. Stream the results in the AWS Console under DevOps Agent → Investigations.



Keeping Skills Up to Date

Skills are living documents. Treat them like code:
  • Store them in your repository alongside cluster configuration.
  • Update the Skill whenever your runbook changes — new metrics, new tools, changed thresholds.
  • Use a  version  field in frontmatter to track changes:
---
name: oomkilled-investigation
version: 1.2.0
description: ...
---

Run  update-asset  after any change and the agent will use the latest version on the next investigation:
aws devops-agent update-asset \
--agent-space-id "$AGENT_SPACE_ID" \
--asset-id "<asset-id-from-list-assets>" \
--content "{\"text\": $(cat skills/oomkilled-investigation/SKILL.md | python3 -c 'import json,sys; print(json.dumps(sys.stdin.read()))')}" \
--region us-east-1




What’s Next

You now have four Skills that encode your team’s Kubernetes expertise into AWS DevOps Agent. The agent can investigate OOM kills, crash loops, DNS failures, and node pressure at 3 AM — following your exact procedure, not a generic one.
From here, consider:
  • Adding a Skill for HPA failures — when pods aren’t scaling despite high CPU utilization.
  • Integrating with Taskei to automatically create a task when an investigation surfaces a recurring issue.
  • Sharing Skills across teams by storing them in a shared package that multiple Agent Spaces can reference.
The institutional knowledge that used to live in one person’s head is now part of your infrastructure.
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article