AWS Builder Center

Generating Operational Runbooks with Kiro: A Practical Guide for SREs

Show Kiro generating a runbook for a common operational task ‚deploying a new service version, responding to a high-error-rate alert, or rotating a database credential.

On-call at 3 AM. Your pager fires. Error rates are spiking. You open the runbook and find… three bullet points from 2022 and a link to a wiki that no longer exists.
Sound familiar?
Good runbooks save incidents. Bad ones — or missing ones — extend them. The problem isn't that teams don't value runbooks. It's that writing them is tedious, they rot quickly, and nobody wants to be the one maintaining them.
Kiro changes that equation. In this post, we'll walk through using Kiro to generate a production-quality runbook for a common operational scenario: responding to a high-error-rate alert. Along the way, we'll break down what separates a useful runbook from a decorative one, and show how to iterate on the output until it matches your environment exactly.

What Makes a Good Runbook

Before we generate anything, let's establish the bar. A runbook that actually helps during an incident has a few non-negotiable qualities:
  • Clear, sequential steps. Each action should be unambiguous. "Check the logs" is not a step. "Run  kubectl logs -l app=api-gateway --tail=100 -n production " is.
  • Decision points. Incidents branch. A good runbook acknowledges this with explicit if/then logic: "If error rate drops below 1% after rollback, proceed to verification. If not, escalate."
  • Rollback procedures. Every action that changes state needs an undo path. If step 4 says "scale down the canary," the runbook should tell you how to reverse that if things get worse.
  • Context, not novels. SREs don't need paragraphs during an incident. They need the right command, the right threshold, and the right next step.

The Scenario: High Error Rate Alert on an API Service

Let's say your team runs a microservices platform on Kubernetes. You have a Datadog monitor that fires when the 5xx error rate on your  order-service  exceeds 5% over a 5-minute window. Today, we're generating the runbook for that alert.

Generating the Runbook with Kiro

Open Kiro and describe what you need in plain language:
Generate an operational runbook for responding to a high 5xx error rate alert
on our order-service. The service runs on Kubernetes (EKS), uses Datadog for
monitoring, and is deployed via ArgoCD. Include triage steps, decision points
for rollback vs. hotfix, rollback procedures, and an escalation path.

Kiro produces a structured runbook. Let's walk through the key sections of what it generates and why each one matters.

Section 1: Alert Context

## Alert: order-service 5xx Error Rate > 5%

**Monitor:** Datadog – "order-service High Error Rate"
**Threshold:** 5xx responses exceed 5% of total traffic over 5 minutes
**Severity:** P2 (auto-escalates to P1 after 15 minutes unacknowledged)
**Owner:** Platform Team (#platform-oncall in Slack)

This section exists so the responder immediately knows what fired, why, and who owns it. No hunting through Datadog to figure out the threshold. No Slack archaeology to find the right channel.

Section 2: Initial Triage

## Step 1: Confirm the Alert

1. Open the Datadog dashboard:
https://app.datadoghq.com/dashboard/abc-123/order-service-overview

2. Verify the error rate is still elevated:
- Check the "5xx Rate" graph (top-right panel)
- Confirm the spike is not a single burst that already resolved

3. Check for related alerts:
- Is the database alert also firing? → Jump to "Database Issues" appendix
- Is the upstream payment-service degraded? → Jump to "Upstream Dependency Failure"

> **Decision Point:** If the error rate has already dropped below 2%
> and is trending down, monitor for 10 minutes before taking action.
> If it's sustained or climbing, proceed to Step 2.

Notice the decision point. This prevents unnecessary rollbacks when a transient spike resolves on its own. It also cross-references related alerts — because in practice, the root cause is often one layer removed from the service that's paging you.

Section 3: Identify the Cause

## Step 2: Identify the Cause

1. Check recent deployments:
```bash
argocd app history order-service --output json | jq '.[0:3]'

  • Was there a deployment in the last 30 minutes? Note the commit SHA.
  1. Check pod health: __PRE_PLACEHOLDER_4__
  • Are pods in CrashLoopBackOff? → Likely a bad deploy. Go to Step 3a.
  • Are pods running but error rate is high? → Go to Step 3b.
  1. Check application logs for the most common error: __PRE_PLACEHOLDER_5__
Decision Point:* Recent deploy + CrashLoopBackOff → Step 3a: Rollback
  • No recent deploy + elevated errors → Step 3b: Investigate Runtime Issue
  • Recent deploy + pods healthy + new error pattern → Step 3a: Rollback

This is where the branching logic earns its keep. The responder doesn't have to guess what to do next — the runbook routes them based on what they're actually seeing.

### Section 4: Rollback Procedure

```markdown
## Step 3a: Rollback via ArgoCD

1. Identify the last healthy revision:
```bash
argocd app history order-service

Note the revision number before the suspected bad deploy.
  1. Roll back: __PRE_PLACEHOLDER_7__
  1. Verify the rollback: __PRE_PLACEHOLDER_8__
  • Health: Healthy
  • Sync: OutOfSync (expected — you've rolled back)
  1. Confirm error rate is dropping:
  • Return to the Datadog dashboard
  • Wait 5 minutes for metrics to stabilize
  • Error rate should drop below 2%
If error rate does NOT drop after rollback: The issue may not be deployment-related. Proceed to Step 3b. Notify the on-call lead in #platform-oncall.

Every mutating action (the rollback itself) has a verification step and a fallback path. The responder is never left wondering "did that work?" or "what if it didn't?"

### Section 5: Escalation

```markdown
## Escalation Path

| Condition | Action |
|---|---|
| Error rate > 10% for 10+ minutes | Page the on-call lead |
| Rollback fails or doesn't resolve | Page the service owner |
| Customer-facing impact confirmed | Notify #incident-response, open incident ticket |
| Database or infrastructure root cause | Page the infrastructure on-call |

A table works well here. During an incident, the responder can scan it in seconds.

Iterating: Adding a Troubleshooting Section

The generated runbook is solid, but your team also wants a section for the runtime investigation path (Step 3b) that covers common non-deployment causes. In Kiro, you just ask:
Add a troubleshooting section for Step 3b that covers: database connection
pool exhaustion, downstream service timeouts, and memory pressure on pods.
Include diagnostic commands for each.

Kiro adds the section inline, maintaining the same structure and tone:
## Step 3b: Investigate Runtime Issue

### Database Connection Pool Exhaustion
```bash
kubectl exec -it $(kubectl get pod -l app=order-service -n production \
-o jsonpath='{.items[0].metadata.name}') -n production -- \
curl -s localhost:8080/actuator/metrics/hikaricp.connections.active

  • If active connections are near the pool max (default: 20), the pool is saturated.
  • Mitigation: Scale up replicas to distribute load: __PRE_PLACEHOLDER_12__
  • Rollback: Scale back down once the root cause is resolved: __PRE_PLACEHOLDER_13__

Downstream Service Timeouts

kubectl logs -l app=order-service -n production --tail=200 | \
grep -i "timeout\|timed out" | awk '{print $NF}' | sort | uniq -c | sort -rn

  • Identify which downstream service is timing out.
  • Check that service's health independently.
  • Mitigation: If the downstream is degraded, enable the circuit breaker: __PRE_PLACEHOLDER_15__

Memory Pressure

kubectl top pods -l app=order-service -n production

  • Compare current memory usage against the pod's memory limit.
  • If pods are near their limit, check for memory leaks in recent changes.
  • Mitigation: Restart pods to reclaim memory (rolling restart): __PRE_PLACEHOLDER_17__

Same pattern: diagnose, mitigate, and include the reversal path. Kiro follows the structure it already established.

## Adding a Specific Step

You can also ask Kiro to insert a step at a precise location. For example:

After Step 1, add a step to check if a maintenance window is currently active in PagerDuty before proceeding with triage.

Kiro inserts it as "Step 1.5" or renumbers the steps, depending on your preference, and adjusts all the cross-references ("Go to Step 3a" becomes "Go to Step 4a" if needed). This kind of structural awareness is where generating runbooks in an IDE pays off compared to editing a wiki page and hoping you caught every reference.

## Why Generate Runbooks in Kiro

DevOps teams have been writing runbooks in wikis, Google Docs, and Notion for years. Generating them in Kiro offers a few practical advantages:

- **Speed.** A first draft in seconds instead of an hour. You spend your time refining, not starting from scratch.
- **Consistency.** Every runbook follows the same structure: context, triage, decision points, actions, rollback, escalation. No more runbooks that are just a pile of links.
- **Iteration in context.** You can ask Kiro to add, remove, or restructure sections conversationally. "Add a step for checking the CDN cache hit rate" is faster than reformatting a wiki page.
- **Living alongside code.** Runbooks stored in your repo (next to the service they describe) are versioned, reviewable, and discoverable. They don't rot in a wiki nobody visits.

## Tips for Better Runbooks

A few things we've learned from teams using Kiro for operational docs:

1. **One runbook, one scenario.** Don't combine "high error rate" and "high latency" into a single doc. They share some steps but diverge quickly. Separate runbooks are easier to follow under pressure.

2. **Include the "do nothing" path.** Not every alert requires action. If the runbook's first decision point can result in "monitor and wait," say so explicitly. It prevents unnecessary interventions.

3. **Test your runbooks.** Run through them during game days. If a command doesn't work or a dashboard link is broken, fix it before the next real incident.

4. **Keep commands copy-pasteable.** Avoid placeholders where possible. If the namespace is always `production`, hardcode it. The fewer things a responder has to fill in at 3 AM, the better.

5. **Date your runbooks.** Add a "Last verified" date. A runbook from 18 months ago with no verification date inspires zero confidence.

## Wrapping Up

Runbooks are one of those things every team knows they need and nobody enjoys writing. Kiro doesn't make the operational knowledge appear from nowhere — you still need to know your systems. But it handles the structure, the formatting, and the tedious parts so you can focus on getting the content right.

Start with one alert. Generate the runbook. Run through it on your next game day. Iterate from there. That's the fastest path from "we should really document this" to a runbook that actually helps when the pager goes off.
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article