KiroCrew: From Single Agents to Multi-Agent Orchestration: Building Production DevOps Automation on AWS

This article explores how multi-agent orchestration fundamentally changes incident response and infrastructure automation — moving from a single AI assistant running commands sequentially to a coordinated team of specialized agents working in parallel on the same problem, each constrained to its domain, sharing durable state, and converging on a governed solution.

Roman Ceresnak

Cloud Architect

Published Aug 17, 2026

Last modified Aug 17, 2026

Most teams that deploy AI into production start with a single-agent model: send a prompt, get a response, execute the suggestion. It's roughly one integration pattern, and it's good enough to handle predictable queries and script-writing tasks. It's also why every AI-assisted deployment looks the same after the first hour — the agent knows only what fits in one context window, can't reason across multiple domains, and can't delegate to specialists without human handoff.

Organizations running production infrastructure face a different problem. When a Kubernetes cluster goes down, you need simultaneous expertise in security, infrastructure, observability, and application-level debugging — not sequential queries to the same assistant. When you need to audit infrastructure as code for compliance, you need exact policy enforcement, not best-effort suggestions. When a deployed fix needs to land in production in minutes, you need convergence, not iteration.

This is where the architecture of multi-agent orchestration becomes visible as something different from "a smarter single assistant."

The fundamental shift: orchestration, not chaining

Most AI automation today works inside a prompt. Agents are programmed as role-playing scenarios — "pretend you are a Terraform expert, now pretend you are a security reviewer" — all running inside one context window, one model invocation, one serialized thought process.

Multi-agent orchestration moves the coordination outside the prompt:

Single agent (traditional):

Diagram showing sequential single-agent flow: reasoning loop connects to Terraform reasoning, which connects to security checks. Text states: Sequential. Slow. Limited by one window. — Single agent reasoning is sequential and constrained by one context window.

Traditional single-agent systems serialize all reasoning into one context window. The agent must role-play as a Terraform expert, then switch to security expert, then switch to monitoring expert — all within the same reasoning loop. Each domain gets partial attention. When the agent reaches the context limit, reasoning stops. Handoffs between domains happen implicitly within the prompt, not as explicit architectural boundaries.

Multi-agent orchestration (new):

iagram showing coordinator at top connected to three specialist agents below: Security Agent (IAM, SG), Terraform Agent (IaC, modules), and Observability Agent (alarms, logs). Agents work in parallel. — Multi-agent orchestration distributes reasoning across specialized agents coordinated by a central orchestrator.

The coordinator assigns work to specialized agents in parallel. The Security Agent focuses exclusively on IAM and security groups. The Terraform Agent focuses on infrastructure as code validation. The Observability Agent designs alarms and logging strategy. Each agent has 100% of its context budget for its domain. The coordinator tracks dependencies, collects proposals, and enforces convergence — not the prompt.

Each agent has its own context. Each has specialized knowledge and constraints. The coordinator assigns work, aggregates results, and enforces convergence rules. Critically: agents propose changes, not necessarily execute them. A promotion gate decides what's written to shared state.

This architecture solves a specific problem: domain-specific reasoning under concurrency. It's not faster if you don't need parallelism. It becomes necessary exactly when you do.

Five operational properties that change with multi-agent systems

Scope enforcement moves to the coordinator

In a single-agent system, scope is a rule in the prompt: "only touch Kubernetes files in the production namespace." In a multi-agent system, scope has to be enforced where it matters — at the task boundary, before delegation, and in the promotion gate that decides what gets written.

The coordinator keeps a model of what each agent can do:

- Kubernetes Detective → read all manifests, propose changes, validate syntax
- Terraform Agent → read/write IaC, check module contracts, plan output
- Security Agent → audit IAM, check SG rules, enforce guardrails
- Observability Expert → design alarms, dashboard config, retention rules

Each agent gets read access to the full repository but write access only to its domain. A Terraform agent can't modify Kubernetes manifests even if it proposes the change. The promotion gate enforces this, not the prompt.

Work is delegated, not sequentially queued

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
Single agent (chaining):
Turn 1: "Analyze Kubernetes deployment"
  → responds with findings
Turn 2: "Now audit the Terraform"
  → responds with findings
Turn 3: "Now design monitoring"
  → responds with findings
Total time: T1 + T2 + T3

Multi-agent (orchestration):
Coordinator spawns Agent A, B, C simultaneously
  Agent A analyzes Kubernetes
  Agent B audits Terraform
  Agent C designs monitoring
  (all happening in parallel, wall-clock time ≈ max(T1, T2, T3))

The speedup is real when the work is independent. When Agent A is waiting for a database query, Agents B and C are working. When Agent B hits a decision boundary (should we change this security group?), it can ask for approval while Agents A and C keep moving.

State is durable and scoped, not ephemeral

Single-agent systems keep state in the context window of one session. Close the tab, restart the service, that state evaporates.

Multi-agent systems need durable state:

Proposal store (DynamoDB) — what each agent proposed, why, confidence level
Decision log (S3) — what was approved, what was rejected, who approved
Generated artifacts (S3 + GitHub) — files that were changed, with provenance

Every artifact carries metadata: which agent created it, which turn of the orchestration, what approvals it went through.

1
2
3
4
5
6
7
8
9
10
11
12
13
{
  "proposal_id": "sec-sg-001",
  "agent": "Security Agent",
  "type": "security_group_rule_removal",
  "proposed_change": "Remove 0.0.0.0/0 ingress on port 5432",
  "confidence": 0.98,
  "source": "automated_audit",
  "approval_status": "approved_by_human",
  "approved_at": "2024-08-17T15:32:00Z",
  "approved_by": "security-team",
  "applied_at": "2024-08-17T15:35:00Z",
  "trace_back": "incident_20240817_memory_leak"
}

That log is the audit trail. It's also how you know which decisions to reverse if a change causes problems.

The Kubernetes Detective identified memory exhaustion as the primary cause and proposed three changes: increasing the memory limit from 128Mi to 512Mi, adding missing health probes (readiness, liveness, startup), and enabling HPA with proper thresholds for CPU and memory-based scaling.

Terraform diffs showing security group restrictions and port 5432 removal

The Terraform Agent restricted database access by removing the 0.0.0.0/0 CIDR block and applied common tagging standards across all infrastructure resources, eliminating the security group misconfiguration.

RCA and Preventive Actions Documentation — The coordinator generated two critical documents: a complete root cause analysis with timeline and evidence, and a preventive actions runbook with step-by-step measures and success criteria.

The Incident Manager synthesized findings into two documents: a complete Root Cause Analysis with timeline and evidence, and Preventive Actions with step-by-step remediation measures and success criteria.

Table showing 8 files changed and 2 created: Kubernetes deployment memory increase, HPA configuration, Terraform security restrictions, monitoring alarms and rules, and RCA documentation — What was fixed (8 files changed, 2 created)

Convergence and approval are explicit gates, not implicit

A single agent generates one response. Multi-agent systems can produce conflicting proposals: Agent A says "increase memory limit to 512Mi," Agent B says "512Mi is wasteful, optimize the app instead." The coordinator doesn't pick one — it surfaces both and waits for a decision.

This requires a promotion gate with clear rules:

Automatic approval: policies that are fully determined (lint rules, syntax validation)
Human approval: changes to security, cost impact, or user-facing behavior
Conditional approval: changes that depend on other agents' decisions (e.g., "approve Kubernetes HPA if Terraform agent approved the CloudWatch alarms")

1
2
3
4
5
Proposal from Agent → Confidence check → Policy check → Scope check → Approval queue
  ✓ high confidence    ✓ matches guard   ✓ in scope                  Auto-approved
  ? medium confidence  ✓ matches guard   ✓ in scope                  → Human approval
  ? low confidence     ✓ matches guard   ✓ in scope                  → Rejected
  ✓ any confidence     ✗ violates guard  ✓ in scope                  → Rejected

The gate is deterministic. The same proposal always follows the same path. That consistency is what makes the system auditable.

Not all proposals are permanent fixes. The memory limit increase stops the immediate crash, but the underlying application design—unconditional 2.3 GB cache load at startup—requires deeper architectural change.

The coordinator collected proposals from all five agents and the promotion gate approved the changes. Eight existing files were modified across Kubernetes, Terraform, and monitoring infrastructure. Two new documents were created: a comprehensive root cause analysis linking the deterministic OOMKill to the 256Mi memory limit and app cache loading, and a preventive actions runbook with five concrete measures to prevent recurrence.

Specialization lets each agent be good at one thing

A single general-purpose agent trying to be an expert on Kubernetes, Terraform, AWS security, and monitoring is a jack-of-all-trades-master-of-none. It makes trade-offs internally: spend more tokens on infrastructure, less on security? More on Kubernetes, less on application logs?

Specialized agents don't make those trade-offs. A Kubernetes Detective has 100% of its context budget for Kubernetes. It knows the exact names of API objects, understands probe configurations deeply, and can validate against the actual cluster schema.

This shows up in the details:

Kubernetes Detective catches that .spec.containers[].livenessProbe is missing
Terraform Agent verifies that the Terraform module contract is satisfied
Security Agent enforces that ingress rules don't use 0.0.0.0/0 on restricted ports
Observability Agent ensures that every critical metric has a corresponding alarm

A single agent might catch one of these. Specialized agents catch all of them, reliably.

The incident response case: why multi-agent becomes essential

Production incidents are where multi-agent orchestration shows its value most clearly. Consider a real scenario:

The problem:

Kubernetes pods restart every 2-3 minutes
Users report 40% error rate
No clear root cause

Single-agent approach (traditional):

Prompt: "Analyze these logs and manifests"
Response: "Memory limit might be low. Also check HPA config."
You manually verify, make changes, wait for results
Repeat

Multi-agent orchestration:

Coordinator spawns five agents simultaneously
Each analyzes their domain in parallel:
- Kubernetes Detective reads deployment.yaml: finds memory limit 256Mi, app needs 400Mi+
- Terraform Agent reads AWS config: finds database security group open to world
- Observability Expert reads metrics: finds no CloudWatch alarms despite high memory pressure
- Security Agent audits IAM: finds overly permissive roles
- Incident Manager collates findings into RCA
All five propose changes
Promotion gate validates each proposal
All approved changes apply simultaneously
One RCA document, not five reports to reconcile

Wall-clock time: ~5 minutes. Human intervention: one approval batch.

Building a multi-agent system: five components

1. The coordinator

The coordinator is a state machine, not a reasoning engine. It:

Accepts a goal ("investigate this incident, fix critical issues, generate RCA")
Maps the goal to agent tasks and dependencies
Spawns agents with scope constraints
Collects proposals from all agents
Routes proposals to the promotion gate
Tracks convergence (are all agents done? are all decisions made?)
Synthesizes final output

The coordinator runs as a persistent process, not inside a request/response cycle. It can wait for human approval without timing out. It can retry a failed agent without losing the work from other agents.

2. Agents (specialized, scoped)

Each agent is similar to a single-agent system, but narrower:

Kubernetes Detective:

Read access: all .yaml files in kubernetes/
Write access: kubernetes/deployment.yaml, kubernetes/hpa.yaml (after approval)
Domain knowledge: Kubernetes API, best practices, common misconfigurations
Example task: "Analyze this deployment, identify issues, propose fixes"

Terraform Agent:

Read access: all .tf files in terraform/
Write access: any .tf file (after approval)
Domain knowledge: Terraform modules, state management, AWS best practices
Example task: "Audit this Terraform for security issues, cost, and best practices"

Security Agent:

Read access: all configuration files
Write access: none (proposals only)
Domain knowledge: IAM, security groups, encryption, compliance
Example task: "Find all security issues in this infrastructure"

Observability Expert:

Read access: metrics, logs, alarm configuration
Write access: CloudWatch alarms, dashboards (after approval)
Domain knowledge: CloudWatch, Prometheus, threshold tuning, alert design
Example task: "Design alarms for this service"

Incident Manager:

Read access: all proposals from other agents, logs, metrics
Write access: analysis/root-cause-analysis.md, analysis/remediation-checklist.md
Domain knowledge: RCA structure, incident communication
Example task: "Synthesize all findings into a coherent incident report"

Each agent is specialized enough to be good at its job, but constrained enough that it can't accidentally do something outside its domain.

3. The promotion gate

The promotion gate decides: does this change go into durable state?

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
class PromotionGate:
    def evaluate(self, proposal, agent, confidence):
        # 1. Confidence check
        if confidence < 0.7:
            return Decision.HOLD_FOR_REVIEW
        
        # 2. Scope check
        if proposal.file_path not in agent.write_paths:
            return Decision.REJECTED
        
        # 3. Policy check
        if not self.passes_policy(proposal):
            return Decision.REJECTED
        
        # 4. Deduplication
        if self.already_proposed(proposal):
            return Decision.REINFORCE_EXISTING
        
        # 5. Approval routing
        if proposal.type in ["security", "cost", "policy"]:
            return Decision.REQUIRES_HUMAN_APPROVAL
        else:
            return Decision.APPROVED

The gate is the highest-risk component. Promote everything and the system fills with speculation. Promote nothing and the agent can't learn. The gate balances these by:

Logging every decision (approved, rejected, held)
Deduplicating by content hash
Separating confidence from approval (high confidence doesn't mean auto-approve)
Requiring human approval for high-impact changes
Durable state stores

Multi-agent systems need real databases, not just files:

Proposal store (DynamoDB):

1
2
3
4
5
6
PK: proposal_id | SK: agent_id
- proposed_change: {...}
- confidence: float
- state: approved | rejected | held
- created_at: timestamp
- approved_by: user_id

Decision log (S3):

1
2
3
s3://incident-responses/{incident_id}/decisions.jsonl
{ "proposal_id": "...", "decision": "approved", "reason": "...", "timestamp": "..." }
{ "proposal_id": "...", "decision": "applied", "timestamp": "..." }

Generated artifacts (S3 + git):

1
2
3
4
5
6
s3://incident-responses/{incident_id}/
  ├── deployment-fixed.yaml
  ├── terraform-audited.tf
  ├── root-cause-analysis.md
  ├── preventive-actions.md
  └── runbook.md

Every artifact carries metadata. Every decision is logged. The entire incident response is auditable.

Approval workflow

Approval can't be ad-hoc. It needs rules:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
approval_rules:
  security_changes:
    - type: "security_group_rule"
      action: "remove_ingress"
      auto_approve: true
      reason: "Removing overly permissive rules is always safe"
    
    - type: "iam_policy"
      action: "restrict"
      auto_approve: true
      reason: "Restricting permissions is always safe"
    
    - type: "iam_policy"
      action: "expand"
      auto_approve: false
      reason: "Expanding permissions needs human review"
  
  resource_changes:
    - type: "memory_limit"
      from: "256Mi"
      to: "512Mi"
      cost_impact: "minimal"
      auto_approve: true
    
    - type: "database_instance_class"
      from: "t3.micro"
      to: "t3.small"
      cost_impact: "moderate"
      auto_approve: false

The incident response journey: when multi-agent makes sense

You don't need multi-agent orchestration on day one. A working progression looks like this:

Stage 1: Single agent, command-line tasks

1
2
3
"Write me a Terraform module for VPC"
→ Agent writes module.tf
→ You review, apply manually

This works fine for one-off tasks.

Stage 2: Single agent, file analysis

1
2
3
4
"Audit my Kubernetes deployment for best practices"
→ Agent reads deployment.yaml
→ Suggests changes
→ You apply them

Faster than manual review, but still human-driven.

Stage 3: Single agent with approval gates

1
2
3
4
5
"Find security issues in my infrastructure and propose fixes"
→ Agent analyzes all files
→ Proposes changes with confidence levels
→ High-confidence changes auto-apply
→ Low-confidence changes need approval

Works well for single-domain tasks.

Stage 4: Multi-agent, parallel investigation

1
2
3
4
5
6
7
"We have a production incident. Investigate and fix it."
→ Five agents work in parallel
→ Each proposes changes in their domain
→ Coordinator collects proposals
→ Approval gate filters them
→ All approved changes apply simultaneously
→ Incident manager synthesizes RCA

This is where you stop losing time to sequential handoffs.

Stage 5: Continuous multi-agent oversight

1
2
3
4
5
6
Every night, orchestrate a full infrastructure audit:
- Kubernetes Detective: check all pods, alerts
- Terraform Agent: validate IaC, check for drift
- Security Agent: find policy violations
- Observability Expert: verify alarms are configured
- Generate morning report with findings & recommendations

This is where human time becomes the bottleneck, not AI capability.

Most teams hit "Stage 4 is necessary" exactly when they have:

Parallel expertise required — incident response needs security + infrastructure + observability simultaneously
High cost of sequential handoff — "wait for one agent to finish, then the next" is too slow
Multi-domain scope — incident touches Kubernetes, Terraform, CloudWatch, and application code
Durable decision trail — you need to know who approved what and why

If your incident investigations are single-domain (just Kubernetes, just Terraform), a single agent is fine. If they're "first debug Kubernetes, then audit Terraform, then check CloudWatch," multi-agent starts looking like an efficiency multiplier.

Common signals that multi-agent is right for you

Parallelizable work is being done sequentially

If incident response looks like:

Debug Kubernetes (20 min)
Audit Terraform (15 min)
Check security groups (10 min)
Design alarms (15 min)

That's 60 minutes of wall-clock time. In parallel, it's 20 minutes. Multi-agent changes the bottleneck from "how fast can one agent think" to "how long is the slowest parallel task."

Decisions get re-audited across teams

If your approval process is "Kubernetes team reviews, then security team reviews, then ops reviews," multi-agent can consolidate that into one approval batch where each specialist agent has already done its review.

Specialized knowledge is being asked for generalizations

When your Terraform expert is asked to evaluate Kubernetes configuration (because the single agent needs to be expert on everything), something is being done inefficiently. Specialized agents let each expert be expert.

Scope enforcement is hard to get right in code

If every microservice, every team member, and every CI/CD system is responsible for remembering "only touch your files," eventually someone forgets. Multi-agent systems move scope enforcement to the coordinator and the database, not the prompt.

Incident investigations take hours because of sequential handoffs

This is the big one. If debugging a production incident requires calling people sequentially ("Kubernetes team, then infra team, then security team, then observability team"), multi-agent can compress that to parallel work and one approval meeting.

Getting started: a minimal multi-agent incident response system

You don't need to build everything at once. A working v1 system looks like this:

Build the coordinator

A state machine that:

Takes an incident description
Spawns N agents with clear task boundaries
Collects proposals from all
Routes to approval gate
Applies approved changes

1
2
3
4
5
6
7
8
9
10
11
12
13
14
class IncidentOrchestrator:
    def investigate(self, incident_description):
        tasks = [
            Task("kubernetes_detective", "Analyze deployment for issues"),
            Task("terraform_agent", "Audit infrastructure for security"),
            Task("security_agent", "Find IAM and network risks"),
            Task("observability_expert", "Check if alerts are configured"),
        ]
        
        proposals = self.run_parallel(tasks)
        decisions = self.promotion_gate.evaluate_all(proposals)
        self.apply_approved(decisions)
        
        return self.incident_manager.generate_rca(proposals, decisions)

Define agent scope boundaries

Clear rules: which files can each agent read? Which can it write? What's it an expert on?

Implement the promotion gate

Start simple:

Confidence >= 0.8 → auto-approve syntax changes
Confidence >= 0.9 → auto-approve security rule removal
Everything else → needs approval

Add durable state

Use DynamoDB for proposals, S3 for artifacts. Every decision is logged. Every artifact is traced back to an agent and a decision.

Run in shadow mode first

Don't auto-apply anything. Collect proposals, review them, then decide. This is where you learn what your agents are actually good at before you trust them to change production.

Expand to continuous operation

Once incident response works, run the same orchestration every night as an audit, generating a morning report instead of a response to an active incident.

Conclusion

The difference between a single AI agent and multi-agent orchestration isn't speed alone — it's a shift in what's possible. Single agents work inside a context window, reason sequentially, and can't specialize deeply without losing generality. Multi-agent systems coordinate specialists, work in parallel, and enforce scope and approval at the system level instead of the prompt level.

For DevOps teams, this means incident investigation that takes minutes instead of hours, infrastructure audits that cover all domains simultaneously, and a durable record of what changed, why, and who approved it. None of it requires exotic infrastructure — it's Kubernetes, Terraform, DynamoDB, S3, and one coordinator orchestrating them.

The signal to start building multi-agent systems is the same signal that leads to specialized teams in the first place: when the problem is big enough that one person (or one agent) can't hold all of it in their head at once, coordination becomes essential. Multi-agent orchestration is what makes that coordination actually scale.

# multi-agent-orchestrator # devops # agentic-ai # security-identity # ai

Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.

Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article

Builder Center

KiroCrew: From Single Agents to Multi-Agent Orchestration: Building Production DevOps Automation on AWS

Roman Ceresnak

The fundamental shift: orchestration, not chaining

Single agent (traditional):

Multi-agent orchestration (new):

Five operational properties that change with multi-agent systems

Scope enforcement moves to the coordinator

Work is delegated, not sequentially queued

State is durable and scoped, not ephemeral

Convergence and approval are explicit gates, not implicit

Specialization lets each agent be good at one thing

The incident response case: why multi-agent becomes essential

Building a multi-agent system: five components

1. The coordinator

2. Agents (specialized, scoped)

3. The promotion gate

Logging every decision (approved, rejected, held)Deduplicating by content hashSeparating confidence from approval (high confidence doesn't mean auto-approve)Requiring human approval for high-impact changesDurable state stores

Approval workflow

The incident response journey: when multi-agent makes sense

Stage 1: Single agent, command-line tasks

Stage 2: Single agent, file analysis

Stage 3: Single agent with approval gates

Stage 4: Multi-agent, parallel investigation

Stage 5: Continuous multi-agent oversight

Common signals that multi-agent is right for you

Parallelizable work is being done sequentially

Decisions get re-audited across teams

Specialized knowledge is being asked for generalizations

Scope enforcement is hard to get right in code

Incident investigations take hours because of sequential handoffs

Getting started: a minimal multi-agent incident response system

Build the coordinator

Define agent scope boundaries

Implement the promotion gate

Add durable state

Run in shadow mode first

Expand to continuous operation

Conclusion

Logging every decision (approved, rejected, held)
Deduplicating by content hash
Separating confidence from approval (high confidence doesn't mean auto-approve)
Requiring human approval for high-impact changes
Durable state stores