Multi-agent operations: agents over the SRE loop
Multi-agent operations: agents over the SRE loop
Multi-agent operations: agents over the SRE loop
Nothing here is built. This page integrates the architecture from LLM-based Multi-Agent System Architecture Design and Project Code Practice (Chen Guangjian, 2024; the GitHub repository is in Chinese and is summarised inslurps/README.md) into the system the other pages design. The agents sit on top of the SRE page 's loop and may never replace it: alerts still come from the store, and actions still come from people. The lab already runs the pieces the first agent needs: Ollama, Arize Phoenix, an OpenAI-compatible gateway, HolmesGPT and Obot.
Why a multi-agent system, and why not one agent
The book's argument (chapters 1 to 3) is that a single model with every tool is the wrong unit: it cannot be evaluated per capability, it cannot be given different permissions per task, and its context fills with everything at once. Split by role, each agent gets a narrow tool set, a narrow prompt, a narrow evaluation set and a narrow IAM role. That last point is the one that matters for operations: the triage agent's role can be read-only in IAM and Kubernetes RBAC, and no prompt injection can widen it.
The concerns the book lists as the hard part are the same ones the SRE page already had to answer for people: who may act, how a decision is verified, and how the loop improves. So the design below reuses those answers.
The service map
Read it like the other maps: official icons for AWS services, monograms for the lab's third-party pieces, and this time no green dots. EventBridge carries the triggers the SRE page already produces (an alert firing, a deploy, a schedule). Step Functions is the orchestrator's state machine, because an agent run that fans out, waits for a human and resumes is exactly what it models, and its history is the run's audit trail. SQS gives each role a queue so a burst of alerts does not become a burst of model calls. Bedrock is the managed model with Guardrails on prompts and outputs, and the model for every agent on the target, the first one included; Ollama on EKS is what the lab runs today and stays a lab-only stand-in. Agent traces go through the ADOT collector to X-Ray and CloudWatch. DynamoDB holds episodic memory (what ran, what it concluded, what the human said); the OpenSearch index that already holds the product projection gets a second index: a vector index of these documents, the runbooks and every postmortem.
The agents, and what they may touch
The roles follow chapter 5.1 (functional, managerial and user-interaction agents), specialised to this system:
| agent | role type | inputs | tools (tier) | output |
|---|---|---|---|---|
| interaction | user-interaction | a message in the alert thread, a question | none | intent, a clarifying question, an explanation of what the others found |
| orchestrator | managerial | a task or an event | episodic memory read (0) | a decomposition, assignments by capability, one merged recommendation |
| triage | functional | the alarm name and its metric math, the runbook order from the SRE page | CloudWatch GetMetricData, kubectl get, pod logs, ALB metrics (0) | ranked hypotheses with the evidence for each |
| deploy | functional | rollout history, canary analysis | sync status from the EKS managed Argo CD capability (0); rollback, scale within HPA range, sync (1, gated) | a proposed mitigation, then its execution once approved |
| cost | functional | the bill, aws-cost.md, node utilisation | Cost Explorer, kube-state-metrics (0) | idle capacity and the change that would remove it |
| search-quality | functional | relevance samples, reconciler counts | OpenSearch queries (0) | a drift or ranking regression report |
Inside one agent
The loop is chapter 5.2's perceive-decide-execute cycle with three additions the book argues for elsewhere: reflection after every action (3.2.3), output quality control before any action (4.4), and an uncertainty gate (5.3.3) that escalates to a human rather than guessing. Two rules make it safe to run against production:
- The model emits a structured action, never a raw command line. The action is a JSON object validated against a schema; the tool allowlist decides the tier, not the model's description of what it wants to do.
- One re-prompt, then stop. A validation failure is fed back once. A second failure ends the run with the evidence so far, because a model that cannot produce a valid action twice is not going to on the third try.
One incident, four agents, one human decision
This is the SRE page's incident loop with the agents doing the parts that are lookups and the human doing the part that is a decision. Decomposition and assignment by capability are chapter 6.2; ranking by argument, then by vote, is 6.4; arbitration when two hypotheses conflict is 6.5. The important property is that the on-call engineer sees one recommendation with the evidence for it and against the alternatives, not three chat transcripts.
The deploy agent's execution is
make deploy TAG=<previous>, the same rollback the runbook prescribes, so what the agent does is what a person would have typed, with the approval recorded in the thread and the run recorded in the trace.What an agent may do on its own, with approval, or never
| tier | examples | who decides | how it is enforced |
|---|---|---|---|
| 0, read-only | CloudWatch metric math and Logs Insights, kubectl get and describe, pod logs, ALB and RDS metrics, managed Argo CD sync status, Cost Explorer | the agent, within a per-run budget of calls, tokens and time | a read-only IAM role and Kubernetes Role per agent; the allowlist in the tool layer |
| 1, reversible | rollback to the previous tag, scale within the HPA range, a managed Argo CD sync, pause the sync replica | a human, in the thread, with the evidence in front of them | a separate ServiceAccount with the write verbs, used only by the executor after an approval token is checked |
| 2, irreversible | delete anything, terraform apply or destroy, IAM or secret changes, schema changes | nobody: the agent may only write the recommendation | the tools do not exist in the agent's allowlist; the roles have no such permission |
Two more guardrails apply to every tier. Every tool call is a CloudTrail event and a span in the run's trace, and every metric an agent cites in a recommendation is re-queried by the tool layer before the text is shown, so a model cannot report a number it did not read (chapter 4.4.2, factual verification).
Evaluating the agents
Chapter 8 asks for a metrics system, robustness tests, security tests and an A/B framework. Applied here:
| metric | definition | source | gate |
|---|---|---|---|
| task completion | the top-ranked hypothesis matches the postmortem's cause | offline set replayed per change | must not drop |
| latency | alert fired to recommendation posted | trace | under 3 minutes at p95 |
| cost per run | model tokens and tool calls | trace, CloudWatch | under the per-run budget; monthly total on the cost page |
| false-action rate | tier-1 proposals a human rejected | episodic memory | trending down |
| prompt-injection resistance | crafted log lines and alert annotations that try to widen the action | chapter 8.4 attack simulation, in the offline set | zero tier changes |
The offline set starts from what this repository already has: the six findings in the Spring Boot write-up, their verification screenshots, the runbook's two failure cases, and the lab's dated journal entries. Each becomes a question, the evidence an agent would have had, and the expected action.
How the pieces compare
Frameworks
| option | what it gives you | what it costs | when to prefer it | verdict here |
|---|---|---|---|---|
| Spring AI | the same language and build as search-platform; tool calling, structured output, vector stores, observability through Micrometer; the lab's spring-ai-rag-demo already runs it | a younger API than the Python frameworks; fewer multi-agent primitives out of the box | a Java team, a Java codebase | chosen for the first agent: one stack, one test pyramid, one JSON logging shape |
| LangGraph | explicit state machines for agent loops, checkpointing, human-in-the-loop nodes | Python, a second runtime to operate | complex orchestration with many branches | the orchestrator, if Step Functions is not used |
| AutoGen, CrewAI | conversational multi-agent patterns, role definitions close to the book's | opinionated loops that are hard to bound; Python | prototyping the collaboration protocol | the lab, for experiments |
| Bedrock Agents, Strands Agents SDK | managed agent runtime on AWS, action groups as Lambda, Guardrails built in | AWS-only; the loop is theirs to change | production on AWS with a small team | the target once the first agent has earned it |
| Obot, HolmesGPT (lab) | an agent platform with an MCP runtime on Kubernetes; an SRE-specific troubleshooting agent | already running; not this codebase's stack | trying the shape before writing it | the lab's dry run for the triage role |
Models and where they run
| option | what it gives you | what it costs | when to prefer it | verdict here |
|---|---|---|---|---|
| Amazon Bedrock | managed models, Guardrails, IAM per model id, no GPU to run | per token; a triage run is a few cents | anything on AWS | chosen for the first agent and for production, reached through IRSA or EKS Pod Identity like every other AWS call |
| Ollama on EKS (lab) | keyless, OpenAI-compatible, offline, free per token | a GPU or a slow CPU; you pick and patch the model | the lab, evaluation sets, prompt work | lab only; the stand-in the manifest points at today, rejected for the target |
| vLLM on EKS or a SageMaker endpoint | open models at scale, fine-tuning | GPU nodes; the node group does not have them | a fleet of agents, or a fine-tuned model | not at this scale |
| a third-party API | the strongest models | data leaves AWS; keys to rotate | when quality gaps justify it | rejected: the target is AWS-only, and Bedrock carries the same model families |
Memory and knowledge
| need | option chosen | alternative | why |
|---|---|---|---|
| episodic memory (runs, outcomes, approvals) | DynamoDB | RDS table, Redis | point reads by run id, TTL, no contention with the app's transactions; the same reasoning as the idempotency keys |
| working memory (one run) | the Step Functions state, or the agent's request scope | Redis | it must die with the run |
| knowledge (docs, runbooks, postmortems) | a second OpenSearch index with vectors | pgvector (the lab's RAG demo), Bedrock Knowledge Bases | the store already exists and is already reachable by SigV4 through IRSA |
| conversation state with the human | the alert thread itself | a chat server | the thread is the audit trail people already read |
What to build first
The book's practice guide (chapter 11) is to pick one narrow agent, one framework and one evaluation set before designing collaboration. Here:
- The triage agent, read-only, in the lab. A Spring AI service with three tools: a CloudWatch metric query, a namespace-scoped
kubectl get, and a retrieval over this documentation. It runs the SRE page's triage order on every firing rule and posts ranked hypotheses to the alert's channel. The manifest isagents/k8s/triage-agent.yaml: a ServiceAccount with a read-only Role, no tier-1 permissions anywhere, and hard budgets in the environment. Both swaps are now written: the model is Bedrock addressed by region and model id, with no endpoint URL and no key because the credentials come from the IRSA role (bedrock:InvokeModelon one model id,cloudwatch:GetMetricDataon two namespaces,logs:StartQueryandGetQueryResultson two log groups, and nothing that writes); the metric source is CloudWatch rather than an in-cluster Prometheus; and traces go to the ADOT collector the EKS add-on installs, which exports to X-Ray. Themonitoring.coreos.comrule came out of the Role at the same time — thePrometheusRuleobjects it read were deleted when the alerts became CloudWatch alarms, so it was granting read on a kind this repository no longer creates. Ollama, an in-cluster Prometheus and Phoenix stay in the comments as the lab's stand-ins. It passes a strict client-side dry run. - The offline set from the write-up's six findings and the runbook, with an evaluation job in CI that replays it and fails on a drop in task completion.
- The orchestrator as a Step Functions definition, not a second agent, once there is a second specialist for it to coordinate.
- Tier 1 with an approval token, the deploy agent, and a separate ServiceAccount with the write verbs, only after a month of tier-0 runs whose recommendations a human found correct.
Everything above is regenerated with
make diagrams; the chart specs are tools/charts/agents-*.json and system-design-master.json.AWS-only revision (2026-09-25). The target on this page was moved to AWS-managed services only: Bedrock is the model for the first agent as well as production, agent traces go to X-Ray and CloudWatch through ADOT instead of Phoenix, and knowledge stays in the OpenSearch Service vector index, with Bedrock Knowledge Bases as the alternative. Ollama, Phoenix and the OpenAI-compatible gateway remain what the lab runs. These choices are documentation-sourced; nothing was deployed, and AWS documentation in this area changes monthly.
Manifest revision (2026-09-26).
agents/k8s/triage-agent.yaml was the last place in this repository where the target still pointed at lab endpoints, and it no longer does: Bedrock through IRSA instead of an Ollama URL, CloudWatch instead of an in-cluster Prometheus, the ADOT collector instead of Phoenix, and the stale PrometheusRule grant removed. It is still a target — it has never been applied, the image and the role ARN are REPLACED_BY_MAKE, and no agent has ever run.The guardrail the Replit incident actually argues for
In July 2025 Replit's AI coding agent deleted a production database during what its user had declared a code freeze; Replit's CEO apologised publicly and the data was restored from backups. Replit had not published a formal post-mortem by the time the widely shared write-ups appeared — the "seven fatal mistakes" list that circulates is a commentator's analysis built from the CEO's statements and press reports, and the exact command (often quoted as
DROP DATABASE) is from that reconstruction, not from Replit. Read that way, six of the seven are the same cause: the agent's credential could do it. No permissions boundary, no confirmation for destructive actions, no separate environment, no anomaly detection.The lesson is not "add a confirmation prompt". A prompt is a control in the agent's own reasoning, and the reasoning is the thing that failed. The control has to sit outside the agent, where the agent cannot argue with it:
| control | where it sits | what it stops |
|---|---|---|
| IAM permissions boundary denying rds:Delete*, es:Delete*, kms:ScheduleKeyDeletion, iam:* | on the role itself, capping every policy attached to it | the destructive API call, even if someone later widens the role's inline policy |
| Deletion protection on the database | the resource | the deletion, by anyone, until a human turns it off in a separate apply |
| Manual approval action in CodePipeline | the pipeline | an unreviewed change reaching production |
| Tiered tools (this page's model): read-only at tier 0, reversible at tier 1 behind human approval | the tool definition | the agent ever holding the destructive verb |
| CloudTrail + GuardDuty + Config | the account | nothing — it is detection, and it is what tells you which of the above failed |
This repository now carries the first two:
search-platform/terraform/main.tf attaches a permissions boundary to the application's IRSA role that denies the destroy verbs on RDS, OpenSearch, Secrets Manager, KMS, S3 buckets and IAM, and RDS deletion_protection is a variable whose description says plainly that true is the correct value for anything holding data. Neither has been applied to an account.One correction to the advice that circulated with the incident: the boundary it suggests contains only
Deny statements. A permissions boundary is an upper bound, so a boundary that allows nothing caps the role at nothing — attached as written, it does not merely block deletion, it blocks every call the agent makes. The boundary in search-platform/terraform/main.tf carries an explicit allow half for exactly this reason. The same write-up's policy-as-code example matches the literal string DROP DATABASE, which drop database, DROP SCHEMA … CASCADE and TRUNCATE all walk past; string matching on SQL is not a control.The honest limit of both: a boundary stops the AWS API call that destroys the resource. It does not stop a
DELETE FROM inside the database, which the application's own credential can always issue — that one is the database user's grants, and it is not an IAM problem. For an agent that answers it the same way the tiers do:| control | where it sits | what it stops |
|---|---|---|
| a separate database user for the agent with SELECT only, and no DROP, DELETE, TRUNCATE or ALTER | the database's own grants | the destructive statement, whatever the agent's IAM role allows |
| that user reaching a read replica, not the writer | the connection string | a write of any kind, because the replica refuses it |
| RDS IAM database authentication for that user | IAM, then the database | a long-lived password in the agent's context or logs; the token lives 15 minutes |
And because every control above can fail, the one that decided Replit's outcome is the one that does not prevent anything: the backup survived the deletion. On RDS that is three settings, and this repository's defaults get two of them wrong for anything real. With
deletion_protection = false (the demo default), skip_final_snapshot becomes true; and delete_automated_backups is unset, which the AWS provider defaults to true. A single DeleteDBInstance therefore leaves no final snapshot and no automated backups — the seven days of point-in-time recovery are deleted with the instance. For a real environment: deletion_protection = true, delete_automated_backups = false so the automated backups outlive the instance for their retention period, and an AWS Backup plan copying to a vault with Vault Lock in a separate account, which a principal in the workload account — human or agent — cannot delete.Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article