AWS DevOps Agent investigations in Kubernetes and OpenSearch
This post shows how I used AWS DevOps Agent to diagnose an incident on a SaaS platform running a silo (single-tenancy) model on Kubernetes. I pointed it at a CloudWatch alarm and had root cause plus a proposed fix, with no human digging through different dashboards or services.
This post shows how I used AWS DevOps Agent to diagnose an incident on a SaaS
platform running a silo (single-tenancy) model on Amazon EKS. I pointed it at a
CloudWatch alarm and had root cause plus a proposed fix, with no human
digging through different dashboards or services.
In a silo model, every customer gets a dedicated Kubernetes namespace containing
all the services provisioned for that tenant. Clean isolation, harder incident
response: when error rates climb, the on-call engineer has to answer four
questions before the incident is over.
- Which tenant(s) are affected?
- Where are the relevant logs, out of dozens of per-tenant indices?
- What actually broke — which service, which version, which error?
- How do you resolve it — which action, can it be rolled back?
Let's see how DevOps Agent worked in my proof of concept.
The environment
I built a proof of concept that mirrors a realistic silo-model platform:

- Amazon EKS cluster with three tenant namespaces (
tenant-alpha,tenant-beta,tenant-gamma), each running an identical stack: an ingress proxy in front of a backend service.
- Application Load Balancer with a per-tenant target group, so HTTP 5xx counts are reported to CloudWatch scoped to each tenant.
- FluentBit running as a DaemonSet, shipping pod logs to Amazon OpenSearch,
indexed per namespace (
app-eks-<namespace>-*).
- CloudWatch alarms on each tenant's 5xx rate, as the incident signal.
- AWS DevOps Agent, connected to the account and to OpenSearch.

The agent reaches CloudWatch and EKS through native integrations — it already
knows how to call those APIs; it just needs an IAM role that permits it.
OpenSearch isn't a native integration, so I connected it through a custom MCP
(Model Context Protocol) server, giving the agent a set of read-only tools to
query tenant logs.
How the agent investigates
I simulated a classic bad deployment: a canary version of one tenant's backend
that passes health checks (so it stays in load-balancer rotation) but throws an
unhandled exception on roughly half of its requests. This produces an
intermittent failure — harder to diagnose than a total outage, and a faithful
stand-in for a canary or rolling deployment gone wrong.
When the alarm fired, I started the investigation manually and let the agent take
it from there. It worked much like a seasoned SRE would:
- Confirmed the symptom. It pulled the CloudWatch metric and saw a sustained 5xx rate — steady, not a transient spike — indicating a persistent backend fault.
- Localized the blast radius. It compared all three tenants and determined that only one was erroring; the others were clean. No false positives.
- Queried the logs. Through the MCP server, it inspected the OpenSearch indices, noticed a distinguishing label on the affected tenant, and retrieved the error logs scoped to that tenant only.
- Identified root cause. It pinpointed the specific pod and version emitting a specific exception on every request, and tied it back to the deployment.
- Proposed a mitigation. It produced a human-readable plan — scale the faulty deployment to zero — with pre-checks, post-validation, and a rollback step.
All of that came in a single pass: no dashboard hopping, no hunting for the right
log index.
The DevOps Agent capabilities I used
- Agent Spaces as a security boundary — each space defines exactly what the agent can access.
- Native AWS integrations for CloudWatch (metrics/alarms) and EKS (pods/namespaces), with access granted via a scoped, assumable IAM role.
- Custom MCP server integration to bring a non-native data source (OpenSearch) into the agent's reach.
- Tool safety classification — the agent runs read-only tools automatically, requires explicit approval for mutating actions, and never executes destructive ones. My PoC ran observe-only.
DevOps Agent benefits beyond this PoC
AWS DevOps Agent does considerably more than I demonstrated here. Teams evaluating
it should know it also offers:
- Human-in-the-loop auto-remediation — with approval gates, the agent can execute the fix it proposes, not just recommend it.
- Multi-account and AWS Organizations support — cross-account IAM roles let one Agent Space investigate across many accounts, with fine-grained scoping.
- A broad set of native integrations — beyond CloudWatch and EKS, it natively connects to popular observability and operations tools (for example Datadog, Dynatrace, New Relic, Splunk) and collaboration/ticketing systems (for example Slack, ServiceNow, PagerDuty).
- Extensibility via MCP — anything not natively supported (OpenSearch, Prometheus/Grafana, in-house tools) can be connected through an MCP server with read-only access.
- Multicloud and on-premises reach — investigations aren't limited to AWS; any system reachable via MCP can participate.
- Continuous learning — the agent builds an understanding of your applications and their relationships, so investigations get sharper over time.
- Post-incident reporting — it can generate operational reports with preventative recommendations across testing, observability, and infrastructure.
Conclusion
Let me know in the comments what you think about AWS DevOps Agent, and your use cases!
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article