From DevOps to AgentOps: Building an AI-Powered Kubernetes Incident Response System on AWS
What if an AI agent could investigate a Kubernetes incident, analyze logs and metrics, identify the probable root cause, and recommend the next action? In this article, I explore a production-oriented architecture using Amazon Bedrock, Bedrock AgentCore, Amazon EKS, observability, and security guardrails—and explain why AI agents should investigate first and automate changes only with appropriate controls.
From DevOps to AgentOps: Building an AI-Powered Kubernetes Incident Response System on AWS
Imagine this.
It is 2:13 AM.
A production application is returning 5xx errors.
The Kubernetes Pods are technically running, but users are experiencing failures.
A DevOps engineer starts investigating:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
Check Pods
↓
Check Services
↓
Check Gateway
↓
Check application logs
↓
Check metrics
↓
Check recent deployments
↓
Check traces
↓
Find the probable root cause
↓
Decide what to do
Now imagine an AI agent that can perform the investigation, correlate the available evidence, explain what it found, and recommend the next action.
That is where I think the combination of DevOps and Agentic AI becomes particularly interesting.
The goal is not to give an AI unlimited access to production.
The goal is to build an agent that can investigate first, explain its reasoning, and operate within carefully defined permissions and approval boundaries.
Why Agentic AI Is Interesting for DevOps
Traditional AI applications usually follow a relatively simple pattern:
1
2
3
4
5
6
7
User
↓
Application
↓
LLM
↓
Response
An agentic system can go further:
1
2
3
4
5
6
7
8
9
10
11
12
13
User
↓
Agent
↓
Plan
↓
Use tools
↓
Collect information
↓
Analyze results
↓
Take or recommend an action
For DevOps, this opens interesting possibilities.
An incident-response agent could potentially:
- Inspect Kubernetes resources
- Query application logs
- Analyze metrics
- Examine traces
- Check recent deployments
- Compare the current state with the expected state
- Identify suspicious changes
- Suggest remediation
- Ask for human approval before making high-impact changes
Instead of replacing the DevOps engineer, the agent becomes an additional troubleshooting layer.
A Possible AWS Architecture
A production-oriented design could look like this:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
┌─────────────────────┐
│ Engineer │
└──────────┬──────────┘
│
▼
┌─────────────────────┐
│ Incident Agent │
└──────────┬──────────┘
│
┌──────────────┼──────────────┐
│ │ │
▼ ▼ ▼
┌──────────┐ ┌──────────┐ ┌──────────┐
│ Logs │ │ Metrics │ │ Traces │
└──────────┘ └──────────┘ └──────────┘
│ │ │
└──────────────┼──────────────┘
▼
┌─────────────────────┐
│ Amazon EKS │
│ │
│ Applications │
│ Services │
│ Envoy Gateway │
└─────────────────────┘
AI / Agent Layer
│
▼
┌───────────────────────┐
│ Amazon Bedrock │
│ AgentCore │
└───────────────────────┘
This is not a recommendation to put every component into every application.
The architecture should depend on the problem being solved.
A smaller incident-response workflow may need only one agent and a few tools.
A larger platform may require multiple specialized agents.
Why Amazon Bedrock?
Amazon Bedrock provides access to foundation models and capabilities for building generative AI applications.
For an incident-response system, the model is only one component.
The more interesting part is the connection between the model and operational tools.
For example:
1
2
3
4
5
6
7
8
9
10
11
Agent
│
├── Kubernetes tool
│
├── Log search tool
│
├── Metrics query tool
│
├── Deployment history tool
│
└── Documentation search
The agent can use these tools to gather evidence instead of trying to answer everything from its training data.
This distinction is extremely important.
An LLM may know what a Kubernetes CrashLoopBackOff means.
But it does not automatically know:
"Why is this specific Pod in my cluster currently restarting?"
For that, it needs access to current operational data.
Where Bedrock AgentCore Fits
Building an AI agent is relatively easy.
Operating an agent reliably in a production environment is much harder.
This is where managed agent infrastructure becomes interesting.
Amazon Bedrock AgentCore provides capabilities intended to help developers build, deploy, and operate AI agents.
For a DevOps use case, this changes the problem from:
"How do I build an LLM chatbot?"
to:
"How do I safely operate an AI system that can interact with real infrastructure?"
That is a much more interesting engineering problem.
The Agent Should Not Have Unlimited Kubernetes Access
This is probably the most important design decision.
Giving an AI agent unrestricted cluster-admin access would be a terrible production design.
Consider this:
1
2
3
4
5
Agent
↓
kubectl
↓
cluster-admin
If something goes wrong, the blast radius can be enormous.
Instead, the agent should have narrowly scoped permissions.
For example:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
Read Pods
Read Services
Read Events
Read Deployments
Read ConfigMaps
Read Logs
↓
Investigate
↓
Recommendation
↓
Human approval
↓
Controlled action
This follows an important principle:
The agent should have the minimum permissions required for its job.
Read-Only First
If I were designing the first version of this system, I would make the agent completely read-only.
For example, the agent could answer:
"Why is the checkout service returning 5xx errors?"
The agent could investigate:
1
2
3
4
5
6
7
8
9
1. Pod status
2. Pod events
3. Container restart count
4. Application logs
5. Service endpoints
6. Gateway configuration
7. Recent deployment
8. CPU and memory metrics
9. Distributed traces
Then it could produce something like:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
Incident Summary
Service:
checkout
Observed problem:
HTTP 5xx responses increased after the latest deployment.
Evidence:
- New ReplicaSet created 8 minutes ago
- Error rate increased immediately afterward
- Application logs show database connection failures
- Pod CPU is normal
- Pod memory is normal
Probable root cause:
Database connection configuration changed in the latest deployment.
Recommended action:
Review the database connection configuration and consider
rolling back the latest deployment after human verification.
Notice something important.
The agent did not immediately execute a rollback.
It investigated first.
Connecting Observability to the Agent
This is where the architecture becomes particularly powerful.
A modern Kubernetes environment can generate several types of operational data.
Metrics
Metrics answer:
What is happening?
Examples:
1
2
3
4
5
6
CPU usage
Memory usage
Request rate
Error rate
Latency
Pod restarts
Logs
Logs answer:
What happened?
For example:
1
2
3
4
Database connection timeout
Authentication failure
Connection refused
Out of memory
Traces
Traces answer:
Where did the request spend its time?
For a distributed application:
1
2
3
4
5
6
7
8
9
Client
↓
Gateway
↓
Frontend
↓
API
↓
Database
A trace can help identify which part of the request path is experiencing latency or failure.
The Agent's Investigation Loop
A useful incident-response agent could follow a structured workflow.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
Incident received
↓
Understand the symptom
↓
Collect cluster state
↓
Query metrics
↓
Search logs
↓
Inspect traces
↓
Check recent changes
↓
Correlate evidence
↓
Generate hypothesis
↓
Validate hypothesis
↓
Recommend action
↓
Human approval
↓
Execute controlled action
The validation step is critical.
The agent should not see one error message and immediately conclude:
"This is the root cause."
It should collect multiple pieces of evidence.
What About Multiple Agents?
For a more advanced architecture, the investigation could be divided between specialized agents.
For example:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
┌─────────────────┐
│ Orchestrator │
└────────┬────────┘
│
┌─────────────────┼─────────────────┐
│ │ │
▼ ▼ ▼
┌─────────────┐ ┌─────────────┐ ┌─────────────┐
│ K8s Agent │ │ Log Agent │ │ Metrics │
│ │ │ │ │ Agent │
└─────────────┘ └─────────────┘ └─────────────┘
│ │ │
└─────────────────┼─────────────────┘
▼
┌─────────────────┐
│ Incident Report │
└─────────────────┘
The Kubernetes agent could inspect cluster resources.
The log agent could search application logs.
The metrics agent could analyze resource usage and application performance.
The orchestrator could combine the findings.
But there is an important trade-off.
More agents do not automatically mean a better system.
More agents can also mean:
- More latency
- More cost
- More communication
- More failure modes
- More security boundaries
- More complex debugging
So I would start with one agent and introduce additional agents only when specialization provides a real benefit.
What Should the Agent Be Allowed to Change?
This is where AgentOps becomes different from a simple chatbot.
Not every action has the same risk.
For example:
| Action | Risk |
|---|---|
| Read Pod status | Low |
| Read logs | Low |
| Query metrics | Low |
| Describe deployment | Low |
| Restart development Pod | Medium |
| Scale production workload | High |
| Roll back production | High |
| Delete database resources | Critical |
The system should treat these actions differently.
A useful policy could be:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
LOW RISK
↓
Agent can execute
MEDIUM RISK
↓
Agent proposes
↓
Approval required
HIGH RISK
↓
Human approval
↓
Controlled execution
CR
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article