
I deployed kagent on EKS — Here's Every Error I Hit and How I Fixed It
It's 2 AM. PagerDuty fires. A pod is stuck in CrashLoopBackOff. You copy the logs, paste them into a chat AI, apply its suggestion, and get paged again 20 minutes later — because the AI never actually saw your cluster, it only saw what you pasted. That's the pattern kagent is built to break, and this post walks through getting it running for real on an Amazon EKS cluster.
Series: Learn-Agentic-AI-RAG-Vector-by-doing (2 articles)
- 2I deployed kagent on EKS — Here's Every Error I Hit and How I Fixed It This article
📋 Before we jump in let's listed out the topics to be focus as follows :-
- The Problem: You Are the Middleman
- What kagent Actually Is (and Isn't)
- Architecture — Four Layers, One Request Flow
- Why EKS Specifically Needs a Few Things First
- Step 1 — Prepare the EKS Cluster (Storage + Identity)
- Step 2 — Install kagent via Helm
- Step 3 — Configure a Model Provider and Talk to Your First Agent
- kagent vs. the Rest of the Kubernetes AI Toolbox
- Six Real Errors I Hit Getting This Running (and the Exact Fix)
- Hands-on project
- ⚠️ Gotchas Worth Knowing Before Production
- Key Takeaways
🚨 The Problem: You Are the Middleman
The 2 AM scenario above has a name: you've started traditional Copy paste into ChatGPT , installed K8'sGPT & thinking to auto resolve everything or Claude code, what are the actual flaws 🤔
- Human as relay — every diagnosis step needs you to manually shuttle data back and forth.
- Stateless between incidents — the next alert starts from scratch, with no memory of what was tried or rolled back last time.
- Suggests, never acts — you still have to run the command, verify it worked, and decide what's next, alone.kagent's answer isn't a smarter chat window — it's removing the middleman entirely by putting the agent inside the cluster it's diagnosing.
- 🤖 What kagent Actually Is ?
- kagent is a Kubernetes-native framework for building AI agents — Kubernetes is the most popular orchestration platform for running , managing , scaling containerized workloads and kagent makes it easy to build, deploy, and manage AI agents inside it.It's isn't a chatbot that calls your cluster's API from the outside , An "Agent" in kagent is a Custom Resource Definition (CRD) — you
kubectl applyit exactly like a Deployment or a Service, and it shows up in your normal RBAC and audit trail.Kagent ships with an MCP server bundling tools for Kubernetes, Istio, Helm, Argo, Prometheus, Grafana, Cilium, and others — all exposed as Kubernetes custom resources called ToolServers that multiple agents can share.It's declarative, Kubernetes-native, and built to be extensible, so teams add their own agents and tools on top rather than being boxed into a fixed set. It's currently a CNCF Sandbox project originally built by Solo.io.
- Two protocols do the heavy lifting under the hood . a. MCP (Model Context Protocol) provides a standardized interface for agents to invoke tools and services, while the A2A (Agent-to-Agent) protocol enables communication between clients and agents, as well as inter-agent communication.It means under the hood of Kagent has multiple tools belongs to Kubectl , Prometheus , Istio etc to perform diffirent purpose operations .
-

The building blocks behind those layers are also Kubernetes-native resources you'll interact with directly:-
Agent CRD:- The declarative spec — persona, model, and tools an agent is allowed to use, defined in YAML
ModelConfig:- Points an agent at any LLM provider — OpenAI, Anthropic, Azure, or a local model .
ToolServer:- A small service handing agents real tools (kubectl, Helm, Prometheus, Argo, Istio), reusable across every agent, built with kmcp — kagent's own toolkit for MCP.
Controller:- Reconciles every CRD into a running agent — the same control-loop pattern as any Kubernetes operator.
UI & CLI:- Chat with any agent, inspect runs, manage the fleet, from browser or terminal.
A2A layer:- Lets agents call other agents as services, composing specialists into multi-agent workflows .
☁️ Why EKS Specifically Needs a Few Things First ?
kagent will run on any conformant Kubernetes cluster — kind, EKS, AKS, GKE, EKS Anywhere. But EKS has two prerequisites worth handling before you touch the kagent Helm chart, because the chart doesn't manage AWS-side IAM for you .
- A working default StorageClass. kagent's AWS Marketplace listing explicitly calls out that a default StorageClass with a working provisioner is required for the bundled PostgreSQL — on EKS, that means installing the
aws-ebs-csi-driveradd-on first, or pointing kagent at an external PostgreSQL viadatabase.postgres.url.
- An LLM provider API key. kagent does not run any agents until a provider is configured — you create the provider Secret and a ModelConfig in the kagent namespace after install. Both are one-time cluster setup steps, not kagent-specific — but skipping them is the single most common reason a first EKS install stalls with a Postgres pod stuck in
Pending.
- 🛠️ Step 1 — Prep the EKS Cluster (Storage + Identity)
If your EKS cluster already has the EBS CSI driver and OIDC set up, skip to Step 2. If not, here's the current recommended path.
First, make sure your cluster has an IAM OIDC provider (required for IRSA):
eksctl utils associate-iam-oidc-provider \ --cluster=$CLUSTER_NAME \ --approveThen create the IAM role and install the EBS CSI driver as a managed add-on. Use the AWS managed policy for the driver and create the IAM role via
eksctl create iamserviceaccount , then install the add-on itself.CLUSTER_NAME="my-eks-cluster"ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)EBS_CSI_POLICY_ARN="arn:aws:iam::aws:policy/service-role/AmazonEBSCSIDriverPolicyV2"
eksctl create iamserviceaccount \ --cluster=$CLUSTER_NAME \ --namespace=kube-system \ --name=ebs-csi-controller-sa \ --attach-policy-arn=$EBS_CSI_POLICY_ARN \ --approve \ --role-only \ --role-name AmazonEKS_EBS_CSI_DriverRole
aws eks create-addon \ --cluster-name $CLUSTER_NAME \ --addon-name aws-ebs-csi-driver \ --service-account-role-arn arn:aws:iam::$ACCOUNT_ID:role/AmazonEKS_EBS_CSI_DriverRole🧭 IRSA vs. Pod Identity: IRSA vs. Pod Identity — what these terms actually mean:
Both solve the same problem: letting a pod call AWS APIs (like the EBS CSI driver provisioning a volume above) without hardcoding an AWS access key inside the container. The difference is how the pod proves its identity to AWS.
- IRSA (IAM Roles for Service Accounts) — the approach used in the commands above. It relies on your EKS cluster's OIDC provider . AWS trusts tokens issued by that specific cluster's OIDC endpoint, and each IAM role's trust policy has to explicitly list which Kubernetes ServiceAccount is allowed to assume it.
-
- EKS Pod Identity — a newer, simpler mechanism. Instead of OIDC trust policies, an agent runs on every node and a separate "Pod Identity association" (a small AWS-side mapping, not a Kubernetes annotation) tells AWS which ServiceAccount maps to which IAM role. No OIDC provider federation to configure, no cluster-specific trust policy conditions to get right.
Confirm the add-on is healthy before moving on:
kubectl get pods -n kube-system | grep ebs-csi📦 Step 2 — Install kagent via Helm:-
kagent ships as two Helm charts —
kagent-crds and kagent — and the CRDs chart is required to be installed first, a deliberate separation so that helm uninstall kagent later doesn't accidentally wipe your custom resources. 1. Install the CRDs chart firsthelm upgrade --install kagent-crds oci://ghcr.io/kagent-dev/kagent/helm/kagent-crds \ --namespace kagent \ --create-namespace
2. Install the main kagent chart helm upgrade --install kagent oci://ghcr.io/kagent-dev/kagent/helm/kagent \ --namespace kagent** The main installation requires configuring at least one LLM in the provider block where you can choose between OpenAI, Anthropic, Azure OpenAI, Gemini, or Ollama as the default. A minimal values override for Anthropic looks like this:-
providers: default: openai anthropic: provider: openai model: gpt-4-1-mini apiKeySecretRef: kagent-openai apiKeySecretKey: openaiIf you'd rather skip the two-chart dance entirely, kagent is also listed directly on AWS Marketplace — deployable to EKS (and EKS-Anywhere) via a Helm chart or an EKS Add-on, aimed at platform and AI engineering teams, DevOps engineers, and SREs running agentic and LLM workloads on Kubernetes. That path is worth it if your org already standardizes on Marketplace-sourced software for procurement/compliance reasons.
Once installed, verify the core pods are running:-
kubectl get pods -n kagentA healthy install looks roughly like:
kagent-controller , kagent-kmcp-controller-manager , kagent-postgresql , kagent-tools , and kagent-ui . 🔑 Step 3 — Configure a Model Provider and Talk to Your First Agent
With the pods up, create the provider Secret and confirm the ModelConfig registered correctly .
**I am actively contributing the Bedrock part of this Kagent which is underbaked .
Post proper & correct Modelconfig done all of the Kagent pods which is showing CreateContainerConfigError will resolve to Running state .

kubectl create secret generic kagent-openai --from-literal=OPENAI_API_KEY=<your-openai-key> -n kagent
Kagent is also listed directly on AWS Marketplace — deployable to EKS (and EKS-Anywhere) via a Helm chart or an EKS Add-on, aimed at platform and AI engineering teams, DevOps engineers, and SREs running agentic and LLM workloads on Kubernetes. That path is worth it if your org already standardizes on Marketplace-sourced software for procurement/compliance reasons.
The Agentic AI landscape is changing from traditional agenticbot to autonomous Agentic-AI solution which brings to the below topic as follows :-
Tool | Execution Model | Write Access | Scope | Deployment | Best For |
|---|---|---|---|---|---|
kagent | Multi-step agentic loop (Reason → Act → Observe) as long-lived in-cluster Agent CRDs | Direct cluster writes, gated by human-in-the-loop approval per tool call | Kubernetes-native — kubectl, Helm, Istio, Argo, Prometheus; extensible via MCP | Runs inside the cluster as Kubernetes workloads | Closing the loop — detection through an approved, in-cluster fix |
K8sGPT | Rule-based scanner (20+ analyzers); LLM only explains findings | Read-only by design; no writes | Kubernetes-only | CLI tool or in-cluster operator (~30s reconciliation) | Fast, zero-risk first-pass diagnostics |
kubectl-ai | Natural language → kubectl command translation | Human runs the generated command — no autonomous writes | Kubernetes-only | CLI tool | Turning a plain-English request into the right kubectl command |
Six Real Errors I Hit Getting This Running (and the Exact Fix) :-
- Postgres stuck
Pending(missing default StorageClass)
- Controller
CrashLoopBackOff(downstream of #1, not a separate bug)
- Pre-built agents failing on a missing
kagent-openaisecret
- The misleading
"bedrock model config is required"error (really just a missing nestedbedrock.regionfield)
- Helm field-manager conflicts from mixing
kubectl patchwith Helm-managed resources
- ServiceAccount deleted along with a namespace reset, even though the IAM role survivesAll of these fixes step by step commands will be available in the below repo :- https://github.com/Soumya14041987/kagent-on-kubernetes.gitNow access the Kagent UI dashboard from majorly distinguish its features from others :-
kubectl port-forward -n kagent svc/kagent-ui 8080:8080
In right hand panel all listed are the Kubectl tools , similarly there is also prometheus , Helm associated tools as well .
In the below snippet you will found all user interactions with this Kagent UI to not only debug but HITL (HumanIn the Loop) autonomous AI Agent detect, discover & troubleshoot those issues .
Intentionally I create some pods failed as Crashloopbackoff & OOM & expeience as how Kagent detect , debug & troubleshoot those issues :-






⚠️ Gotchas Worth Knowing Before Production
- Don't skip the StorageClass check. If
kagent-postgresqlsits inPending, it's almost always the EBS CSI driver — checkkubectl describe pvc -n kagentfirst.
- Scope RBAC tightly, per agent. A troubleshooting agent doesn't need cluster-admin; tighten the default tools ServiceAccount before shared production use.
- Require HITL approval for destructive actions. Delete, scale-down, and rollback should require explicit approval, configured from day one.
- Trace every tool call with OpenTelemetry. You want an audit trail of what an agent ran, not just the outcome.
- Dry-run before granting write access. Read-only in staging for a couple of weeks, same caution as a new hire's access scope.
- Express constraints as policy, not prompt text. OPA or Kyverno policies are enforceable; system-prompt instructions are suggestions.
- External PostgreSQL for real production. The bundled Postgres is fine for evaluation, but kagent's own docs recommend an external instance for production.
- Pin your chart version. As a CNCF Sandbox project, breaking changes happen between releases — some features require pinning to a minimum chart version, since earlier versions don't expose those values.
🔑 Key Takeaways
- Chat AI is the wrong interface for Kubernetes ops — it's blind to live cluster state, stateless between incidents, and requires a human to relay every step.
- kagent agents are Kubernetes CRDs, not external services — they inherit your existing RBAC and audit trail by default, and communicate via MCP (to tools) and A2A (to other agents).
- On EKS specifically, get the EBS CSI driver, a default StorageClass, and IAM OIDC provider sorted before installing kagent — PostgreSQL storage is the single most common install blocker.
- Most first-run failures are one misleading error away from a one-line fix — a missing
bedrock.regionfield, akubectl patchfighting Helm's ownership, a ServiceAccount lost in a namespace reset. None of it means the tool is broken; it means the CRD schema needs checking, not guessing.
- The spectrum of Kubernetes AI tools runs roughly: K8sGPT (diagnose) → HolmesGPT (investigate) → kagent (autonomous, approved action) — pick based on how much of the loop you want closed.
- Guardrails come first: tight RBAC, human-in-the-loop approval, OTel tracing, and a dry-run staging period — then production.
Series: Learn-Agentic-AI-RAG-Vector-by-doing (2 articles)
- 2I deployed kagent on EKS — Here's Every Error I Hit and How I Fixed It This article
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article