AI Agents on Kubernetes, Part 2: Deploying kagent and Putting Agents to Work
Part 2 of 2. Part 1 built an EKS Auto Mode cluster with eksctl. This part deploys AI agents on it.
Series: Platform enigneering (2 articles)
- 2AI Agents on Kubernetes, Part 2: Deploying kagent and Putting Agents to Work This article
Part 2 of 2. Part 1 built an EKS Auto Mode cluster with eksctl. This part deploys AI agents on it.
In Part 1 we built a cluster that provisions compute on demand, has storage and load balancing built in, and gives pods AWS credentials through Pod Identity. Now we put AI agents on it.
The goal is not a chatbot that knows Kubernetes trivia. It is agents that look at a real cluster, collect evidence, and explain what is wrong. We build two practical use cases:
- A troubleshooting buddy. A read-only agent that diagnoses a crash-looping workload from its spec, events, and logs.
- A multi-agent orchestrator. A coordinator that hands work to specialist agents. One of them understands EKS Auto Mode capacity (NodePools, NodeClaims, limits) and explains why a workload can't be scheduled.
The agent runtime is kagent, a CNCF Sandbox project that defines agents, models, and tools as Kubernetes custom resources. The model is Amazon Nova Pro on Amazon Bedrock, reached without any static AWS keys.
Why kagent
There is no single standard way to run AI agents on Kubernetes yet. Many teams wrap an agent framework in a container and deploy it like any other service. That works, but each team ends up building its own way to configure models, register tools, and expose agents.
kagent's approach suits a platform team: agents are Kubernetes resources.
Resource | What it describes |
|---|---|
ModelConfig | Which LLM provider and model to use, and where its credentials live |
Agent | The agent's instructions (system message), its model, and the tools it may use |
RemoteMCPServer / MCPServer | Tool servers that speak the Model Context Protocol (MCP) |
The kagent controller reconciles these resources into running agents. kagent also uses the open protocols that are gaining real adoption: MCP for tools, and A2A (Agent2Agent) for agents calling other agents. That matters for the long term. If you later move away from kagent, the tools and agent interfaces you built still speak standard protocols.
Because they are plain YAML, your agents get the same lifecycle as everything else on the platform: code review, GitOps, RBAC, and
kubectl describe when something breaks.
Architecture

Two design decisions shape everything else.
The model path. kagent agents call an OpenAI-compatible endpoint served by LiteLLM, running in the cluster. LiteLLM signs requests to Bedrock with temporary credentials from Pod Identity, the mechanism we tested in Part 1. No AWS keys are stored anywhere. We also get one place in the cluster to control model access, change models, and later add budgets or logging. kagent also has a native Bedrock provider, but the proxy keeps credentials out of kagent entirely and gives us one central model endpoint.
The tool path. Agents never talk to the Kubernetes API directly. They call tools on an MCP server, and each agent is allowed only a specific list of tools. The tool server's ServiceAccount is the real security boundary. We look at it closely later.
Prerequisites
- The
agents-democluster from Part 1, with thespot-burstNodePool and thedemonamespace
- Access to the
apac.amazon.nova-pro-v1:0inference profile, confirmed in Part 1, Step 8
-
kubectl,eksctl, AWS CLI v2, and Helm 3.8 or later (for OCI charts)
Set a few variables used throughout:
export AWS_REGION=ap-southeast-1export CLUSTER_NAME=agents-demoexport AWS_ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)export KAGENT_VERSION="0.10.4"cd part-2
Step 1: Give the cluster access to Bedrock without keys
1a. An IAM policy that can only invoke models
{ "Version": "2012-10-17", "Statement": [ { "Sid": "InvokeNovaModelsViaInferenceProfiles", "Effect": "Allow", "Action": [ "bedrock:InvokeModel", "bedrock:InvokeModelWithResponseStream" ], "Resource": [ "arn:aws:bedrock:*:<AWS_ACCOUNT_ID>:inference-profile/*", "arn:aws:bedrock:*::foundation-model/amazon.nova-*" ] } ]}This allows invoking Amazon Nova models and inference profiles, and nothing else: no listing, no fine-tuning, no access to other providers. Fill in your account ID and create the policy:
sed "s/<AWS_ACCOUNT_ID>/${AWS_ACCOUNT_ID}/" iam/bedrock-invoke-policy.json > /tmp/bedrock-invoke-policy.json
aws iam create-policy \ --policy-name agents-demo-bedrock-invoke \ --policy-document file:///tmp/bedrock-invoke-policy.json1b. Associate it with the LiteLLM ServiceAccount
kubectl create namespace kagent
eksctl create podidentityassociation \ --cluster "${CLUSTER_NAME}" \ --region "${AWS_REGION}" \ --namespace kagent \ --service-account-name litellm \ --role-name agents-demo-litellm \ --permission-policy-arns "arn:aws:iam::${AWS_ACCOUNT_ID}:policy/agents-demo-bedrock-invoke"1c. Deploy LiteLLM
LiteLLM needs a master key, so that only clients in the cluster that have it can use the proxy. kagent will use this key as its "API key".
kubectl create secret generic litellm-master-key \ --namespace kagent \ --from-literal=key="sk-$(openssl rand -hex 24)"Here is the manifest. Notice the
model_name in the config: it is the name kagent will ask for, and it is not tied to a specific Bedrock model ID. You can change the model later without editing any agents.# LiteLLM: an OpenAI-compatible proxy in front of Amazon Bedrock.# It authenticates to AWS through EKS Pod Identity (no static keys).apiVersion: v1kind: ServiceAccountmetadata: name: litellm namespace: kagent---apiVersion: v1kind: ConfigMapmetadata: name: litellm-config namespace: kagentdata: config.yaml: | model_list: - model_name: nova-pro # the name kagent will ask for litellm_params: model: bedrock/apac.amazon.nova-pro-v1:0 # APAC cross-Region inference profile aws_region_name: ap-southeast-1 litellm_settings: drop_params: true general_settings: master_key: os.environ/LITELLM_MASTER_KEY---apiVersion: apps/v1kind: Deploymentmetadata: name: litellm namespace: kagent labels: app: litellmspec: replicas: 1 selector: matchLabels: app: litellm template: metadata: labels: app: litellm spec: serviceAccountName: litellm containers: - name: litellm image: ghcr.io/berriai/litellm:main-stable # pin a release tag for production args: ["--config", "/app/config.yaml", "--port", "4000"] ports: - name: http containerPort: 4000 env: - name: LITELLM_MASTER_KEY valueFrom: secretKeyRef: name: litellm-master-key key: key volumeMounts: - name: config mountPath: /app/config.yaml subPath: config.yaml readOnly: true readinessProbe: httpGet: path: /health/readiness port: http initialDelaySeconds: 10 periodSeconds: 10 livenessProbe: httpGet: path: /health/liveliness port: http initialDelaySeconds: 30 periodSeconds: 20 resources: requests: cpu: 250m memory: 512Mi limits: memory: 1Gi securityContext: allowPrivilegeEscalation: false volumes: - name: config configMap: name: litellm-config---apiVersion: v1kind: Servicemetadata: name: litellm namespace: kagentspec: selector: app: litellm ports: - name: http port: 4000 targetPort: httpApply it:
kubectl apply -f 01-litellm.yamlkubectl rollout status deployment/litellm -n kagent --timeout=180sAuto Mode provisions a node for LiteLLM if none has room for it. You can watch it happen with
kubectl get nodeclaims -w , just as in Part 1.1d. Test the model path before adding agents
Test each layer on its own. If something fails later, you will already know the model path works.
kubectl port-forward -n kagent svc/litellm 4000:4000 &sleep 3 # give the port-forward time to connectLITELLM_KEY=$(kubectl get secret litellm-master-key -n kagent -o jsonpath='{.data.key}' | base64 -d)
curl -s http://localhost:4000/v1/chat/completions \ -H "Authorization: Bearer ${LITELLM_KEY}" \ -H "Content-Type: application/json" \ -d '{"model": "nova-pro", "messages": [{"role": "user", "content": "Reply with the single word: pong"}]}' \ | python3 -m json.tool | head -30
kill %1A JSON response with a
choices array means kagent can now reach Bedrock through Pod Identity. If you get an error, see the troubleshooting section before going on.
Step 2: Install kagent
kagent ships as two Helm charts: the CRDs and the controller, UI, and bundled tools.
helm install kagent-crds oci://ghcr.io/kagent-dev/kagent/helm/kagent-crds \ --version "${KAGENT_VERSION}" \ --namespace kagent
helm install kagent oci://ghcr.io/kagent-dev/kagent/helm/kagent \ --version "${KAGENT_VERSION}" \ --namespace kagent \ --set providers.default=openAI \ --set providers.openAI.apiKey=unused-see-bedrock-modelconfigThe chart creates a default
ModelConfig for its bundled sample agents. We won't use it. Our agents point at our own ModelConfig . The chart also installs about ten sample agents and a Grafana MCP server. They are harmless, but to save resources you can turn them off by adding flags such as --set k8s-agent.enabled=false and --set grafana-mcp.enabled=false to the install. helm show values lists them all.Check the install:
kubectl get pods -n kagentkubectl get crds | grep kagent.devkubectl get remotemcpservers -n kagentYou should see the controller, UI, and tool server pods
Running , the kagent CRDs, and a RemoteMCPServer for the built-in tool server. Note its name. Our Agent manifests use kagent-tool-server ; if yours is named differently, update the manifests.

Step 3: Point kagent at Bedrock
# kagent talks to LiteLLM using the OpenAI provider; LiteLLM talks to Bedrock.apiVersion: kagent.dev/v1alpha2kind: ModelConfigmetadata: name: bedrock-nova namespace: kagentspec: provider: OpenAI model: nova-pro # must match model_name in litellm-config apiKeySecret: litellm-master-key apiKeySecretKey: key openAI: baseUrl: http://litellm.kagent.svc.cluster.local:4000/v1kubectl apply -f 02-modelconfig.yamlkubectl get modelconfig bedrock-nova -n kagent -o yaml | tail -20Check the
status conditions. If the controller rejects the resource, it is usually because a field name differs in your kagent version. The CRD schema will tell you the correct names:kubectl explain modelconfig.spec --api-version=kagent.dev/v1alpha2
Use case 1: A troubleshooting buddy
Break something realistic
Here is a failure that happens in real teams. The application reads
DATABASE_URL , but someone wrote the ConfigMap with DB_URL . Nothing fails at deploy time. The container starts, finds the variable empty, and exits.# Intentionally broken: the app expects DATABASE_URL, the ConfigMap provides DB_URL.apiVersion: v1kind: Namespacemetadata: name: demo---apiVersion: v1kind: ConfigMapmetadata: name: checkout-config namespace: demodata: DB_URL: "postgres://checkout@checkout-db.demo.svc.cluster.local:5432/checkout" LOG_LEVEL: "info"---apiVersion: apps/v1kind: Deploymentmetadata: name: checkout-api namespace: demo labels: app: checkout-apispec: replicas: 2 selector: matchLabels: app: checkout-api template: metadata: labels: app: checkout-api spec: containers: - name: app image: public.ecr.aws/docker/library/busybox:1.36 command: - sh - -c - | echo "checkout-api starting (log level: ${LOG_LEVEL})" if [ -z "${DATABASE_URL}" ]; then echo "FATAL: required environment variable DATABASE_URL is not set" >&2 exit 1 fi echo "database configured, serving" sleep 3600 envFrom: - configMapRef: name: checkout-config resources: requests: cpu: 50m memory: 32Mi securityContext: allowPrivilegeEscalation: false runAsNonRoot: true runAsUser: 65532kubectl apply -f 03-demo-checkout-api.yamlkubectl get pods -n demo -l app=checkout-api -wAfter a minute the pods are in
CrashLoopBackOff . You probably already know how to debug this by hand. The point is to see whether the agent reaches the same conclusion, with evidence.Define the agent
apiVersion: kagent.dev/v1alpha2kind: Agentmetadata: name: k8s-troubleshooter namespace: kagentspec: description: >- Read-only Kubernetes troubleshooter. Diagnoses failing workloads using resource specs, events, and container logs. Use it for pods that crash, restart, or fail readiness. type: Declarative declarative: modelConfig: bedrock-nova systemMessage: | You are a Kubernetes troubleshooting assistant working for a platform team.
Rules: - Your tools are read-only. Never claim you changed anything. - Never read or print the contents of Secrets. Referencing a Secret by name is fine. - Gather evidence before concluding: the workload spec, pod status, events, container logs (including previous container logs for restarting pods), and any ConfigMaps the workload references. - Separate what you observed from what you infer.
Answer in this format: 1. Symptom 2. Evidence (which resources and log lines you relied on) 3. Most likely root cause 4. Suggested fix (kubectl commands or YAML for a human to apply) 5. How to verify the fix tools: - type: McpServer mcpServer: name: kagent-tool-server kind: RemoteMCPServer apiGroup: kagent.dev toolNames: - k8s_get_resources - k8s_describe_resource - k8s_get_resource_yaml - k8s_get_events - k8s_get_pod_logsEvery part of this manifest does a job:
-
descriptionis not just documentation. Other agents read it to decide when to delegate to this one, which matters in use case 2.
-
systemMessagedescribes a method, not a personality. It says what evidence to gather, sets a fixed output format, and requires separating what was observed from what was inferred. Without that rule, agents tend to state guesses as facts.
-
toolNamesis an allowlist. The built-in tool server has tools that can apply, patch, and delete resources. This agent gets only the read tools.
kubectl apply -f 04-agent-k8s-troubleshooter.yamlkubectl get agents -n kagentkubectl get deploy,pods -n kagent | grep "troubleshoot"
Ask it
Open the UI:
kubectl port-forward -n kagent svc/kagent-ui 8080:8080Go to
http://localhost:8080 , choose k8s-troubleshooter, and ask:The checkout-api deployment in the demo namespace keeps restarting. Find out why and tell me how to fix it.

How to judge the answer. A good diagnosis will:
- quote the
FATAL: required environment variable DATABASE_URL is not setlog line;
- notice that the Deployment loads the environment from
checkout-config;
- point out that the ConfigMap has
DB_URLbut notDATABASE_URL.
If the agent only says "the container is crashing, check the logs", the evidence-gathering instructions in the system message need tightening. That kind of tuning is normal. Treat the system message like code: change it, test it again, and commit it.
Apply the fix yourself
The agent is read-only on purpose. A human applies the change:
kubectl create configmap checkout-config -n demo \ --from-literal=DATABASE_URL="postgres://checkout@checkout-db.demo.svc.cluster.local:5432/checkout" \ --from-literal=LOG_LEVEL=info \ --dry-run=client -o yaml | kubectl apply -f -
kubectl rollout restart deployment/checkout-api -n demokubectl rollout status deployment/checkout-api -n demokubectl logs -n demo deploy/checkout-apiThe restart is required. Environment variables loaded with
envFrom are read only when the container starts, so updating the ConfigMap alone does nothing for pods that are already running. If the agent didn't mention this in its "how to verify" section, that's a useful addition to its system message.
What the agent can actually reach
The
toolNames allowlist limits what this agent can call. It does not limit what the tool server can do, and every agent that uses the tool server shares its ServiceAccount. Look at it:# Which ServiceAccount does each kagent component run as?kubectl get deploy -n kagent \ -o custom-columns=NAME:.metadata.name,SA:.spec.template.spec.serviceAccountName
# What can the tool server's ServiceAccount do?export KAGENT_TOOLS_SA=<tool server ServiceAccount from the output above>kubectl auth can-i --list --as=system:serviceaccount:kagent:${KAGENT_TOOLS_SA}
kubectl auth can-i get secrets -A --as=system:serviceaccount:kagent:${KAGENT_TOOLS_SA}
If the tool server can read Secrets, any agent with
k8s_get_resource_yaml can pull Secret data into the model's context, and from there it can end up in logs or responses. The "never read Secrets" line in our system message is a polite request, not a control.For a real platform, I'd treat this as the most important hardening step:
- Run a separate read-only tool server for diagnostic agents. Give it its own ServiceAccount bound to the built-in
viewClusterRole, which excludes Secrets by design.
- Keep the write-capable tool server (if you need it at all) for agents that sit behind human approval.
- Bind RBAC to the tool server, not to agents. It is the only component that holds Kubernetes credentials.
This tutorial doesn't take the hardening further, but you should know where the boundary is before you put agents on a cluster that matters.
Use case 2: A multi-agent orchestrator
One agent with every tool tends to do badly: too many tools blur its instructions, and its answers get harder to check. A common alternative is a coordinator that splits the question up and hands each piece to a specialist with a narrow job and a narrow set of tools.
A second problem for the cluster
Re-apply the broken ConfigMap so
checkout-api crash-loops again, and add a workload that cannot be scheduled:# Intentionally unschedulable: asks for 12 vCPU on the spot-burst NodePool,# which is capped at 8 vCPU in total (see part-1/02-nodepool-spot-burst.yaml).apiVersion: apps/v1kind: Deploymentmetadata: name: report-worker namespace: demo labels: app: report-workerspec: replicas: 1 selector: matchLabels: app: report-worker template: metadata: labels: app: report-worker spec: nodeSelector: karpenter.sh/nodepool: spot-burst tolerations: - key: workload-type operator: Equal value: burst effect: NoSchedule containers: - name: worker image: public.ecr.aws/eks-distro/kubernetes/pause:3.7 resources: requests: cpu: "12" memory: 8Gi securityContext: allowPrivilegeEscalation: false runAsNonRoot: true runAsUser: 65532report-worker asks for 12 vCPU on the spot-burst NodePool. In Part 1 we limited that pool to 8 vCPU in total. Auto Mode will not launch a node that breaks the limit, so the pod stays Pending . This is the cost cap working as designed, but to the team that owns the workload it just looks broken.kubectl apply -f 03-demo-checkout-api.yamlkubectl rollout restart deployment/checkout-api -n demokubectl apply -f 07-demo-report-worker.yaml
kubectl get pods -n demokubectl describe pod -n demo -l app=report-worker | tail -15
The capacity specialist
apiVersion: kagent.dev/v1alpha2kind: Agentmetadata: name: capacity-analyst namespace: kagentspec: description: >- Read-only capacity analyst for EKS Auto Mode. Explains why pods are Pending by comparing resource requests, nodeSelectors, tolerations, NodePool requirements and limits, and NodeClaims. type: Declarative declarative: modelConfig: bedrock-nova systemMessage: | You are a capacity analyst for an Amazon EKS Auto Mode cluster. Auto Mode uses Karpenter-style NodePools (karpenter.sh/v1), NodeClaims, and NodeClasses (eks.amazonaws.com/v1).
Rules: - Your tools are read-only. Never claim you changed anything. - For a Pending pod, check in this order: pod events, the pod's resource requests, nodeSelector, tolerations and affinity, the matching NodePool's requirements, taints and limits, and existing NodeClaims. - Quantify: state the requested resources and the relevant NodePool limit. - Separate observations from inferences.
Answer in this format: 1. Symptom 2. Evidence 3. Most likely cause 4. Options to resolve, with trade-offs (cost, blast radius) 5. How to verify tools: - type: McpServer mcpServer: name: kagent-tool-server kind: RemoteMCPServer apiGroup: kagent.dev toolNames: - k8s_get_resources - k8s_describe_resource - k8s_get_resource_yaml - k8s_get_eventsThis agent has to read Auto Mode resources:
NodePool , NodeClaim , and NodeClass . Check whether the tool server can:kubectl auth can-i list nodepools.karpenter.sh --as=system:serviceaccount:kagent:${KAGENT_TOOLS_SA}kubectl auth can-i list nodeclaims.karpenter.sh --as=system:serviceaccount:kagent:${KAGENT_TOOLS_SA}If either says
no , grant read-only access to just those resources:# Only needed if the kagent tool server cannot read Auto Mode capacity resources.# Check first:# kubectl auth can-i list nodepools.karpenter.sh \# --as=system:serviceaccount:kagent:<KAGENT_TOOLS_SA># Replace <KAGENT_TOOLS_SA> with the tool server's ServiceAccount name.apiVersion: rbac.authorization.k8s.io/v1kind: ClusterRolemetadata: name: agents-capacity-readrules: - apiGroups: ["karpenter.sh"] resources: ["nodepools", "nodeclaims"] verbs: ["get", "list", "watch"] - apiGroups: ["eks.amazonaws.com"] resources: ["nodeclasses"] verbs: ["get", "list", "watch"] - apiGroups: [""] resources: ["nodes"] verbs: ["get", "list", "watch"]---apiVersion: rbac.authorization.k8s.io/v1kind: ClusterRoleBindingmetadata: name: agents-capacity-readroleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: agents-capacity-readsubjects: - kind: ServiceAccount name: <KAGENT_TOOLS_SA> namespace: kagentsed "s/<KAGENT_TOOLS_SA>/${KAGENT_TOOLS_SA}/" 08-tools-karpenter-read-rbac.yaml | kubectl apply -f -kubectl apply -f 05-agent-capacity-analyst.yamlThe orchestrator
apiVersion: kagent.dev/v1alpha2kind: Agentmetadata: name: platform-orchestrator namespace: kagentspec: description: >- Entry point for cluster health questions. Delegates to specialist agents and combines their findings into one report. type: Declarative declarative: modelConfig: bedrock-nova systemMessage: | You coordinate specialist agents for a Kubernetes platform team. You have no direct cluster access; you only delegate.
Specialists: - k8s-troubleshooter: crashing, restarting, or unhealthy pods. - capacity-analyst: Pending pods, scheduling, NodePools, NodeClaims, capacity.
Process: 1. Split the user's request into independent questions. 2. Send each question to the right specialist with the namespace and workload names. Ask for evidence, not just conclusions. 3. If a specialist's answer is ambiguous, ask it one follow-up. 4. Combine the results into one report: - Summary table: workload | status | root cause | owner action - Per-workload details (keep the specialists' evidence) - Recommended order of fixes and why Never invent findings a specialist did not report. tools: - type: Agent agent: name: k8s-troubleshooter - type: Agent agent: name: capacity-analystThe orchestrator has no Kubernetes tools at all. Its only tools are the two other agents. Under the hood, kagent offers each specialist to the orchestrator as a callable tool, described by that agent's
description field, and makes the call over A2A. That is why the descriptions were written to say when to use each agent.kubectl apply -f 06-agent-platform-orchestrator.yamlkubectl get agents -n kagentAsk for a health report
In the UI, choose platform-orchestrator and ask:
Give me a health report for the demo namespace. Some workloads are not running. For each one, tell me why and what we should do.


For
report-worker , a good answer will:- state the numbers: 12 vCPU requested against an 8 vCPU NodePool limit;
- recognize that the nodeSelector and toleration deliberately target
spot-burst;
- give options with trade-offs, not just one fix:
- reduce the request, if the work can be split;
- raise the NodePool limit, which raises the cost ceiling for everything on that pool;
- move the workload to
general-purpose, which means on-demand pricing.
The decision is a cost and risk trade-off, and it belongs to a human. An agent that just says "increase the limit" is a warning sign.
Calling an agent over A2A directly
Every kagent agent is also an A2A endpoint. Other systems, such as a chat bot or an incident tool, can call it without going through the kagent UI. Each agent publishes an agent card describing itself:
kubectl port-forward -n kagent svc/kagent-controller 8083:8083 &sleep 3curl -s http://localhost:8083/api/a2a/kagent/platform-orchestrator/.well-known/agent-card.json | python3 -m json.toolkill %1Keep these endpoints inside the cluster. Anything that can call the orchestrator can, indirectly, use everything the tool server is allowed to do.
When multi-agent is not worth it
Splitting work across agents has real costs:
- Tokens multiply. Each delegation is a full model conversation of its own. One question to the orchestrator can mean three or more model calls, each carrying its own context.
- Latency adds up. Delegations that run one after another add their response times together.
- Errors spread. If a specialist states a guess as a finding, the orchestrator may combine it confidently with the rest. "Never invent findings a specialist did not report" reduces this risk but doesn't remove it.
If one agent with five tools answers your question reliably, keep one agent. Move to orchestration when the specialists need different tools, different permissions, or different instructions that would conflict if you put them in one system message.
Cost and operations notes
- Watch Bedrock usage. Bedrock publishes per-model invocation and token metrics in CloudWatch. Set a budget alarm before you let others use these agents.
- LiteLLM is your control point. Because every model call goes through it, LiteLLM is where per-key budgets, rate limits, and request logging can be added later, without changing any agent.
- Auto Mode handles the compute. kagent, LiteLLM, and the agent pods land on the
general-purposepool automatically. None of them tolerate thespot-bursttaint, so they never land on Spot by accident.
Troubleshooting
LiteLLM returns 401. The key kagent or your curl sent doesn't match
litellm-master-key . Check that the ModelConfig references the right Secret and key.LiteLLM returns a Bedrock
AccessDeniedException . Check, in order:- Is the Pod Identity association on the
litellmServiceAccount inkagent?
- Was the pod created after the association? (Run
kubectl rollout restartif not.)
- Does the IAM policy cover the inference profile and the underlying model ARNs?
- Is model access enabled for Amazon Nova in this account?
kubectl logs -n kagent deploy/litellm shows the exact Bedrock error.Bedrock
ValidationException about the model ID. The inference profile ID is wrong for this region, or you used a foundation-model ID where a profile is required. List valid profiles as in Part 1, Step 8.The Agent is never ready.
kubectl describe agent <name> -n kagent shows the controller's reason. A ModelConfig name that doesn't match, or a tool server name that doesn't match, are the most common causes.The agent says a tool doesn't exist. The names in
toolNames must match what the tool server actually exposes. Inspect the RemoteMCPServer status to see the tools it found:kubectl get remotemcpserver kagent-tool-server -n kagent \ -o jsonpath='{range .status.discoveredTools[*]}{.name}{"\n"}{end}'The capacity analyst says it can't see NodePools. This is RBAC on the tool server, not the agent. Apply
08-tools-karpenter-read-rbac.yaml .
Clean up
Delete in this order. The AWS resources created outside eksctl's cluster stacks won't be removed by
eksctl delete cluster .# Workloads and agentskubectl delete -f 06-agent-platform-orchestrator.yaml -f 05-agent-capacity-analyst.yaml \ -f 04-agent-k8s-troubleshooter.yaml --ignore-not-foundkubectl delete -f 07-demo-report-worker.yaml -f 03-demo-checkout-api.yaml --ignore-not-found
# kagent and LiteLLMhelm uninstall kagent -n kagenthelm uninstall kagent-crds -n kagentkubectl delete -f 01-litellm.yaml --ignore-not-found
# Pod Identity associations and IAMeksctl delete podidentityassociation --cluster "${CLUSTER_NAME}" --region "${AWS_REGION}" \ --namespace kagent --service-account-name litellmeksctl delete podidentityassociation --cluster "${CLUSTER_NAME}" --region "${AWS_REGION}" \ --namespace demo --service-account-name bedrock-readeraws iam delete-policy --policy-arn "arn:aws:iam::${AWS_ACCOUNT_ID}:policy/agents-demo-bedrock-invoke"
# The clustereksctl delete cluster --name "${CLUSTER_NAME}" --region "${AWS_REGION}"Then check the IAM console for leftover
agents-demo-* roles, and the EC2 console for leftover volumes or load balancers.
What we built, and what's still missing
Across both parts we went from an empty AWS account to:
- an EKS Auto Mode cluster that manages its own compute, storage, and load balancing;
- agents defined as Kubernetes resources, calling Amazon Nova Pro on Bedrock without a single static credential;
- a read-only troubleshooting agent that gathers evidence before it concludes;
- an orchestrator that combines specialists, one of which understands the platform's own capacity rules.
There is still work before I'd call this production-ready:
- a separate,
view-only tool server for diagnostic agents;
- NetworkPolicies so agents can reach only LiteLLM and the tool server;
- human approval for any agent that can change resources;
- tracing of every model and tool call;
- agent definitions delivered through GitOps.
Each of those is a good follow-up topic.
The idea I'd take from this series is that an agent's usefulness depends on the boundaries around it: the tools it may call, the permissions behind those tools, and the evidence it is required to show. Kubernetes already has good primitives for every one of those boundaries, which is a strong reason to run agents on it.
Series: Platform enigneering (2 articles)
- 2AI Agents on Kubernetes, Part 2: Deploying kagent and Putting Agents to Work This article
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article