AWS Builder Center
AI Agents on Kubernetes, Part 1: Building the Platform with EKS Auto Mode and eksctl

AI Agents on Kubernetes, Part 1: Building the Platform with EKS Auto Mode and eksctl

Part 1 of 2. This part builds the cluster. Part 2 deploys kagent on it and puts AI agents to work.

DevOps Engineer | AWS Community Builder

AI Agents on Kubernetes, Part 1: Building the Platform with EKS Auto Mode and eksctl

Part 1 of 2. This part builds the cluster. Part 2 deploys kagent on it and puts AI agents to work.
AI agents behave differently from the workloads most platforms were designed for. Their resource usage comes in bursts that are hard to predict, they call cloud APIs that need credentials, and you usually want to add or remove them without filing a capacity ticket each time. That makes them a good fit for a cluster where you don't have to think about nodes much.
That is why this series starts with Amazon EKS Auto Mode. In Auto Mode, AWS takes over node provisioning, scaling, patching, and several core add-ons: networking, storage, load balancing, and Pod Identity. You still own the workloads, the policies, and the cost decisions. In this part we build an Auto Mode cluster with  eksctl  and then test its features one at a time, so you can see what it really does for you and where its limits are.
By the end of this article you will have:
  • An EKS Auto Mode cluster called  agents-demo , created from one eksctl config file
  • Seen nodes appear from nothing when a pod needs them, and disappear when it doesn't
  • A custom Spot NodePool, behind a taint, for workloads that opt in
  • Dynamic EBS storage and an AWS load balancer, without installing any controllers yourself
  • Pod Identity working against Amazon Bedrock, which is the foundation for the agents in Part 2
All the files used here are in the  companion repo  under  part-1/ .



What Auto Mode changes

On a standard EKS cluster, the control plane is managed but the data plane is your job. You pick and maintain node groups or run Karpenter yourself. You install and upgrade the VPC CNI, CoreDNS, kube-proxy, the EBS CSI driver, the AWS Load Balancer Controller, and the Pod Identity agent, and you rotate AMIs for security patches.
Auto Mode moves most of that to AWS:
Area
Standard EKS (managed node groups)
EKS Auto Mode
Node provisioning
You size node groups or run Karpenter
Built-in, Karpenter-based provisioning, driven by pending pods
Node OS and patching
You roll AMIs
AWS-managed, immutable Bottlerocket-based nodes, replaced automatically
Node lifetime
Until you replace it
Capped (21 days maximum) so nodes are recycled regularly
VPC CNI, kube-proxy, CoreDNS
Add-ons you upgrade
Built in and managed
EBS CSI driver
Add-on you install
Built-in capability (you still define the StorageClass)
Load balancing
You install AWS Load Balancer Controller
Built-in capability
Pod Identity agent
Add-on you install
Built in
Node access
SSH / SSM possible
No SSH or SSM; logs through the  NodeDiagnostic  API instead
Pricing
EC2 + control plane
EC2 + control plane + a per-instance Auto Mode management fee
The last two rows are the trade-off. You give up direct access to nodes and pay a premium on compute, and in return you have much less undifferentiated work to do. For a platform that will host fast-changing agent workloads, I think that trade is worth it. It isn't always: if you run large, steady fleets on deeply customized nodes, self-managed Karpenter on standard EKS may still be cheaper and more flexible.

EKS Auto Mode capabilities


Prerequisites

Tool
Version used
Notes
AWS CLI
v2
Configured with credentials for the target account
eksctl
A release with  autoModeConfig  support
 eksctl version 
kubectl
Within one minor version of the cluster
 kubectl version --client 
IAM: the identity that runs  eksctl  needs to create EKS clusters, IAM roles, a VPC (eksctl creates a new one by default), and CloudFormation stacks. In a personal sandbox account, an admin role is simplest. In a shared account, ask for a scoped role. Do not run this from long-lived root credentials.
Cost: you pay while the cluster exists. The main items are the EKS control plane (USD 0.10 per hour on standard support), the EC2 instances Auto Mode launches, the Auto Mode management fee on those instances, a NAT gateway, and any load balancers or EBS volumes you create. Check the  EKS pricing page  for current rates. Delete the cluster when you are done, or at least between Part 1 and Part 2 if there will be a long gap.
Check who you are before creating anything:
aws sts get-caller-identity
eksctl version
kubectl version --client



Step 1: Describe the cluster in one file

# eksctl ClusterConfig for EKS Auto Mode
# Region and Kubernetes version are placeholders; adjust to your environment.
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
name: agents-demo
region: ap-southeast-1
version: "1.36"
tags:
project: ai-agents-on-k8s
owner: platform-demo

autoModeConfig:
enabled: true
# Built-in NodePools managed by AWS. Omitting this field creates both by default.
nodePools: ["general-purpose", "system"]
Notice what is not in this file: no  managedNodeGroups , no  addons  list, no IAM OIDC setup.  autoModeConfig.enabled: true  is what replaces all of that. The two built-in NodePools:
  •  general-purpose  runs your normal workloads on on-demand instances.
  •  system  is tainted for critical add-ons that you choose to run there.
eksctl also creates the IAM role that Auto Mode nodes use.
Pin the Kubernetes version explicitly. Relying on the default makes your tutorial (and your platform) behave differently depending on which day someone runs it.



Step 2: Create the cluster

cd part-1
eksctl create cluster -f cluster.yaml
This takes around 15 to 20 minutes. eksctl creates CloudFormation stacks for the VPC and the cluster, then writes a context for the new cluster into your kubeconfig.
CloudFormation stacks created by eksctl

Terminal showing eksctl create cluster finishing successfully

Confirm Auto Mode is enabled from the API's point of view, not just from eksctl's:
aws eks describe-cluster \
--name agents-demo \
--region ap-southeast-1 \
--query '{compute: cluster.computeConfig, storage: cluster.storageConfig, lb: cluster.kubernetesNetworkConfig.elasticLoadBalancing}'
You should see  computeConfig.enabled: true  with both node pools listed and a  nodeRoleArn , block storage enabled, and elastic load balancing enabled. Those three flags are the Auto Mode capabilities we test in the rest of this article.



Step 3: An empty data plane

This is the first thing that surprises people:
kubectl get nodes
No resources found
A brand-new Auto Mode cluster can have zero nodes. Core components that you would normally see as pods (CoreDNS, kube-proxy, the CNI) don't run as pods you manage in Auto Mode. Nodes appear only when there is a pod to place on them.
Look at what Auto Mode gives you to control provisioning:
kubectl get nodepools
kubectl get nodeclasses
NAME NODECLASS NODES READY AGE
general-purpose default 0 True 14m
system default 1 True 14m
NAME ROLE READY AGE
default eksctl-agents-demo-cluster-AutoModeNodeRole-qUvmdaKsl7Ib True 14m
 NodePool  is the Karpenter API ( karpenter.sh/v1 ).  NodeClass  is Auto Mode's own API ( eks.amazonaws.com/v1 ), which takes the place of Karpenter's  EC2NodeClass  and controls subnets, security groups, ephemeral storage, and network policy defaults. If you have used Karpenter, you already know most of this.

Kubectl get nodes (empty) and kubectl get nodepools


Step 4: Watch nodes appear on demand

Create the demo namespace and a Deployment that requests 3 vCPU in total and does nothing else:
# A workload that requests CPU but does nothing.
# Used to watch Auto Mode provision (and later remove) nodes.
apiVersion: apps/v1
kind: Deployment
metadata:
name: inflate
namespace: demo
spec:
replicas: 3
selector:
matchLabels:
app: inflate
template:
metadata:
labels:
app: inflate
spec:
terminationGracePeriodSeconds: 0
containers:
- name: pause
image: public.ecr.aws/eks-distro/kubernetes/pause:3.7
resources:
requests:
cpu: "1"
memory: 256Mi
securityContext:
allowPrivilegeEscalation: false
runAsNonRoot: true
runAsUser: 65532
Open a second terminal and watch NodeClaims. A NodeClaim is Karpenter's record of "I am launching an instance for these pods":
kubectl get nodeclaims -w
Then apply the workload in the first terminal:
kubectl apply -f 00-namespace.yaml
kubectl apply -f 01-inflate.yaml
kubectl get pods -n demo -w
The pods are  Pending  at first. Within a few seconds a NodeClaim appears, an EC2 instance launches, the node joins, and the pods move to  Running .
Now look at what was chosen:
kubectl get nodes -L karpenter.sh/nodepool,node.kubernetes.io/instance-type,karpenter.sh/capacity-type,topology.kubernetes.io/zone
The provisioner looks at the pending pods' requests and picks an instance type that fits them, instead of you guessing a node group size upfront.
NodeClaim being created and pods transitioning to Running


Kubectl get nodes with nodepool/instance-type/capacity-type labels


Step 5: A custom Spot NodePool for workloads that opt in

The built-in  general-purpose  pool uses on-demand capacity. For work that can survive interruptions, such as batch jobs, experiments, or (later) some agent tasks, Spot is much cheaper. We don't want everything to land on Spot by accident, though, so the pool gets a taint and a hard limit:
# Custom NodePool for opt-in, interruption-tolerant workloads.
# Spot capacity, capped at 8 vCPU / 32 GiB total, tainted so only
# workloads that explicitly tolerate it land here.
apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
name: spot-burst
spec:
template:
metadata:
labels:
workload-type: burst
spec:
nodeClassRef:
group: eks.amazonaws.com
kind: NodeClass
name: default
taints:
- key: workload-type
value: burst
effect: NoSchedule
requirements:
- key: karpenter.sh/capacity-type
operator: In
values: ["spot"]
- key: eks.amazonaws.com/instance-category
operator: In
values: ["c", "m"]
- key: eks.amazonaws.com/instance-generation
operator: Gt
values: ["5"]
- key: kubernetes.io/arch
operator: In
values: ["amd64", "arm64"]
expireAfter: 168h
limits:
cpu: "8"
memory: 32Gi
disruption:
consolidationPolicy: WhenEmptyOrUnderutilized
consolidateAfter: 1m
Three design choices are worth explaining:
  • The taint. Workloads must explicitly tolerate  workload-type=burst:NoSchedule . Without a taint, the scheduler might put ordinary pods on Spot nodes.
  •  limits . The pool can never add up to more than 8 vCPU and 32 GiB, no matter what is requested. This caps your spending, and it will come back in Part 2 as a failure scenario for the agents to diagnose.
  • Broad instance requirements. Several instance families and both architectures make it more likely Spot capacity is available. If you narrow it to one instance type, Spot will fail far more often.
If this account has never used Spot, create the EC2 Spot service-linked role once. If it already exists, the command returns an error, which you can ignore:
aws iam create-service-linked-role --aws-service-name spot.amazonaws.com
Apply the pool and a workload that opts in:
# Opts in to the spot-burst NodePool via nodeSelector + toleration.
apiVersion: apps/v1
kind: Deployment
metadata:
name: burst-demo
namespace: demo
spec:
replicas: 2
selector:
matchLabels:
app: burst-demo
template:
metadata:
labels:
app: burst-demo
spec:
terminationGracePeriodSeconds: 0
nodeSelector:
karpenter.sh/nodepool: spot-burst
tolerations:
- key: workload-type
operator: Equal
value: burst
effect: NoSchedule
containers:
- name: pause
image: public.ecr.aws/eks-distro/kubernetes/pause:3.7
resources:
requests:
cpu: "1"
memory: 512Mi
securityContext:
allowPrivilegeEscalation: false
runAsNonRoot: true
runAsUser: 65532
kubectl apply -f 02-nodepool-spot-burst.yaml
kubectl apply -f 03-burst-demo.yaml
kubectl get nodeclaims -L karpenter.sh/nodepool,karpenter.sh/capacity-type
You should see a new NodeClaim from  spot-burst  with capacity type  spot . The  inflate  pods stay on the on-demand node, because they don't tolerate the taint.



Step 6: Persistent storage without installing a driver

Auto Mode includes the EBS CSI capability, but it does not decide your storage defaults for you. You create the StorageClass:
# Auto Mode ships the EBS CSI capability, but you still define the StorageClass.
# Note the provisioner name differs from the self-managed EBS CSI driver.
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: auto-ebs-gp3
annotations:
storageclass.kubernetes.io/is-default-class: "true"
provisioner: ebs.csi.eks.amazonaws.com
volumeBindingMode: WaitForFirstConsumer
reclaimPolicy: Delete
allowVolumeExpansion: true
parameters:
type: gp3
encrypted: "true"
The provisioner is  ebs.csi.eks.amazonaws.com , which is not the self-managed driver's  ebs.csi.aws.com . Copying an old StorageClass from a non-Auto Mode cluster is a common reason PVCs stay  Pending  forever.
Test it with a PVC and a pod that writes to it:
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: scratch-data
namespace: demo
spec:
accessModes: ["ReadWriteOnce"]
storageClassName: auto-ebs-gp3
resources:
requests:
storage: 4Gi
---
apiVersion: v1
kind: Pod
metadata:
name: pvc-writer
namespace: demo
spec:
containers:
- name: writer
image: public.ecr.aws/docker/library/busybox:1.36
command: ["sh", "-c", "date >> /data/proof.txt && cat /data/proof.txt && sleep 3600"]
volumeMounts:
- name: data
mountPath: /data
resources:
requests:
cpu: 50m
memory: 32Mi
volumes:
- name: data
persistentVolumeClaim:
claimName: scratch-data
kubectl apply -f 04-storageclass.yaml
kubectl apply -f 05-pvc-test.yaml
kubectl get pvc -n demo -w
Because of  WaitForFirstConsumer , the PVC stays  Pending  until the pod is scheduled. Then the volume is created in the same Availability Zone as the node and the claim becomes  Bound . That ordering matters: if you bind immediately, you can end up with a volume in one AZ and a node in another.
kubectl logs -n demo pvc-writer
The log shows the timestamp written to the volume.
PVC moving from Pending to Bound, and pod log output




Step 7: A load balancer without installing a controller

Auto Mode includes load balancing. A  Service  of type  LoadBalancer  with  loadBalancerClass: eks.amazonaws.com/nlb  gets you a Network Load Balancer. An  Ingress  with an Auto Mode  IngressClass  gets you an ALB.
Warning: this creates a public, internet-facing NLB. It is only for testing; delete it as soon as you have checked it works. Nothing in Part 2 exposes the agents publicly.
# WARNING: creates an internet-facing NLB. Delete it after testing.
apiVersion: apps/v1
kind: Deployment
metadata:
name: hello-web
namespace: demo
spec:
replicas: 2
selector:
matchLabels:
app: hello-web
template:
metadata:
labels:
app: hello-web
spec:
containers:
- name: nginx
image: public.ecr.aws/nginx/nginx:1.27
ports:
- containerPort: 80
resources:
requests:
cpu: 100m
memory: 64Mi
---
apiVersion: v1
kind: Service
metadata:
name: hello-web
namespace: demo
annotations:
service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing
spec:
type: LoadBalancer
loadBalancerClass: eks.amazonaws.com/nlb
selector:
app: hello-web
ports:
- port: 80
targetPort: 80
kubectl apply -f 06-nlb-service.yaml
kubectl get svc -n demo hello-web -w
When  EXTERNAL-IP  shows a DNS name, wait a minute or two for the targets to pass health checks, then:
LB=$(kubectl get svc -n demo hello-web -o jsonpath='{.status.loadBalancer.ingress[0].hostname}')
curl -sI "http://$LB" | head -n 1
You should get  HTTP/1.1 200 OK . Now delete it:
kubectl delete -f 06-nlb-service.yaml
Delete load balancers before you delete the cluster. A leftover NLB keeps network interfaces and security groups in the VPC, and that can make  eksctl delete cluster  fail.
HTTP call to the NLB




Step 8: Pod Identity, ready for Part 2

In Part 2, the AI agents will call Amazon Bedrock. Static AWS access keys in a Kubernetes Secret are exactly what we want to avoid, and that is what EKS Pod Identity is for. On a standard cluster you would install the Pod Identity agent add-on first. In Auto Mode it is already there.
Pod Identity maps a Kubernetes ServiceAccount to an IAM role. Pods using that ServiceAccount get temporary credentials automatically. Create the association first, with read-only Bedrock permissions:
eksctl create podidentityassociation \
--cluster agents-demo \
--region ap-southeast-1 \
--namespace demo \
--service-account-name bedrock-reader \
--role-name agents-demo-bedrock-reader \
--permission-policy-arns arn:aws:iam::aws:policy/AmazonBedrockReadOnly
Then create the ServiceAccount and a pod with the AWS CLI:
# ServiceAccount + pod used to prove Pod Identity works.
# The IAM role is associated with this ServiceAccount via eksctl (see article).
apiVersion: v1
kind: ServiceAccount
metadata:
name: bedrock-reader
namespace: demo
---
apiVersion: v1
kind: Pod
metadata:
name: aws-cli
namespace: demo
spec:
serviceAccountName: bedrock-reader
containers:
- name: aws-cli
image: public.ecr.aws/aws-cli/aws-cli:latest # VERIFY: pin a version for reproducibility
command: ["sleep", "3600"]
resources:
requests:
cpu: 50m
memory: 128Mi
kubectl apply -f 07-pod-identity-test.yaml
kubectl wait -n demo --for=condition=Ready pod/aws-cli --timeout=120s
The order matters. Credentials are injected when a pod is created. If the pod existed before the association, delete it and create it again.
Check three things: that the credential environment variables were injected, which role the pod is using, and that Bedrock answers:
kubectl exec -n demo aws-cli -- sh -c 'env | grep AWS_CONTAINER'

kubectl exec -n demo aws-cli -- aws sts get-caller-identity

kubectl exec -n demo aws-cli -- aws bedrock list-inference-profiles \
--region ap-southeast-1 \
--query "inferenceProfileSummaries[?contains(inferenceProfileId, 'claude')].inferenceProfileId" \
--output table
 get-caller-identity  should show an assumed-role ARN for  agents-demo-bedrock-reader , not your own user. The last command lists the Claude inference profiles available in your region. Check that  global.anthropic.claude-opus-5-5  is in the list, because it is the model ID we use in Part 2.
If the list is empty, you may need to enable access to Anthropic models in the Bedrock console for this account and region first.



Step 9: Watch nodes go away

Scaling up only gets you half the savings. Remove the test workloads:
kubectl delete -f 01-inflate.yaml -f 03-burst-demo.yaml
kubectl get nodeclaims -w
Within a minute or two, the empty nodes are drained and terminated. The  spot-burst  pool uses  consolidateAfter: 1m . The built-in pools have their own consolidation settings, which you can read with  kubectl get nodepool general-purpose -o yaml .
Consolidation also moves pods off under-used nodes. That is good for cost, but it means pods get evicted. For anything stateful or latency-sensitive, set  PodDisruptionBudget s. Agent workloads with long-running sessions will need this in a real deployment.

NodeClaims being deleted after workload removal


Limitations to know before you commit

Auto Mode isn't a free win. These are the points I'd check before using it for a real platform:
  • No SSH or SSM to nodes. For node-level debugging you use the  NodeDiagnostic  resource to collect logs, or  kubectl debug . If your incident runbooks depend on getting a shell on a node, they need rewriting.
  • Node customization is limited. No custom AMIs or arbitrary user data. Anything you'd normally do with a bootstrap script needs another approach, such as a DaemonSet or a NodeClass setting.
  • Nodes are recycled on a schedule. The maximum lifetime keeps nodes patched, but every workload must tolerate being rescheduled. That is good discipline, but you have to plan for it.
  • There is a compute premium. The management fee is charged per instance, so it grows with your fleet. Compare it with the engineering time you would otherwise spend on node and add-on upkeep, not with zero.
  • You still own a lot. Observability, policy engines, GitOps controllers, ingress design, secrets management, and of course the applications are all still your job. Auto Mode makes the data plane smaller, not your whole platform.



Troubleshooting

Pods stay Pending and no NodeClaim appears. Run  kubectl describe pod  and read the events. The usual causes are a nodeSelector or toleration that no NodePool matches, or requests larger than any allowed instance type.  kubectl get events -A --sort-by=.lastTimestamp | tail -30  usually shows the reason.
A NodeClaim is created but never becomes Ready.  kubectl describe nodeclaim <name>  shows launch errors. For Spot, the most common cause is no capacity for the allowed instance types: widen the requirements. Also check the Spot service-linked role from Step 5.
PVC stuck Pending after the pod is scheduled. Check that the StorageClass provisioner is  ebs.csi.eks.amazonaws.com  and that  kubectl get storageclass  shows the one you expect as the default.
The pod has no AWS credentials. The pod was probably created before the association. Delete and recreate it. Also check  eksctl get podidentityassociation --cluster agents-demo --region ap-southeast-1 .
 eksctl delete cluster  hangs or fails on VPC deletion. Look for leftover load balancers or ENIs from Services you didn't delete, and remove them.



Clean up or keep going

If you are continuing to Part 2 soon, keep the cluster, the  spot-burst  NodePool, the StorageClass, and the  demo  namespace. Remove only the test resources:
kubectl delete -f 05-pvc-test.yaml -f 07-pod-identity-test.yaml --ignore-not-found
kubectl delete -f 06-nlb-service.yaml --ignore-not-found
If you are stopping here, delete everything:
kubectl delete -f 06-nlb-service.yaml --ignore-not-found
eksctl delete podidentityassociation --cluster agents-demo --region ap-southeast-1 \
--namespace demo --service-account-name bedrock-reader
eksctl delete cluster -f cluster.yaml
Afterwards, check that the  agents-demo-bedrock-reader  IAM role is gone, and check the EC2 console for any leftover volumes or load balancers.



What we have and what comes next

We now have a cluster where:
  • Compute appears when pods need it and goes away when they don't.
  • Spot capacity is available only to workloads that opt in, and only up to a limit.
  • Storage and load balancing work without installing any controllers.
  • Pods can get AWS credentials for Bedrock without a single static key.
What we haven't done is run anything intelligent on it. In Part 2 we deploy kagent, a CNCF project that defines AI agents as Kubernetes resources. We connect it to Amazon Bedrock through Pod Identity, build a read-only troubleshooting agent, and then set up a multi-agent orchestrator. It hands work to specialist agents, one of which reads the NodePools and NodeClaims we created today to explain why a workload can't be scheduled.
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article