
AI Agents on Kubernetes, Part 1: Building the Platform with EKS Auto Mode and eksctl
Part 1 of 2. This part builds the cluster. Part 2 deploys kagent on it and puts AI agents to work.
Series: Platform enigneering (2 articles)
AI Agents on Kubernetes, Part 1: Building the Platform with EKS Auto Mode and eksctl
Part 1 of 2. This part builds the cluster. Part 2 deploys kagent on it and puts AI agents to work.
AI agents behave differently from the workloads most platforms were designed for. Their resource usage comes in bursts that are hard to predict, they call cloud APIs that need credentials, and you usually want to add or remove them without filing a capacity ticket each time. That makes them a good fit for a cluster where you don't have to think about nodes much.
That is why this series starts with Amazon EKS Auto Mode. In Auto Mode, AWS takes over node provisioning, scaling, patching, and several core add-ons: networking, storage, load balancing, and Pod Identity. You still own the workloads, the policies, and the cost decisions. In this part we build an Auto Mode cluster with
eksctl and then test its features one at a time, so you can see what it really does for you and where its limits are.By the end of this article you will have:
- An EKS Auto Mode cluster called
agents-demo, created from one eksctl config file
- Seen nodes appear from nothing when a pod needs them, and disappear when it doesn't
- A custom Spot NodePool, behind a taint, for workloads that opt in
- Dynamic EBS storage and an AWS load balancer, without installing any controllers yourself
- Pod Identity working against Amazon Bedrock, which is the foundation for the agents in Part 2
What Auto Mode changes
On a standard EKS cluster, the control plane is managed but the data plane is your job. You pick and maintain node groups or run Karpenter yourself. You install and upgrade the VPC CNI, CoreDNS, kube-proxy, the EBS CSI driver, the AWS Load Balancer Controller, and the Pod Identity agent, and you rotate AMIs for security patches.
Auto Mode moves most of that to AWS:
Area | Standard EKS (managed node groups) | EKS Auto Mode |
|---|---|---|
Node provisioning | You size node groups or run Karpenter | Built-in, Karpenter-based provisioning, driven by pending pods |
Node OS and patching | You roll AMIs | AWS-managed, immutable Bottlerocket-based nodes, replaced automatically |
Node lifetime | Until you replace it | Capped (21 days maximum) so nodes are recycled regularly |
VPC CNI, kube-proxy, CoreDNS | Add-ons you upgrade | Built in and managed |
EBS CSI driver | Add-on you install | Built-in capability (you still define the StorageClass) |
Load balancing | You install AWS Load Balancer Controller | Built-in capability |
Pod Identity agent | Add-on you install | Built in |
Node access | SSH / SSM possible | No SSH or SSM; logs through the NodeDiagnostic API instead |
Pricing | EC2 + control plane | EC2 + control plane + a per-instance Auto Mode management fee |
The last two rows are the trade-off. You give up direct access to nodes and pay a premium on compute, and in return you have much less undifferentiated work to do. For a platform that will host fast-changing agent workloads, I think that trade is worth it. It isn't always: if you run large, steady fleets on deeply customized nodes, self-managed Karpenter on standard EKS may still be cheaper and more flexible.

Prerequisites
Tool | Version used | Notes |
|---|---|---|
AWS CLI | v2 | Configured with credentials for the target account |
eksctl | A release with autoModeConfig support | eksctl version |
kubectl | Within one minor version of the cluster | kubectl version --client |
IAM: the identity that runs
eksctl needs to create EKS clusters, IAM roles, a VPC (eksctl creates a new one by default), and CloudFormation stacks. In a personal sandbox account, an admin role is simplest. In a shared account, ask for a scoped role. Do not run this from long-lived root credentials.Cost: you pay while the cluster exists. The main items are the EKS control plane (USD 0.10 per hour on standard support), the EC2 instances Auto Mode launches, the Auto Mode management fee on those instances, a NAT gateway, and any load balancers or EBS volumes you create. Check the EKS pricing page for current rates. Delete the cluster when you are done, or at least between Part 1 and Part 2 if there will be a long gap.
Check who you are before creating anything:
aws sts get-caller-identityeksctl versionkubectl version --client
Step 1: Describe the cluster in one file
# eksctl ClusterConfig for EKS Auto Mode# Region and Kubernetes version are placeholders; adjust to your environment.apiVersion: eksctl.io/v1alpha5kind: ClusterConfig
metadata: name: agents-demo region: ap-southeast-1 version: "1.36" tags: project: ai-agents-on-k8s owner: platform-demo
autoModeConfig: enabled: true # Built-in NodePools managed by AWS. Omitting this field creates both by default. nodePools: ["general-purpose", "system"]Notice what is not in this file: no
managedNodeGroups , no addons list, no IAM OIDC setup. autoModeConfig.enabled: true is what replaces all of that. The two built-in NodePools:-
general-purposeruns your normal workloads on on-demand instances.
-
systemis tainted for critical add-ons that you choose to run there.
eksctl also creates the IAM role that Auto Mode nodes use.
Pin the Kubernetes version explicitly. Relying on the default makes your tutorial (and your platform) behave differently depending on which day someone runs it.
Step 2: Create the cluster
cd part-1eksctl create cluster -f cluster.yamlThis takes around 15 to 20 minutes. eksctl creates CloudFormation stacks for the VPC and the cluster, then writes a context for the new cluster into your kubeconfig.


Confirm Auto Mode is enabled from the API's point of view, not just from eksctl's:
aws eks describe-cluster \ --name agents-demo \ --region ap-southeast-1 \ --query '{compute: cluster.computeConfig, storage: cluster.storageConfig, lb: cluster.kubernetesNetworkConfig.elasticLoadBalancing}'You should see
computeConfig.enabled: true with both node pools listed and a nodeRoleArn , block storage enabled, and elastic load balancing enabled. Those three flags are the Auto Mode capabilities we test in the rest of this article.
Step 3: An empty data plane
This is the first thing that surprises people:
kubectl get nodesNo resources foundA brand-new Auto Mode cluster can have zero nodes. Core components that you would normally see as pods (CoreDNS, kube-proxy, the CNI) don't run as pods you manage in Auto Mode. Nodes appear only when there is a pod to place on them.
Look at what Auto Mode gives you to control provisioning:
kubectl get nodepoolskubectl get nodeclassesNAME NODECLASS NODES READY AGEgeneral-purpose default 0 True 14msystem default 1 True 14mNAME ROLE READY AGEdefault eksctl-agents-demo-cluster-AutoModeNodeRole-qUvmdaKsl7Ib True 14mNodePool is the Karpenter API ( karpenter.sh/v1 ). NodeClass is Auto Mode's own API ( eks.amazonaws.com/v1 ), which takes the place of Karpenter's EC2NodeClass and controls subnets, security groups, ephemeral storage, and network policy defaults. If you have used Karpenter, you already know most of this.

Step 4: Watch nodes appear on demand
Create the demo namespace and a Deployment that requests 3 vCPU in total and does nothing else:
# A workload that requests CPU but does nothing.# Used to watch Auto Mode provision (and later remove) nodes.apiVersion: apps/v1kind: Deploymentmetadata: name: inflate namespace: demospec: replicas: 3 selector: matchLabels: app: inflate template: metadata: labels: app: inflate spec: terminationGracePeriodSeconds: 0 containers: - name: pause image: public.ecr.aws/eks-distro/kubernetes/pause:3.7 resources: requests: cpu: "1" memory: 256Mi securityContext: allowPrivilegeEscalation: false runAsNonRoot: true runAsUser: 65532Open a second terminal and watch NodeClaims. A NodeClaim is Karpenter's record of "I am launching an instance for these pods":
kubectl get nodeclaims -wThen apply the workload in the first terminal:
kubectl apply -f 00-namespace.yamlkubectl apply -f 01-inflate.yamlkubectl get pods -n demo -wThe pods are
Pending at first. Within a few seconds a NodeClaim appears, an EC2 instance launches, the node joins, and the pods move to Running .Now look at what was chosen:
kubectl get nodes -L karpenter.sh/nodepool,node.kubernetes.io/instance-type,karpenter.sh/capacity-type,topology.kubernetes.io/zoneThe provisioner looks at the pending pods' requests and picks an instance type that fits them, instead of you guessing a node group size upfront.


Step 5: A custom Spot NodePool for workloads that opt in
The built-in
general-purpose pool uses on-demand capacity. For work that can survive interruptions, such as batch jobs, experiments, or (later) some agent tasks, Spot is much cheaper. We don't want everything to land on Spot by accident, though, so the pool gets a taint and a hard limit:# Custom NodePool for opt-in, interruption-tolerant workloads.# Spot capacity, capped at 8 vCPU / 32 GiB total, tainted so only# workloads that explicitly tolerate it land here.apiVersion: karpenter.sh/v1kind: NodePoolmetadata: name: spot-burstspec: template: metadata: labels: workload-type: burst spec: nodeClassRef: group: eks.amazonaws.com kind: NodeClass name: default taints: - key: workload-type value: burst effect: NoSchedule requirements: - key: karpenter.sh/capacity-type operator: In values: ["spot"] - key: eks.amazonaws.com/instance-category operator: In values: ["c", "m"] - key: eks.amazonaws.com/instance-generation operator: Gt values: ["5"] - key: kubernetes.io/arch operator: In values: ["amd64", "arm64"] expireAfter: 168h limits: cpu: "8" memory: 32Gi disruption: consolidationPolicy: WhenEmptyOrUnderutilized consolidateAfter: 1mThree design choices are worth explaining:
- The taint. Workloads must explicitly tolerate
workload-type=burst:NoSchedule. Without a taint, the scheduler might put ordinary pods on Spot nodes.
-
limits. The pool can never add up to more than 8 vCPU and 32 GiB, no matter what is requested. This caps your spending, and it will come back in Part 2 as a failure scenario for the agents to diagnose.
- Broad instance requirements. Several instance families and both architectures make it more likely Spot capacity is available. If you narrow it to one instance type, Spot will fail far more often.
If this account has never used Spot, create the EC2 Spot service-linked role once. If it already exists, the command returns an error, which you can ignore:
aws iam create-service-linked-role --aws-service-name spot.amazonaws.comApply the pool and a workload that opts in:
# Opts in to the spot-burst NodePool via nodeSelector + toleration.apiVersion: apps/v1kind: Deploymentmetadata: name: burst-demo namespace: demospec: replicas: 2 selector: matchLabels: app: burst-demo template: metadata: labels: app: burst-demo spec: terminationGracePeriodSeconds: 0 nodeSelector: karpenter.sh/nodepool: spot-burst tolerations: - key: workload-type operator: Equal value: burst effect: NoSchedule containers: - name: pause image: public.ecr.aws/eks-distro/kubernetes/pause:3.7 resources: requests: cpu: "1" memory: 512Mi securityContext: allowPrivilegeEscalation: false runAsNonRoot: true runAsUser: 65532kubectl apply -f 02-nodepool-spot-burst.yamlkubectl apply -f 03-burst-demo.yamlkubectl get nodeclaims -L karpenter.sh/nodepool,karpenter.sh/capacity-typeYou should see a new NodeClaim from
spot-burst with capacity type spot . The inflate pods stay on the on-demand node, because they don't tolerate the taint.
Step 6: Persistent storage without installing a driver
Auto Mode includes the EBS CSI capability, but it does not decide your storage defaults for you. You create the StorageClass:
# Auto Mode ships the EBS CSI capability, but you still define the StorageClass.# Note the provisioner name differs from the self-managed EBS CSI driver.apiVersion: storage.k8s.io/v1kind: StorageClassmetadata: name: auto-ebs-gp3 annotations: storageclass.kubernetes.io/is-default-class: "true"provisioner: ebs.csi.eks.amazonaws.comvolumeBindingMode: WaitForFirstConsumerreclaimPolicy: DeleteallowVolumeExpansion: trueparameters: type: gp3 encrypted: "true"The provisioner is
ebs.csi.eks.amazonaws.com , which is not the self-managed driver's ebs.csi.aws.com . Copying an old StorageClass from a non-Auto Mode cluster is a common reason PVCs stay Pending forever.Test it with a PVC and a pod that writes to it:
apiVersion: v1kind: PersistentVolumeClaimmetadata: name: scratch-data namespace: demospec: accessModes: ["ReadWriteOnce"] storageClassName: auto-ebs-gp3 resources: requests: storage: 4Gi---apiVersion: v1kind: Podmetadata: name: pvc-writer namespace: demospec: containers: - name: writer image: public.ecr.aws/docker/library/busybox:1.36 command: ["sh", "-c", "date >> /data/proof.txt && cat /data/proof.txt && sleep 3600"] volumeMounts: - name: data mountPath: /data resources: requests: cpu: 50m memory: 32Mi volumes: - name: data persistentVolumeClaim: claimName: scratch-datakubectl apply -f 04-storageclass.yamlkubectl apply -f 05-pvc-test.yamlkubectl get pvc -n demo -wBecause of
WaitForFirstConsumer , the PVC stays Pending until the pod is scheduled. Then the volume is created in the same Availability Zone as the node and the claim becomes Bound . That ordering matters: if you bind immediately, you can end up with a volume in one AZ and a node in another.kubectl logs -n demo pvc-writerThe log shows the timestamp written to the volume.

Step 7: A load balancer without installing a controller
Auto Mode includes load balancing. A
Service of type LoadBalancer with loadBalancerClass: eks.amazonaws.com/nlb gets you a Network Load Balancer. An Ingress with an Auto Mode IngressClass gets you an ALB.Warning: this creates a public, internet-facing NLB. It is only for testing; delete it as soon as you have checked it works. Nothing in Part 2 exposes the agents publicly.
# WARNING: creates an internet-facing NLB. Delete it after testing.apiVersion: apps/v1kind: Deploymentmetadata: name: hello-web namespace: demospec: replicas: 2 selector: matchLabels: app: hello-web template: metadata: labels: app: hello-web spec: containers: - name: nginx image: public.ecr.aws/nginx/nginx:1.27 ports: - containerPort: 80 resources: requests: cpu: 100m memory: 64Mi---apiVersion: v1kind: Servicemetadata: name: hello-web namespace: demo annotations: service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facingspec: type: LoadBalancer loadBalancerClass: eks.amazonaws.com/nlb selector: app: hello-web ports: - port: 80 targetPort: 80kubectl apply -f 06-nlb-service.yamlkubectl get svc -n demo hello-web -wWhen
EXTERNAL-IP shows a DNS name, wait a minute or two for the targets to pass health checks, then:LB=$(kubectl get svc -n demo hello-web -o jsonpath='{.status.loadBalancer.ingress[0].hostname}')curl -sI "http://$LB" | head -n 1You should get
HTTP/1.1 200 OK . Now delete it:kubectl delete -f 06-nlb-service.yamlDelete load balancers before you delete the cluster. A leftover NLB keeps network interfaces and security groups in the VPC, and that can make
eksctl delete cluster fail.
Step 8: Pod Identity, ready for Part 2
In Part 2, the AI agents will call Amazon Bedrock. Static AWS access keys in a Kubernetes Secret are exactly what we want to avoid, and that is what EKS Pod Identity is for. On a standard cluster you would install the Pod Identity agent add-on first. In Auto Mode it is already there.
Pod Identity maps a Kubernetes ServiceAccount to an IAM role. Pods using that ServiceAccount get temporary credentials automatically. Create the association first, with read-only Bedrock permissions:
eksctl create podidentityassociation \ --cluster agents-demo \ --region ap-southeast-1 \ --namespace demo \ --service-account-name bedrock-reader \ --role-name agents-demo-bedrock-reader \ --permission-policy-arns arn:aws:iam::aws:policy/AmazonBedrockReadOnlyThen create the ServiceAccount and a pod with the AWS CLI:
# ServiceAccount + pod used to prove Pod Identity works.# The IAM role is associated with this ServiceAccount via eksctl (see article).apiVersion: v1kind: ServiceAccountmetadata: name: bedrock-reader namespace: demo---apiVersion: v1kind: Podmetadata: name: aws-cli namespace: demospec: serviceAccountName: bedrock-reader containers: - name: aws-cli image: public.ecr.aws/aws-cli/aws-cli:latest # VERIFY: pin a version for reproducibility command: ["sleep", "3600"] resources: requests: cpu: 50m memory: 128Mikubectl apply -f 07-pod-identity-test.yamlkubectl wait -n demo --for=condition=Ready pod/aws-cli --timeout=120sThe order matters. Credentials are injected when a pod is created. If the pod existed before the association, delete it and create it again.
Check three things: that the credential environment variables were injected, which role the pod is using, and that Bedrock answers:
kubectl exec -n demo aws-cli -- sh -c 'env | grep AWS_CONTAINER'
kubectl exec -n demo aws-cli -- aws sts get-caller-identity
kubectl exec -n demo aws-cli -- aws bedrock list-inference-profiles \ --region ap-southeast-1 \ --query "inferenceProfileSummaries[?contains(inferenceProfileId, 'claude')].inferenceProfileId" \ --output tableget-caller-identity should show an assumed-role ARN for agents-demo-bedrock-reader , not your own user. The last command lists the Claude inference profiles available in your region. Check that global.anthropic.claude-opus-5-5 is in the list, because it is the model ID we use in Part 2.If the list is empty, you may need to enable access to Anthropic models in the Bedrock console for this account and region first.
Step 9: Watch nodes go away
Scaling up only gets you half the savings. Remove the test workloads:
kubectl delete -f 01-inflate.yaml -f 03-burst-demo.yamlkubectl get nodeclaims -wWithin a minute or two, the empty nodes are drained and terminated. The
spot-burst pool uses consolidateAfter: 1m . The built-in pools have their own consolidation settings, which you can read with kubectl get nodepool general-purpose -o yaml .Consolidation also moves pods off under-used nodes. That is good for cost, but it means pods get evicted. For anything stateful or latency-sensitive, set
PodDisruptionBudget s. Agent workloads with long-running sessions will need this in a real deployment.

Limitations to know before you commit
Auto Mode isn't a free win. These are the points I'd check before using it for a real platform:
- No SSH or SSM to nodes. For node-level debugging you use the
NodeDiagnosticresource to collect logs, orkubectl debug. If your incident runbooks depend on getting a shell on a node, they need rewriting.
- Node customization is limited. No custom AMIs or arbitrary user data. Anything you'd normally do with a bootstrap script needs another approach, such as a DaemonSet or a NodeClass setting.
- Nodes are recycled on a schedule. The maximum lifetime keeps nodes patched, but every workload must tolerate being rescheduled. That is good discipline, but you have to plan for it.
- There is a compute premium. The management fee is charged per instance, so it grows with your fleet. Compare it with the engineering time you would otherwise spend on node and add-on upkeep, not with zero.
- You still own a lot. Observability, policy engines, GitOps controllers, ingress design, secrets management, and of course the applications are all still your job. Auto Mode makes the data plane smaller, not your whole platform.
Troubleshooting
Pods stay Pending and no NodeClaim appears. Run
kubectl describe pod and read the events. The usual causes are a nodeSelector or toleration that no NodePool matches, or requests larger than any allowed instance type. kubectl get events -A --sort-by=.lastTimestamp | tail -30 usually shows the reason.A NodeClaim is created but never becomes Ready.
kubectl describe nodeclaim <name> shows launch errors. For Spot, the most common cause is no capacity for the allowed instance types: widen the requirements. Also check the Spot service-linked role from Step 5.PVC stuck Pending after the pod is scheduled. Check that the StorageClass provisioner is
ebs.csi.eks.amazonaws.com and that kubectl get storageclass shows the one you expect as the default.The pod has no AWS credentials. The pod was probably created before the association. Delete and recreate it. Also check
eksctl get podidentityassociation --cluster agents-demo --region ap-southeast-1 .eksctl delete cluster hangs or fails on VPC deletion. Look for leftover load balancers or ENIs from Services you didn't delete, and remove them.
Clean up or keep going
If you are continuing to Part 2 soon, keep the cluster, the
spot-burst NodePool, the StorageClass, and the demo namespace. Remove only the test resources:kubectl delete -f 05-pvc-test.yaml -f 07-pod-identity-test.yaml --ignore-not-foundkubectl delete -f 06-nlb-service.yaml --ignore-not-foundIf you are stopping here, delete everything:
kubectl delete -f 06-nlb-service.yaml --ignore-not-foundeksctl delete podidentityassociation --cluster agents-demo --region ap-southeast-1 \ --namespace demo --service-account-name bedrock-readereksctl delete cluster -f cluster.yamlAfterwards, check that the
agents-demo-bedrock-reader IAM role is gone, and check the EC2 console for any leftover volumes or load balancers.
What we have and what comes next
We now have a cluster where:
- Compute appears when pods need it and goes away when they don't.
- Spot capacity is available only to workloads that opt in, and only up to a limit.
- Storage and load balancing work without installing any controllers.
- Pods can get AWS credentials for Bedrock without a single static key.
What we haven't done is run anything intelligent on it. In Part 2 we deploy kagent, a CNCF project that defines AI agents as Kubernetes resources. We connect it to Amazon Bedrock through Pod Identity, build a read-only troubleshooting agent, and then set up a multi-agent orchestrator. It hands work to specialist agents, one of which reads the NodePools and NodeClaims we created today to explain why a workload can't be scheduled.
Series: Platform enigneering (2 articles)
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article