My experience with Amazon EKS has focused on operating cloud-native platforms: provisioning clusters, scheduling workloads, managing access, and building the networking and observability around them. I brought that perspective to the Generative AI on EKS using NVIDIA GPU workshop to see what changes when the application is a language model that needs accelerator capacity.

The short answer is that the Kubernetes foundation is recognizable, but the workload makes some operational details much more visible. GPU capacity must be available in the right place. Model files need to reach the inference server. The serving engine must expose an endpoint that applications can use. And cluster health alone is not enough to explain whether inference is working well.

The Terraform setup builds the foundation: a VPC with public and private subnets, an Amazon EKS Auto Mode cluster, system and general-purpose node pools, and a dedicated GPU NodePool and NodeClass. It also prepares the integrations used by the workshop, including EKS Pod Identity, model storage in S3, an S3 CSI driver, and the Prometheus and Grafana monitoring stack. Supporting resources prepare later exercises, such as the Strands agent image repository, RAG storage, and a Valkey cache.

The architecture screenshot in my notes helped me orient myself before deploying workloads. It shows the EKS control plane and managed capabilities alongside the system and general-purpose pools, with monitoring components sending metrics to Amazon Managed Prometheus.

The other cluster screenshot records the NodePools and monitoring Pods and shows that I restricted the load balancer’s inbound CIDR to my own public IP during setup.

My first hands-on lesson came before model serving. Creating an On-Demand Capacity Reservation for a g6e.xlarge in us-east-1 initially failed with InstanceLimitExceeded, because the account’s applicable vCPU quota was zero. Once a reservation is available, it still has to be connected to the GPU NodeClass: I retrieved its reservation ID, added it to the NodeClass selector, and checked the resulting configuration.

That distinction matters. Creating a reservation does not, by itself, tell EKS Auto Mode to use it. After enabling the GPU NodePool with the ODCR active, I captured the updated architecture.

The GPU pool gives accelerator workloads a dedicated scheduling target. A small nvidia-smi Pod can request a GPU and serve as a first check that a suitable node is provisioned and that the container can see the device. This is a useful infrastructure check before loading a model; it is not an inference benchmark.

The workshop model is Ministral-3-8B-Instruct-2512. Its weights and supporting configuration and tokenizer files are stored in S3. In my setup, a Kubernetes Job stages the model files in the bucket:

For the vLLM serving deployment, Run:ai Model Streamer reads the weights directly from S3, so that particular model-serving Pod does not need a PersistentVolume or PersistentVolumeClaim. The Terraform environment also prepares the S3 CSI driver and an S3-backed PV/PVC for other workshop uses.

The model-serving workload runs as a Kubernetes Deployment and Service using an AWS Deep Learning Container with vLLM. The Pod requests GPU resources; if the GPU pool has no suitable node ready, EKS Auto Mode can provision one. EKS Pod Identity provides the workload’s AWS access to S3 through its service account.

vLLM is a useful fit for this exercise because it is designed for generative inference. Continuous batching helps serve concurrent requests, while PagedAttention manages the memory used by the model’s KV cache. Its OpenAI compatible API also gives applications a familiar way to call the model. These are capabilities of the serving engine, not performance results measured in my environment.

The workshop first tests the vLLM Service with a local port-forward and a completion request. The API response includes generated text and token usage, providing a basic functional check before adding an interface.

I also deployed Open WebUI and accessed it through an Application Load Balancer provisioned from a Kubernetes Ingress. This makes the distinction between the model endpoint and a user-facing application concrete: vLLM serves inference; Open WebUI provides a browser interface that calls the model.

The baseline monitoring stack includes kube-prometheus-stack, Node Exporter, kube-state-metrics, Amazon Managed Prometheus, Grafana, and the Grafana Operator. Prometheus collects cluster and application metrics and sends them to AMP; Grafana queries those metrics. EKS Pod Identity supplies AWS permissions to the monitoring workloads.

The workshop adds NVIDIA DCGM Exporter for GPU telemetry such as utilization, memory, temperature, and power. It then introduces vLLM metrics and dashboards for signals such as token generation, latency, and request processing. These layers answer different questions: whether the cluster is healthy, whether the GPU is being used, and how the inference service is handling requests. My screenshots show the monitoring Pods in the cluster; detailed metric values still need to be recorded from the dashboards during a run.

The Strands module adds an application that uses the vLLM endpoint. The sample agent is packaged as a Python container, published to the pre-created ECR repository, and deployed to EKS. It exposes an HTTP endpoint and registers time and weather tools that the model can call when a question needs external information. In this example, each request creates a fresh agent instance, so it does not keep conversation history between requests.

I tested the deployed agent through its service. For “What time is it in Dallas?” it returned 3:00 PM CDT. For “What is the weather in Denver?” it returned clear sky and 25.9°C. These responses are functional checks from one point in time, not benchmarks; local time and weather can change. The screenshot in my notes captures both outputs.

When a tool is needed, the model first selects the appropriate tool call; after the tool returns data, the model uses that result to compose the answer. This shows how a self-hosted model endpoint can become part of a larger application flow, with the agent and its tools running as Kubernetes workloads.

The workshop connected several familiar EKS concerns capacity planning, scheduling, identity, networking, storage, and observability to the particular needs of inference. The most useful lesson so far is to treat model serving as a system with distinct layers: accelerator infrastructure, model delivery, inference API, application integration, and telemetry.

My EKS background helped me reason about how those layers fit together. The workshop gave me a concrete way to inspect them, from GPU provisioning and S3 access to a working model endpoint and an agent that uses external tools. I would next capture repeatable inference metrics and compare them only under a clearly documented model, instance, and request configuration.