AWS Builder Center

Builder Center

Building a RLHF Training Platform on Amazon EKS

Building a RLHF Training Platform on Amazon EKS

Reinforcement Learning from Human Feedback (RLHF) has become the gold standard for fine-tuning LLMs, transforming raw pre-trained models into the helpful assistants we interact with daily. But here's the problem: RLHF is computationally expensive, operationally complex, and notoriously difficult to scale. This guide shows you how to build a production-ready RLHF training platform on Amazon EKS that's simple, scalable and cost-effective.

Jagdeep Phoolkumar

Specialist Solutions Architect

Introducing Reinforcement Learning from Human Feedback (RLHF)

The landscape of artificial intelligence has been transformed by large language models (LLMs), but their true potential is unlocked only when they're properly aligned with human values and preferences. Training these models with human feedback has become not just important, but essential for building AI systems that are helpful, harmless, and honest.
Traditional supervised learning approaches fall short when it comes to capturing the nuanced, subjective nature of human preferences. How do you quantify "helpfulness" or "safety" in a loss function? How do you teach a model to be "friendly" using only labeled data? These qualities are inherently difficult to formalize, yet they're critical for AI systems that interact with humans. This is precisely where Reinforcement Learning from Human Feedback (RLHF) shines.
RLHF represents a paradigm shift in how we train language models. Instead of relying solely on static datasets, RLHF enables models to learn dynamically from human preferences through an iterative feedback loop. It's the secret sauce that powers ChatGPT's conversational abilities, Claude's safety-conscious responses, and countless other aligned AI systems that millions of users interact with daily.

Challenges with implementing RLHF at production scale

Implementing RLHF at production scale introduces a unique set of engineering challenges:
  • Distributed training infrastructure: RLHF requires coordinating multiple models (actor, critic, reward, reference) across dozens of GPUs, each with different computational requirements
  • Efficient data pipelines: Preprocessing, validation, and continuous data flow must be seamless to avoid bottlenecks
  • Cost optimization: GPU resources are expensive, and RLHF training can run for days or weeks, making cost management critical
  • Scalable inference: Deploying trained models for real-time predictions requires high-throughput, low-latency serving infrastructure
  • Reliability: Training jobs must gracefully handle infrastructure failures, especially when using cost-effective spot instances
This guide shows you how to build an end-to-end RLHF training platform on AWS EKS using the JARK stack ( JupyterHub ,  Argo ,  Ray , Karpenter). We trained a  Qwen2.5-7B  model on the  GSM8K  dataset using VERL framework, achieving 50-70% cost savings through  Amazon EC2 Spot Instances .

What You'll Learn

By the end of this guide, you'll understand how to:
  • Deploy a complete RLHF platform using AI on EKS Blueprints
  • Coordinate multiple models across distributed GPUs using Ray and VERL
  • Optimize costs by 50-70% using spot instances with checkpoint recovery
  • Build efficient data pipelines with FSx for Lustre and Argo Workflows
  • Deploy models for inference using vLLM and Ray Serve
  • Handle spot interruptions gracefully without losing training progress

Architecture Overview: The JARK Stack

Our platform is built on the  JARK stack —four open-source tools that work together to provide a complete ML infrastructure:
jark functional diagram

JupyterHub powers interactive development and experimentation, enabling data scientists to explore datasets, prototype algorithms, and debug issues using the same FSx storage and compute resources as production workloads. This eliminates "works on my machine" problems and ensures local environments mirror production conditions.​
Argo Workflows orchestrates complex data pipelines as Kubernetes-native DAGs, enabling reproducible, version-controlled workflows triggered manually or automatically. This is essential for data preprocessing, validation, and continuous data refresh operations where pipeline repeatability and auditability maintain data quality.​
Ray serves as the distributed computing backbone, with its actor model coordinating multiple specialized models (actor, critic, reward, reference) across GPU workers simultaneously. Its automatic GPU scheduling and fault tolerance make it ideal for RLHF workloads requiring dozens of GPUs across multiple nodes.​
Karpenter provides intelligent cluster auto-scaling that provisions the exact instance type needed—including cost-optimized spot instances—in 30–90 seconds, dramatically faster than traditional autoscalers. It scales down idle nodes within minutes, achieving 30–40% cost reductions through right-sizing and spot instance optimization while maintaining flexibility to handle sudden GPU-intensive demand spikes.

Architecture Diagram

Architecture of JARK Stack


Key Components

Compute:
  • CPU Nodes (m7i, c7i): JupyterHub, Argo, Ray head node, data prep
  • GPU Nodes (g6e.48xlarge): 8× NVIDIA L40S GPUs (48GB each), spot instances for 77-90% cost savings
Storage:
  • Amazon FSx for Lustre: 1.2TB shared storage at  /fsx , delivering up to 100GB/s throughput (12.5GB/s per client with standard configuration, scalable to 150GB/s with EFA+GDS)
  • Amazon EFS: Persistent notebooks and user configurations
  • Amazon ECR: Container image registry
Observability:
  • Prometheus, Grafana, CloudWatch for infrastructure monitoring
  • Ray Dashboard for distributed job tracking and real-time GPU utilization

Why This Architecture?

1. Solves Distributed Training Challenges

Ray + VERL's 3D-HybridEngine automatically coordinates four models (actor, critic, reward, reference) across 16 GPUs with different parallelism strategies, eliminating memory redundancy during phase transitions between training (tensor parallel + data parallel) and generation (tensor parallel only). The hybrid engine provides intelligent GPU allocation based on model sizes, delivering competitive performance for RLHF workloads at scale.

2. Eliminates Data Pipeline Bottlenecks

FSx for Lustre provides 100GB/s+ aggregate throughput with sub-millisecond latency, eliminating I/O bottlenecks across the entire training pipeline. All components (JupyterHub, training jobs, inference services) share the same  /fsx  mount point, eliminating data copying overhead and ensuring consistent dataset access. A typical data preprocessing workload (7,473 training samples) completes in under 5 minutes.

3. Achieves Significant Cost Savings

Spot Instances + Karpenter provision exact instance types needed and scale down idle nodes automatically. g6e.48xlarge spot pricing varies by region but typically offers 50-70% savings versus on-demand. Checkpoints saved to FSx every 10 epochs enable recovery within minutes after spot interruptions (2-minute notice period), with graceful shutdown hooks preventing data loss. A typical 48-hour training run costs approximately $330-462 on Spot versus $1,446 On-Demand, saving $984-1,116 per run.

4. Enables High-Performance Inference

vLLM + Ray Serve delivers production-grade serving with PagedAttention reducing KV cache memory waste by 60-80%, achieving 2-3× higher throughput versus traditional implementations. Continuous batching dynamically handles variable request loads, while tensor parallelism distributes models across 4 GPUs. Auto-scaling based on traffic patterns ensures efficient resource utilization with sub-second latency for typical inference requests.

5. Ensures Reliability

Checkpoint recovery + Karpenter auto-recovery provides comprehensive fault tolerance. Checkpoints persist to FSx storage (surviving node failures), while  preStop  hooks ensure graceful shutdown during planned disruptions. When spot interruptions occur, Karpenter provisions replacement nodes within 30-90 seconds while Ray's automatic fault tolerance redistributes tasks to healthy workers. VERL's checkpoint manager enables seamless training resumption from the last saved state, achieving zero training loss from infrastructure failures.

Understanding VERL

VERL (Volcano Engine Reinforcement Learning) is an open-source framework that makes RLHF training flexible, efficient, and production-ready. It implements the  HybridFlow  architecture, enabling seamless scaling from small models to 671B parameter MoE systems used by leading organizations like DeepSeek and ByteDance.​
Core Innovations:
  • Hybrid Programming Model: Single-controller coordination (Python) with distributed Ray actors for efficient execution, eliminating complex manual configuration
  • 3D-HybridEngine: Automatically adapts parallelism between training (tensor parallel + data parallel) and generation phases (tensor parallel only) while eliminating memory redundancy​
  • Auto-Mapping Algorithm: Intelligently allocates GPU resources to actor, critic, reward, and reference models based on model sizes without manual tuning​
Performance & Algorithm Support: Recent benchmarks show VERL with FSDP backend achieves comparable performance to OpenRLHF, with the hybrid engine's fundamental advantage in zero-redundancy phase transitions during training-to-generation shifts. VERL supports modern RL algorithms including PPO, GRPO, DAPO (50 points on AIME 2024), ReMax, Safe-RLHF, RLOO, REINFORCE++, and PRIME, with seamless integration across PyTorch FSDP, Megatron-LM for large model training, and vLLM/SGLang for inference.​

Code Walkthrough

This section demonstrates the practical implementation of the JARK stack infrastructure, from initial deployment through production RLHF training.

1. Infrastructure Setup (30 minutes)

Deploy the complete JARK stack using  AI on EKS  Blueprints, which handles all infrastructure provisioning automatically:
# Clone the AI on EKS Blueprints repository
git clone https://github.com/awslabs/ai-on-eks.git
cd ai-on-eks/infra/jark-stack/terraform

# Deploy infrastructure with one command
./install.sh

The  install.sh  script automates the entire infrastructure setup:
  • Provisions EKS cluster (v1.33) with optimized managed node groups
  • Deploys dual storage: FSx for Lustre (high-performance training) + EFS (persistent notebooks)
  • Installs JARK stack: JupyterHub, Argo Workflows, Ray, Karpenter
  • Adds enterprise components: ArgoCD (GitOps), Argo Events (event-driven pipelines), Volcano (batch scheduler)
  • Deploys AI/ML observability: Prometheus, Grafana, CloudWatch, Ray Dashboard
  • Configures Karpenter for cost-optimized GPU auto-scaling with spot instances
  •  Volcano  manages job priorities and resource fairness
Infrastructure Configuration ( blueprint.tfvars ):
# Core cluster configuration
name = "aiml-stack"

# JARK Stack - Core Components
enable_jupyterhub           = true  # Interactive development
enable_kuberay_operator     = true  # Distributed Ray clusters
enable_argo_workflows       = true  # DAG pipeline orchestration
enable_aws_fsx_csi_driver   = true  # High-performance storage
deploy_fsx_volume           = true  # Auto-provision Lustre filesystem

# Storage Layer
enable_aws_efs_csi_driver   = true  # Persistent notebooks/configs

# Enhanced Orchestration & GitOps
enable_argo_events          = true  # Event-driven ML pipelines
enable_argocd               = true  # GitOps deployments
enable_volcano              = true  # Advanced batch scheduling

# Observability Stack
enable_ai_ml_observability_stack = true  # GPU metrics, Ray monitoring

# Documentation Indexing (optional for RAG)
enable_aws_docs_crawler     = true
This configuration creates a production-ready ML infrastructure with automatic GPU provisioning, persistent storage, and built-in monitoring—all deployable in under 30 minutes.

2. Data Preparation (Kubernetes Job)

Preprocess the GSM8K dataset and save to FSx shared storage for distributed training:
apiVersion: batch/v1
kind: Job
metadata:
  name: gsm8k-data-prep
  namespace: default
spec:
  template:
    spec:
      containers:
      - name: data-prep
        image: python:3.11-slim
        command: ["/bin/bash", "-c"]
        args:
          - |
            # Install dependencies
            apt-get update && apt-get install -y git
            pip install datasets transformers
            
            # Clone VERL repository
            git clone https://github.com/volcengine/verl.git
            cd verl && pip install -e .
            
            # Run GSM8K preprocessing script
            cd examples/data_preprocess
            python3 gsm8k.py --local_save_dir /fsx/training_data/gsm8k
            
            # Verify output files
            echo "Data preprocessing complete:"
            ls -lh /fsx/training_data/gsm8k
        volumeMounts:
        - name: fsx-storage
          mountPath: /fsx
      volumes:
      - name: fsx-storage
        persistentVolumeClaim:
          claimName: fsx-static-pvc
      restartPolicy: OnFailure

Run the data preparation job:
# Apply the job configuration
kubectl apply -f gsm8k-data-prep-job.yaml

# Monitor job progress
kubectl logs -f job/gsm8k-data-prep

# Check job status
kubectl get jobs

Expected output:​
Data preprocessing complete:
train.parquet: 7,473 samples (1.1MB)
test.parquet:  1,319 samples (195KB)

The preprocessing script transforms raw GSM8K data into VERL's expected format with prompt/response structure and rule-based reward model configuration for math verification.​

3. RLHF Training (RayJob)

Deploy a Ray cluster and submit VERL PPO training job with automatic GPU provisioning:
apiVersion: ray.io/v1
kind: RayJob
metadata:
  name: verl-qwen3-training
  namespace: rayserve-vllm
spec:
  # Training entrypoint with VERL PPO configuration
  entrypoint: |
    python3 -m verl.trainer.main_ppo \
      data.train_files=[/fsx/training_data/gsm8k/train.parquet] \
      data.val_files=[/fsx/training_data/gsm8k/test.parquet] \
      data.train_batch_size=256 \
      data.val_batch_size=1312 \
      actor_rollout_ref.model.path=Qwen/Qwen2.5-7B-Instruct \
      actor_rollout_ref.actor.optim.lr=1e-6 \
      actor_rollout_ref.actor.ppo_mini_batch_size=64 \
      actor_rollout_ref.actor.ppo_micro_batch_size_per_gpu=4 \
      critic.model.path=Qwen/Qwen2.5-7B-Instruct \
      critic.optim.lr=1e-5 \
      algorithm.kl_ctrl.kl_coef=0.001 \
      trainer.n_gpus_per_node=8 \
      trainer.nnodes=2 \
      trainer.total_epochs=15 \
      trainer.save_freq=5 \
      trainer.test_freq=5 \
      trainer.default_local_dir=/fsx/checkpoints/qwen25-7b-verl-ppo
  
  # Configure Ray cluster topology
  rayClusterSpec:
    rayVersion: '2.50.0'
    
    # Head node (CPU-only for coordination)
    headGroupSpec:
      rayStartParams:
        dashboard-host: '0.0.0.0'
        num-cpus: "8"
      template:
        spec:
          containers:
          - name: ray-head
            image: <AWS_ACCOUNT_ID>.dkr.ecr.us-west-2.amazonaws.com/verl-qwen:latest
            resources:
              requests:
                cpu: "8"
                memory: "32Gi"
              limits:
                cpu: "8"
                memory: "32Gi"
            volumeMounts:
            - name: fsx-data
              mountPath: /fsx
          volumes:
          - name: fsx-data
            persistentVolumeClaim:
              claimName: fsx-static-pvc
          nodeSelector:
            NodeGroupType: x86-cpu-karpenter
    
    # GPU workers (2 nodes × 8 GPUs = 16 total)
    workerGroupSpecs:
    - replicas: 2
      groupName: gpu-workers
      minReplicas: 2
      maxReplicas: 2
      rayStartParams:
        num-gpus: "8"
        num-cpus: "96"
        resources: '"{\"accelerator_type_g6e\": 8}"'
      template:
        spec:
          containers:
          - name: ray-worker
            image: <AWS_ACCOUNT_ID>.dkr.ecr.us-west-2.amazonaws.com/verl-qwen:latest
            resources:
              requests:
                cpu: "96"
                memory: "740Gi"
                nvidia.com/gpu: "8"
              limits:
                nvidia.com/gpu: "8"
            volumeMounts:
            - name: fsx-data
              mountPath: /fsx
            - name: shm
              mountPath: /dev/shm
            # Graceful shutdown on spot interruption
            lifecycle:
              preStop:
                exec:
                  command: ["/bin/bash", "-c", "ray stop --grace-period=30; sleep 30"]
          volumes:
          - name: fsx-data
            persistentVolumeClaim:
              claimName: fsx-static-pvc
          - name: shm
            emptyDir:
              medium: Memory
              sizeLimit: "256Gi"
          nodeSelector:
            karpenter.sh/capacity-type: spot
            node.kubernetes.io/instance-type: g6e.48xlarge
          tolerations:
          - key: nvidia.com/gpu
            operator: Exists
            effect: NoSchedule

Key Configuration Explained:
  •  train_batch_size=256 : Global batch size distributed across all 16 GPUs (16 samples per GPU)​
  •  save_freq=5 : Checkpoint every 5 epochs to FSx for spot recovery (not 10—typical recommendation is 5-10)​
  •  kl_coef=0.001 : KL divergence penalty to prevent policy deviation from reference model​
  •  n_gpus_per_node=8, nnodes=2 : Total 16 GPUs across 2 g6e.48xlarge nodes​
  •  preStop  hook: Executes  ray stop --grace-period=30  then waits 30 seconds for graceful shutdown before SIGTERM
  •  shm  volume: 256Gi shared memory for PyTorch multiprocessing and NCCL communication​
Deploy and monitor the training job:
# Deploy training job
kubectl apply -f verl-ray-training.yaml

# Check RayJob status
kubectl get rayjob -n rayserve-vllm
kubectl describe rayjob verl-qwen3-training -n rayserve-vllm

# View training logs (streaming)
kubectl logs -n rayserve-vllm -l ray.io/node-type=head --tail=100 -f

# Access Ray Dashboard for GPU monitoring
kubectl port-forward -n rayserve-vllm svc/verl-qwen3-training-head-svc 8265:8265
# Open browser: http://localhost:8265


4. Model Inference (RayService + vLLM)

Deploy the RLHF-trained model for production inference with automatic scaling:
apiVersion: ray.io/v1
kind: RayService
metadata:
  name: qwen-inference
  namespace: rayserve-vllm
spec:
  serviceUnhealthySecondThreshold: 300
  deploymentUnhealthySecondThreshold: 300
  
  rayClusterConfig:
    rayVersion: '2.50.0'
    
    # Head node
    headGroupSpec:
      rayStartParams:
        dashboard-host: '0.0.0.0'
        num-cpus: "4"
      template:
        spec:
          containers:
          - name: ray-head
            image: <AWS_ACCOUNT_ID>.dkr.ecr.us-west-2.amazonaws.com/verl-qwen:latest
            resources:
              requests:
                cpu: "4"
                memory: "16Gi"
            ports:
            - containerPort: 8265
              name: dashboard
            - containerPort: 8000
              name: serve
    
    # vLLM worker (single g6e.48xlarge with 8 GPUs)
    workerGroupSpecs:
    - replicas: 1
      groupName: vllm-workers
      minReplicas: 1
      maxReplicas: 3
      rayStartParams:
        num-gpus: "8"
        num-cpus: "96"
      template:
        spec:
          containers:
          - name: ray-worker
            image: <AWS_ACCOUNT_ID>.dkr.ecr.us-west-2.amazonaws.com/verl-qwen:latest
            resources:
              requests:
                nvidia.com/gpu: "8"
                memory: "740Gi"
            volumeMounts:
            - name: fsx-data
              mountPath: /fsx
            - name: shm
              mountPath: /dev/shm
          volumes:
          - name: fsx-data
            persistentVolumeClaim:
              claimName: fsx-static-pvc
          - name: shm
            emptyDir:
              medium: Memory
              sizeLimit: "256Gi"
  
  # vLLM serving configuration
  serveConfigV2: |
    applications:
    - name: qwen-serve
      route_prefix: /
      import_path: serve_vllm:app
      runtime_env:
        env_vars:
          MODEL_PATH: "/fsx/checkpoints/qwen25-7b-verl-ppo"
          TENSOR_PARALLEL_SIZE: "4"
          GPU_MEMORY_UTILIZATION: "0.9"
          MAX_MODEL_LEN: "4096"

Serving implementation ( serve_vllm.py ):​
from ray import serve
from vllm import LLM, SamplingParams
import os

@serve.deployment(
    num_replicas=2,
    ray_actor_options={"num_gpus": 4}  # Each replica uses 4 GPUs (TP=4)
)
class VLLMDeployment:
    def __init__(self):
        # Initialize vLLM engine with tensor parallelism
        self.llm = LLM(
            model=os.environ.get("MODEL_PATH", "/fsx/checkpoints/qwen25-7b-verl-ppo"),
            tensor_parallel_size=int(os.environ.get("TENSOR_PARALLEL_SIZE", "4")),
            gpu_memory_utilization=float(os.environ.get("GPU_MEMORY_UTILIZATION", "0.9")),
            max_model_len=int(os.environ.get("MAX_MODEL_LEN", "4096")),
            trust_remote_code=True,
            enable_chunked_prefill=True,  # PagedAttention optimization
            max_num_batched_tokens=8192,
        )
        
        self.sampling_params = SamplingParams(
            temperature=0.7,
            top_p=0.9,
            max_tokens=512,
            stop=["####"]  # GSM8K answer format
        )
    
    async def __call__(self, request):
        prompt = request.query_params.get("prompt", "")
        outputs = self.llm.generate([prompt], self.sampling_params)
        return {
            "response": outputs[0].outputs[0].text,
            "tokens_generated": len(outputs[0].outputs[0].token_ids)
        }

app = VLLMDeployment.bind()

Test the inference endpoint:
# Deploy RayService
kubectl apply -f qwen-inference.yaml

# Wait for service to be ready
kubectl wait --for=condition=ready rayservice/qwen-inference -n rayserve-vllm --timeout=300s

# Test inference
curl -X POST "http://qwen-inference-serve-svc.rayserve-vllm:8000/?prompt=Solve:%20If%20John%20has%205%20apples%20and%20gives%202%20to%20Mary,%20how%20many%20does%20he%20have%20left?"

Expected response:
{
  "response": "Let's think step by step:\n1. John starts with 5 apples\n2. He gives 2 apples to Mary  \n3. 5 - 2 = 3\n\n",
  "tokens_generated": 42
}


5. Karpenter Configuration (GPU Auto-Scaling)

Your Karpenter setup provisions multiple node pools with intelligent spot/on-demand instance selection and consolidation policies:
GPU Node Pools (g6e.48xlarge with L40S GPUs):
# Automatically generated by Terraform data_addons module
# Located in: karpenter_resources_helm_config
apiVersion: karpenter.k8s.aws/v1
kind: EC2NodeClass
metadata:
  name: g6e-gpu-karpenter
spec:
  amiFamily: Bottlerocket  # Optimized container OS
  role: KarpenterNodeRole-aiml-stack
  
  # Subnet selection - secondary CIDR for large IP space
  subnetSelectorTerms:
  - tags:
      karpenter.sh/discovery: "aiml-stack"
      Name: "aiml-stack-private-secondary*"
  
  # Security group
  securityGroupSelectorTerms:
  - tags:
      Name: aiml-stack-node
  
  # Instance store configuration
  instanceStorePolicy: RAID0  # Combine NVMe SSDs for max throughput
  
  # Storage configuration
  blockDeviceMappings:
  - deviceName: /dev/xvda  # Root volume
    ebs:
      volumeSize: 50Gi
      volumeType: gp3
      encrypted: true

---
apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: g6e-gpu-karpenter
spec:
  template:
    metadata:
      labels:
        instanceType: g6e-gpu-karpenter
        type: karpenter
        gpuType: l40s
        accelerator: nvidia
    spec:
      # GPU taint - only GPU pods scheduled here
      taints:
      - key: nvidia.com/gpu
        value: "Exists"
        effect: NoSchedule
      
      # Instance requirements
      requirements:
      - key: karpenter.k8s.aws/instance-family
        operator: In
        values: ["g6e"]
      - key: karpenter.k8s.aws/instance-size
        operator: In
        values: ["2xlarge", "4xlarge", "8xlarge", "12xlarge", 
                 "16xlarge", "24xlarge", "48xlarge"]
      - key: kubernetes.io/arch
        operator: In
        values: ["amd64"]
      - key: karpenter.sh/capacity-type
        operator: In
        values: ["spot", "on-demand"]  # Prefers spot, fallback to on-demand
      
      nodeClassRef:
        group: karpenter.k8s.aws
        kind: EC2NodeClass
        name: g6e-gpu-karpenter
  
  # Resource limits across all nodes in this pool
  limits:
    cpu: 2000
  
  # Disruption/consolidation behavior
  disruption:
    consolidationPolicy: WhenEmpty
    consolidateAfter: 300s    # Terminate empty nodes after 5 minutes
    expireAfter: 720h         # Max node lifetime: 30 days
  
  weight: 100  # Priority for scheduling

CPU Node Pool (x86-cpu-karpenter for Ray head, Argo, JupyterHub):
apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: x86-cpu-karpenter
spec:
  template:
    spec:
      requirements:
      - key: karpenter.k8s.aws/instance-category
        operator: In
        values: ["c", "m", "r"]  # Compute, memory, memory-optimized
      - key: karpenter.sh/capacity-type
        operator: In
        values: ["spot", "on-demand"]
      
      nodeClassRef:
        name: x86-cpu-karpenter
  
  limits:
    cpu: 1000
  
  disruption:
    consolidationPolicy: WhenEmpty
    consolidateAfter: 300s

How Your Karpenter Setup Works:
  1. Multi-Pool Architecture: Separate pools for g6e (L40S), g6 (L4), g5 (A10G), Trainium (trn1), Inferentia (inf2), and CPU workloads—Karpenter selects optimal instance type based on pod requirements
  1. Spot-First Strategy:  capacity-type: ["spot", "on-demand"]  prioritizes spot with automatic on-demand fallback if spot unavailable
  1. Bottlerocket AMI: GPU nodes use Bottlerocket (minimal attack surface, fast boot ~30s) with RAID0 instance store for high-speed local caching
  1. Secondary CIDR Subnets: Nodes deployed in secondary CIDR ranges ( -private-secondary* ) providing larger IP address space for GPU-dense clusters
  1. Consolidation Policy:  WhenEmpty  with 300s delay—Karpenter terminates idle nodes 5 minutes after last pod removal, reducing costs while preventing thrashing
  1. Node Lifecycle: Nodes automatically expire after 720 hours (30 days) and are gracefully replaced to ensure fresh OS patches and prevent drift.
Monitoring Karpenter:
# View active NodePools
kubectl get nodepools

# Check provisioned nodes
kubectl get nodes -l type=karpenter

# View Karpenter controller logs
kubectl logs -n kube-system -l app.kubernetes.io/name=karpenter -f

# Check spot interruption handling
kubectl get events -n kube-system --field-selector reason=SpotInterruption

This configuration ensures your VERL training jobs automatically get g6e.48xlarge spot instances provisioned within 60-90 seconds, scale down idle nodes after 5 minutes, and fallback to on-demand if spot capacity unavailable—achieving massive cost savings with zero manual intervention.​

6. Monitoring and Observability

Ray Dashboard:
The Ray Dashboard provides real-time job tracking, GPU utilization per worker, memory usage, task timeline, and worker health status.​
kubectl port-forward -n rayserve-vllm svc/verl-qwen3-training-head-svc 8265:8265
# Open: http://localhost:8265

Kube-Prometheus Stack:
Standard Kubernetes metrics are automatically scraped and available in Prometheus:
  • Node CPU/memory usage across the cluster
  • Pod resource utilization for Ray workers
  • Karpenter node provisioning and termination metrics
  • Kubecost spending (if enabled)
# Get Grafana admin password
kubectl get secret -n kube-prometheus-stack kube-prometheus-stack-grafana \
  -o jsonpath="{.data.admin-password}" | base64 -d

# Port-forward Grafana
kubectl port-forward -n kube-prometheus-stack svc/kube-prometheus-stack-grafana 3000:80
# Login: admin / <password from above>

# Port-forward Prometheus
kubectl port-forward -n kube-prometheus-stack svc/kube-prometheus-stack-prometheus 9090:9090

Optional:
Out-of-box you get node metrics, pod monitoring, and Karpenter cost tracking. For deeper insights (GPU SM utilization, VERL training loss curves), you'd need to add NVIDIA DCGM exporter or custom metric exporters, but the Ray Dashboard alone provides sufficient visibility for training job monitoring.

Conclusion

This reference architecture demonstrates how to build a production-grade RLHF training platform by combining AI on EKS Blueprints, VERL, Ray, and Karpenter. The JARK stack eliminates traditional ML infrastructure friction—JupyterHub enables interactive experimentation, Argo Workflows manages reproducible pipelines, Ray coordinates distributed training, and Karpenter optimizes costs through intelligent auto-scaling. VERL's 3D-HybridEngine adapts parallelism strategies dynamically, FSx Lustre provides high-performance shared storage, and spot instances with graceful checkpointing deliver 50–70% cost savings without sacrificing reliability. By leveraging open-source components on EKS, research teams and organizations can train aligned AI models at scale with minimal infrastructure overhead, from initial development through production inference.

Disclaimer and License

Disclaimer: This blog post is provided for educational purposes only. The views expressed are those of the author and do not necessarily reflect the official policy of Amazon Web Services, Inc. This is not official AWS documentation.
Code License: All source code examples are licensed under the MIT-0 License.
Cost Estimates: Cost estimates are approximate and based on specific configurations. Actual costs may vary. Use the  AWS Pricing Calculator  for detailed estimates.
Production Use: This is a reference architecture for educational purposes. Conduct thorough testing, security reviews, and compliance assessments before production deployment.
# ai-ml# ai-for-developers# ai-engineering# amazon-eks# reinforcement-learning
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article