AWS Builder Center

Builder Center

Choosing Where to Run LLM Inference on AWS: EC2, EKS, SageMaker, HyperPod, and Bedrock

A practical comparison of GPU compute options for LLM inference on AWS — when to use raw EC2, when containers make sense, when managed services earn their premium, and when you shouldn't manage infrastructure at all.

Arya Subramanyam

AWS silicon note: This post covers NVIDIA GPU paths. For cost-optimized inference at scale on AWS-designed hardware, Trainium2 ( trn2 ) and Inferentia2 ( inf2 ) instances with the Neuron SDK are worth evaluating — vLLM has Neuron backend support for both and unit economics can differ significantly from P5/P6 at sustained throughput.

The Decision

You've decided to host a model yourself (or at least evaluate it). Now: where do you run it?
AWS offers at least five distinct paths to GPU inference, each with a different trade-off surface. The internet discourse tends to reduce this to "EC2 for control, SageMaker for convenience" — but the reality is more nuanced. Your choice depends on your operational maturity, latency requirements, scale profile, and how much undifferentiated heavy lifting you're willing to absorb.
This post maps out the options, what each is actually good at, and when to pick one over the other.

The Options at a Glance

Option
What it is
You manage
AWS manages
Raw EC2
GPU instance + Docker container
Everything: deployment, scaling, monitoring, failover
Hardware, hypervisor
EKS (Kubernetes)
GPU node pools + Kubernetes workloads
Cluster ops, pod scheduling, autoscaling logic
Control plane, node provisioning
SageMaker Inference
Managed model endpoint
Model container, instance selection
Deployment, scaling, load balancing, health checks
SageMaker HyperPod
Managed GPU cluster (primarily training, also supports persistent inference)
Workload orchestration, Slurm/K8s scheduling
Cluster lifecycle, health monitoring, node replacement
Amazon Bedrock
Fully managed API
Nothing
Everything

Raw EC2: Maximum Control, Maximum Responsibility

What it is: Launch a GPU instance, run  docker run  with vLLM, TGI, or TensorRT-LLM. You own the entire stack from container to response.
When it wins:
  • Latency benchmarks and PoCs where you need to isolate model performance from platform overhead
  • Single-model, single-tenant deployments where autoscaling isn't needed
  • Config iteration — restart a container with new parameters in minutes without redeploying infrastructure
  • Cost predictability with Capacity Blocks (flat $/hr, no per-token variability)
  • Maximum latency control — no request routing, no load balancer hops, no queue
What you give up:
  • No autoscaling — if traffic spikes, you're manually scaling or dropping requests
  • No health-check-driven replacement — if the GPU faults or the container OOMs, nobody restarts it but you
  • No multi-model serving out of the box — one container per base model (LoRA adapters allow multi-variant serving from a single container)
  • Monitoring is DIY — Prometheus metrics from vLLM → CloudWatch agent → dashboards, all manual setup
Best for: Benchmarking, PoCs, dedicated single-model inference for a known workload, price-sensitive deployments where you can tolerate ops overhead.
Typical deploy pattern:
Bash
# One command to serving
docker run -d --gpus all --ipc=host -p 8000:8000 \
  vllm/vllm-openai:latest \  # pin to a specific version tag in production for reproducibility
  --model <your-model-id> \
  --tensor-parallel-size 4 \
  --enable-prefix-caching \
  --enable-chunked-prefill

Time to first inference: ~20 minutes for smaller models pulling from HuggingFace Hub; larger models (70B+) can take significantly longer on a cold instance — pull from S3 or pre-bake weights into an AMI for production to avoid cold-start delays.

EKS: Container Orchestration for GPU Workloads

What it is: Kubernetes cluster with GPU node groups (P5, P6, G6e). You deploy inference servers as pods using Karpenter or Cluster Autoscaler for node provisioning.
When it wins:
  • Teams already running Kubernetes for other services — inference becomes another workload, not a separate platform
  • Multi-model serving with shared GPU node pools — Kubernetes handles scheduling and bin-packing
  • Canary deployments and traffic splitting between model versions
  • Horizontal scaling with custom metrics (queue depth, latency percentiles)
  • Organizations with platform engineering teams who maintain shared clusters
What you give up:
  • Kubernetes complexity — GPU scheduling, device plugins, topology-aware placement, NCCL configuration
  • Karpenter/Autoscaler tuning for GPU nodes (scale-up time is 5-20 min for P5/P6)
  • You still manage the inference runtime (vLLM pod configuration, health probes, resource requests)
  • Multi-node tensor parallelism across pods requires additional orchestration (LeaderWorkerSet, MPI operator)
Best for: Platform teams serving multiple models to internal consumers, organizations standardized on Kubernetes, workloads that need traffic management (A/B testing model versions).
Typical pattern:
YAML
apiVersion: apps/v1
kind: Deployment
metadata:
  name: llm-inference
spec:
  replicas: 2
  template:
    spec:
      nodeSelector:
        nvidia.com/gpu.present: "true"  # required — prevents scheduling on non-GPU nodes
      containers:
      - name: vllm
        image: vllm/vllm-openai:latest  # pin to a specific version tag in production
        args: ["--model", "<your-model-id>", "--tensor-parallel-size", "4"]
        resources:
          limits:
            nvidia.com/gpu: 4

Related:  Karpenter GPU support ,  EKS inference best practices 

SageMaker Inference: Managed Endpoints

What it is: You provide a model container (or use a prebuilt one), SageMaker handles deployment behind a managed HTTPS endpoint with autoscaling, health checks, and A/B traffic routing.
When it wins:
  • Production workloads that need autoscaling without writing scaling logic
  • Teams without infrastructure engineers who want model → endpoint in minutes
  • Multi-variant endpoints — route 90% to the stable model, 10% to the candidate
  • Built-in model monitoring (data drift, inference quality) via SageMaker Model Monitor
  • Integration with SageMaker Pipelines for CI/CD of model deployments
What you give up:
  • Cold start latency when scaling up new instances — 5-15 minutes for large models
  • Less config flexibility than raw EC2 — some inference server flags aren't exposed through the container interface
  • Managed routing adds some latency overhead vs direct container access
  • Pricing includes a management premium on top of instance costs
  • Debugging inference issues requires navigating container logs through CloudWatch rather than  docker logs 
Best for: ML teams shipping to production who want operational simplicity, don't need sub-millisecond latency optimization, and value autoscaling + monitoring out of the box.
Typical pattern:
Python
import sagemaker
from sagemaker.huggingface import HuggingFaceModel

# For large models (70B+), pull directly from HuggingFace Hub
# rather than packaging as model.tar.gz (impractical at scale)
hub_env = {
    "HF_MODEL_ID": "<your-hf-model-id>",
    "HF_TASK": "text-generation",
    "SM_NUM_GPUS": "8",
}

# Note: DJL Inference DLCs run DJL backends, not vLLM directly.
# For vLLM, either use the TGI DLC (framework="huggingface-tgi") or bring your own
# container built from the vLLM image. Find available DLC URIs at:
# https://github.com/aws/deep-learning-containers/blob/master/available_images.md
dlc_image_uri = sagemaker.image_uris.retrieve(
    framework="huggingface-tgi",  # or use a custom vLLM container URI
    region="us-east-1",
    version="2.3.1",
)

model = HuggingFaceModel(
    env=hub_env,
    role=role,
    image_uri=dlc_image_uri,
)

predictor = model.deploy(
    instance_type="ml.p5.48xlarge",
    initial_instance_count=1,
    endpoint_name="llm-endpoint",
)


SageMaker HyperPod: Managed GPU Clusters

What it is: Primarily a managed cluster of GPU instances designed for large-scale training, with Slurm or EKS orchestration. AWS handles node health monitoring, automatic replacement of faulty nodes, and cluster lifecycle. HyperPod Inference extends this to managed production serving — worth evaluating if you're already on HyperPod for training.
When it wins:
  • Multi-node inference for models requiring pipeline parallelism across nodes
  • Organizations running both training and inference on the same GPU fleet
  • Long-running GPU allocations where node failures are expected — HyperPod automatically detects and replaces unhealthy nodes
  • Shared GPU clusters across teams with Slurm-based job scheduling
  • Workloads that benefit from persistent cluster state (model weights pre-loaded across warm nodes)
What you give up:
  • Cluster management overhead (less than raw EC2, more than SageMaker endpoints)
  • Slurm learning curve if your team isn't familiar with HPC scheduling
  • Cost of reserved capacity — you pay for the cluster whether or not it's running inference
Best for: Large-scale multi-node inference, organizations with HPC backgrounds, teams that need dedicated GPU clusters with automated health management and are already using HyperPod for training.

Amazon Bedrock: Don't Manage Infrastructure At All

What it is: Fully managed API endpoint. You call a model by name, pay per token, and AWS handles everything — hardware, scaling, availability.
When it wins:
  • You want inference without infrastructure — no instances, no containers, no scaling policies
  • Multi-model access — switch between models with a model ID change
  • Variable or unpredictable traffic — pay-per-token means no idle instance cost
  • Compliance and governance requirements that benefit from AWS-managed security controls
  • Time to production measured in minutes, not days
What you give up:
  • No config control — you can't tune tensor parallelism, prefix caching, or batch sizes
  • Per-token pricing can be more expensive at high sustained throughput vs reserved capacity
  • Latency includes managed routing overhead — typically higher than self-hosted at low concurrency
  • Less observability into inference internals (no  /metrics  endpoint, no decode/prefill breakdown)
  • Limited to models in the Bedrock catalog by default — though Bedrock Custom Model Import now allows importing weights from S3 for supported architectures, so check current support before ruling Bedrock out for custom models.
Best for: Applications where inference is a feature, not the product. Prototyping, multi-model workloads, variable traffic, teams without GPU infrastructure expertise.
Python
import boto3, json

bedrock = boto3.client("bedrock-runtime", region_name="us-east-1")

response = bedrock.invoke_model(
    modelId="amazon.nova-pro-v1:0",  # example — use your target model's Bedrock model ID
    body=json.dumps({
        "messages": [{"role": "user", "content": "Hello!"}],
        "max_tokens": 50
    })
)


Decision Framework

Rather than a simple flowchart, here are the questions that actually determine the right choice:
"How much latency can I tolerate, and do I need to tune for it?"
  • Need to tune inference config (tensor parallelism, prefix cache, chunked prefill) → EC2 or EKS
  • Willing to accept managed defaults → SageMaker Inference or Bedrock
  • Latency isn't the primary concern → Bedrock (simplest)
"Do I need autoscaling?"
  • Yes, with variable traffic → SageMaker Inference or Bedrock
  • Yes, but I'm on Kubernetes → EKS with Karpenter
  • No, traffic is predictable → EC2 with Capacity Blocks
"How many models am I serving?"
  • One model, dedicated → EC2
  • 2-5 models, shared infrastructure → EKS or SageMaker (Inference Components)
  • 10+ models, mix-and-match → Bedrock
"What's my operational maturity?"
  • Full platform engineering team → EKS or EC2
  • ML team with some infra experience → SageMaker
  • Application developers, no infra team → Bedrock
"Is this a benchmark / PoC or production?"
  • Benchmark / PoC with config iteration → EC2 (fastest iteration loop)
  • Production with SLA → SageMaker or EKS (operational safety nets)

Hybrid Patterns

In practice, many organizations use multiple options:
  • Benchmark on EC2, deploy to SageMaker — iterate fast on raw instances, then move the winning config to a managed endpoint for production
  • Bedrock for development, self-hosted for production — prototype with Bedrock's pay-per-token, then self-host on EC2/EKS once the workload is proven and cost-optimize
  • EKS for multi-model, EC2 for latency-critical — shared cluster handles most models, dedicated instances for the latency-sensitive path
  • HyperPod for training, SageMaker Inference for serving — train on the managed cluster, deploy the artifact to a managed endpoint

Summary

There's no universally "right" answer. The choice is a function of your latency requirements, operational maturity, traffic patterns, and how much infrastructure you're willing to own.
If you need...
Use...
Absolute minimum latency, full config control
Raw EC2
Container orchestration, multi-model, team familiarity
EKS
Managed scaling, production safety nets, low ops
SageMaker Inference
Multi-node inference, persistent clusters, auto-healing
SageMaker HyperPod
Zero infrastructure, pay-per-token, fastest time to production
Amazon Bedrock
Start with the simplest option that meets your requirements. You can always move to more control later — but you can't easily get back the operational simplicity you gave up.

Resources

Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article