AWS Builder Center

Deploying OpenAI GPT-OSS Models on Amazon EC2 GPU Instances with vLLM

How to self-host OpenAI's open-source GPT-OSS-120B on EC2 P6 instances with vLLM with prefix caching, structured output, and OpenAI SDK compatibility.

Why Self-Host on EC2?

OpenAI released two open-weight models — GPT-OSS-120B and GPT-OSS-20B — both using MXFP4 quantization on their MoE weights, enabling the 120B to fit on a single 80 GB GPU (NVIDIA H100, AMD MI300X, or Blackwell) and the 20B to run within just 16 GB of memory. This guide focuses on the 120B for frontier-class inference, but the same setup applies to the 20B with smaller hardware requirements. Blackwell has native MXFP4 Tensor Core support which is expected to give better throughput than Hopper-class hardware, but OpenAI officially targets H100 and MI300X as primary deployment platforms — both work well. This means you can run frontier-class inference on your own infrastructure with full control over configuration, latency, and cost.
Three reasons to consider raw EC2 over managed inference for latency-sensitive workloads:
  1. No routing overhead — direct Docker container, no load balancer hops or managed service queuing
  1. Full config control — tensor parallelism, prefix caching, chunked prefill, KV cache tuning
  1. Predictable cost — flat hourly rate via Capacity Blocks, no per-token pricing surprises at scale
The trade-off is clear: you own the ops. No autoscaling, no multi-model routing, no managed failover. For single-model latency benchmarks or dedicated inference for a specific workload, this is the right tool.

Choosing a GPU Instance

EC2 offers several GPU instance families suitable for LLM inference. The right choice depends on model size, latency requirements, and budget:
Instance Family
GPUs
GPU Memory
Best For
P6
8× B200 or B300
1.4-1.5 TB HBM3e
Frontier models (100B+), lowest latency, native MXFP4 support
P5en / P5e
8× H200
1.1 TB HBM3e
Large models (70B+), strong price-performance
P5
8× H100
640 GB HBM3
70B models, widely available
G6e
up to 8× L40S
up to 384 GB
Mid-size models (7B-30B), cost-efficient for development
G6
up to 8× L4
up to 192 GB
Smaller models (7B-13B), inference at scale
For GPT-OSS-120B specifically, you need P5 or higher — the model requires at least ~90 GB of GPU memory across your tensor parallel group. On P6 with MXFP4 quantization, the model fits comfortably across 4 GPUs with headroom for KV cache — use  --tensor-parallel-size 4  for a cost-efficient configuration, or  --tensor-parallel-size 8  to maximise throughput.
Key consideration: MXFP4 quantization (used by GPT-OSS-120B) is optimized for Blackwell's native Tensor Core support. Running on Hopper (P5/P5e) is possible but uses different quantization kernels with potentially different performance characteristics.

Architecture


  • Instance: Any NVIDIA GPU instance (P6, P5e, P5, G6e) with sufficient VRAM
  • Serving: vLLM with OpenAI-compatible API (Chat Completions + Responses API)
  • Access: IP-locked security group, SSM Session Manager for shell access
  • Deployment: Launch instance → run Docker container → healthy in ~20 minutes

Prerequisites

  • AWS account with GPU instance quota (P6, P5, or G6e depending on model size)
  • AWS CLI configured
  • A public subnet with internet access (for model download from HuggingFace)
  • Security group with port 8000 open to your IP

Step 1: Get a GPU Instance

There are several ways to procure GPU capacity on EC2:

Option 1: On-Demand

The simplest path — launch an instance directly if you have quota. No commitment, pay by the second, but availability isn't guaranteed for GPU instances.
aws ec2 run-instances \
--instance-type p6-b200.48xlarge \
--image-id ami-0e2e1c9b9d71cc77f \
--subnet-id subnet-xxxxx \
--security-group-ids sg-xxxxx \
--iam-instance-profile Name=SSMAccessRole \
--block-device-mappings '[{"DeviceName":"/dev/sda1","Ebs":{"VolumeSize":500,"VolumeType":"gp3"}}]' \
--region us-east-1

Option 2: Capacity Blocks

Reserve GPU capacity for a defined window (hours to weeks) at a fixed hourly rate. Best for planned benchmarks or time-bound workloads where you need guaranteed availability.
# Find available Capacity Blocks
aws ec2 describe-capacity-block-offerings \
--instance-type p6-b200.48xlarge \
--capacity-duration-hours 24 \
--region us-east-1

# Purchase a block
aws ec2 purchase-capacity-block \
--capacity-block-offering-id cbr-xxxxx \
--instance-platform Linux/UNIX \
--region us-east-1
Then launch into the reservation:
aws ec2 run-instances \
--instance-type p6-b200.48xlarge \
--capacity-reservation-specification \
"CapacityReservationTarget={CapacityReservationId=cr-xxxxx}" \
--image-id ami-0e2e1c9b9d71cc77f \
--subnet-id subnet-xxxxx \
--region us-east-1

Option 3: On-Demand Capacity Reservations (ODCRs)

For longer-term dedicated capacity without Savings Plan commitment. Useful when you need guaranteed availability but want flexibility to start/stop.

Option 4: Spot Instances

Up to 90% discount but can be interrupted. Suitable for non-production benchmarking and experimentation — not recommended for customer-facing workloads.
aws ec2 request-spot-instances \
--instance-count 1 \
--type one-time \
--launch-specification '{
"ImageId": "ami-0e2e1c9b9d71cc77f",
"InstanceType": "p6-b200.48xlarge",
"SubnetId": "subnet-xxxxx"
}' \
--region us-east-1

AMI Selection

Use the latest Deep Learning OSS NVIDIA Driver AMI — it comes with Docker, NVIDIA drivers, and the container toolkit pre-installed:
# Resolves to the latest NVIDIA GPU AMI in any region
aws ssm get-parameter \
--name "/aws/service/deeplearning/ami/x86_64/oss-nvidia-driver-gpu-pytorch-2.12-ubuntu-24.04/latest/ami-id" \
--region us-east-1 --query "Parameter.Value" --output text
⚠️ Do not use Neuron AMIs (designed for Trainium/Inferentia) — they lack NVIDIA drivers and the container toolkit. If you see "could not select device driver with capabilities: [[gpu]]", you're on the wrong AMI.

IAM Role for SSM Access

Attach a role with  AmazonSSMManagedInstanceCore  so you can connect via Session Manager without SSH keys:
# Create role (one-time)
aws iam create-role --role-name SSMAccessRole \
--assume-role-policy-document '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"Service":"ec2.amazonaws.com"},"Action":"sts:AssumeRole"}]}'

aws iam attach-role-policy --role-name SSMAccessRole \
--policy-arn arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore

aws iam create-instance-profile --instance-profile-name SSMAccessRole
aws iam add-role-to-instance-profile --instance-profile-name SSMAccessRole --role-name SSMAccessRole

Step 2: Deploy vLLM

Once your instance is running, connect via SSM and start the model:
aws ssm start-session --target <INSTANCE_ID> --region us-east-1
Then run:
mkdir -p /opt/hf-cache /opt/vllm-cache

sudo docker run -d --gpus all --ipc=host --init -p 8000:8000 \
-v /opt/hf-cache:/root/.cache/huggingface \
-v /opt/vllm-cache:/root/.cache/vllm \
vllm/vllm-openai:latest \
--model openai/gpt-oss-120b \
--tensor-parallel-size 4 \
--max-model-len 16384 \
--max-num-seqs 16 \
--gpu-memory-utilization 0.95 \
--enable-prefix-caching \
--enable-chunked-prefill \
--host 0.0.0.0 --port 8000 \
--served-model-name gpt-oss-120b
First run takes ~15-20 minutes (image pull + model download + weight loading + CUDA graph warmup). Subsequent restarts with cached weights take ~3-5 minutes.
Running GPT-OSS-20B instead? Swap the model and reduce tensor parallelism — it fits on a single GPU:
sudo docker run -d --gpus all --ipc=host --init -p 8000:8000 \
-v /opt/hf-cache:/root/.cache/huggingface \
vllm/vllm-openai:latest \
--model openai/gpt-oss-20b \
--tensor-parallel-size 1 \
--max-model-len 32768 \
--enable-prefix-caching \
--host 0.0.0.0 --port 8000 \
--served-model-name gpt-oss-20b
Watch progress:
sudo docker logs $(sudo docker ps -q) -f
Important notes:
  •  --ipc=host  is required for vLLM v0.27.x — without it, you get  RuntimeError: Insufficient space in /dev/shm 
  •  --enable-prefix-caching  is the single highest-impact setting for workloads with repeated system prompts
  • Pin to a specific vLLM version ( v0.27.1 ) rather than  latest  for reproducibility
  • If your instance has NVMe ephemeral storage (P6 instances do), mount the HF cache there for faster model loading:  -v /opt/dlami/nvme/hf-cache:/root/.cache/huggingface 
  • Adjust  --tensor-parallel-size  based on your instance: TP=4 for 4 GPUs, TP=8 for 8 GPUs
  •  --max-model-len 16384  caps context to 16k tokens for this example. The model natively supports up to 128k — increase to  131072  for production workloads if VRAM and KV cache budget allow.

Step 3: Invoke the Model

Once the health check passes ( curl http://<PUBLIC_IP>:8000/health ), the endpoint is fully OpenAI SDK-compatible. No API key is needed — access is controlled at the network layer via security group.

Using the OpenAI Python SDK

pip install openai
from openai import OpenAI

client = OpenAI(
base_url="http://<PUBLIC_IP>:8000/v1",
api_key="not-needed", # vLLM doesn't validate, but SDK requires the field
)

# Responses API
response = client.responses.create(
model="gpt-oss-120b",
input=[
{"role": "system", "content": "You are a helpful assistant. Be concise."},
{"role": "user", "content": "Explain TCP in one sentence."},
],
max_output_tokens=100,
temperature=0,
reasoning={"effort": "low"},
)
print(response.output_text)

Chat Completions API

response = client.chat.completions.create(
model="gpt-oss-120b",
messages=[
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "What is the capital of France?"},
],
max_tokens=50,
temperature=0,
)
print(response.choices[0].message.content)

Structured JSON Output

response = client.chat.completions.create(
model="gpt-oss-120b",
messages=[
{"role": "system", "content": "Extract entities from the user message."},
{"role": "user", "content": "John Smith ordered 3 widgets for $29.99"},
],
max_tokens=200,
temperature=0,
response_format={
"type": "json_schema",
"json_schema": {
"name": "entities",
"strict": True,
"schema": {
"type": "object",
"properties": {
"name": {"type": "string"},
"quantity": {"type": "integer"},
"amount": {"type": "string"},
},
"required": ["name", "quantity", "amount"],
"additionalProperties": False,
},
},
},
)
print(response.choices[0].message.content)

Streaming

stream = client.chat.completions.create(
model="gpt-oss-120b",
messages=[{"role": "user", "content": "Tell me a story"}],
max_tokens=200,
stream=True,
)
for chunk in stream:
if chunk.choices[0].delta.content:
print(chunk.choices[0].delta.content, end="")

curl (for quick testing)

curl http://<PUBLIC_IP>:8000/v1/responses \
-H "Content-Type: application/json" \
-d '{
"model": "gpt-oss-120b",
"input": "What is the capital of France?",
"max_output_tokens": 50,
"temperature": 0
}'

Summary

That's it — from zero to running GPT-OSS-120B (or 20B) inference on EC2 in three steps. The OpenAI SDK works out of the box with zero code changes beyond pointing  base_url  at your instance. Structured JSON output, streaming, the Responses API, and Chat Completions are all supported natively through vLLM's OpenAI-compatible server.
For production use, consider adding a load balancer, autoscaling group, or migrating to a managed service like Amazon Bedrock or SageMaker. But for benchmarking, PoCs, or dedicated single-model inference — this is the fastest path to frontier-class performance on your own hardware.

Resources

Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article