Deploying OpenAI GPT-OSS Models on Amazon EC2 GPU Instances with vLLM
How to self-host OpenAI's open-source GPT-OSS-120B on EC2 P6 instances with vLLM with prefix caching, structured output, and OpenAI SDK compatibility.
Why Self-Host on EC2?
OpenAI released two open-weight models — GPT-OSS-120B and GPT-OSS-20B — both using MXFP4 quantization on their MoE weights, enabling the 120B to fit on a single 80 GB GPU (NVIDIA H100, AMD MI300X, or Blackwell) and the 20B to run within just 16 GB of memory. This guide focuses on the 120B for frontier-class inference, but the same setup applies to the 20B with smaller hardware requirements. Blackwell has native MXFP4 Tensor Core support which is expected to give better throughput than Hopper-class hardware, but OpenAI officially targets H100 and MI300X as primary deployment platforms — both work well. This means you can run frontier-class inference on your own infrastructure with full control over configuration, latency, and cost.
Three reasons to consider raw EC2 over managed inference for latency-sensitive workloads:
- No routing overhead — direct Docker container, no load balancer hops or managed service queuing
- Full config control — tensor parallelism, prefix caching, chunked prefill, KV cache tuning
- Predictable cost — flat hourly rate via Capacity Blocks, no per-token pricing surprises at scale
The trade-off is clear: you own the ops. No autoscaling, no multi-model routing, no managed failover. For single-model latency benchmarks or dedicated inference for a specific workload, this is the right tool.
Choosing a GPU Instance
EC2 offers several GPU instance families suitable for LLM inference. The right choice depends on model size, latency requirements, and budget:
Instance Family | GPUs | GPU Memory | Best For |
|---|---|---|---|
P6 | 8× B200 or B300 | 1.4-1.5 TB HBM3e | Frontier models (100B+), lowest latency, native MXFP4 support |
P5en / P5e | 8× H200 | 1.1 TB HBM3e | Large models (70B+), strong price-performance |
P5 | 8× H100 | 640 GB HBM3 | 70B models, widely available |
G6e | up to 8× L40S | up to 384 GB | Mid-size models (7B-30B), cost-efficient for development |
G6 | up to 8× L4 | up to 192 GB | Smaller models (7B-13B), inference at scale |
For GPT-OSS-120B specifically, you need P5 or higher — the model requires at least ~90 GB of GPU memory across your tensor parallel group. On P6 with MXFP4 quantization, the model fits comfortably across 4 GPUs with headroom for KV cache — use
--tensor-parallel-size 4 for a cost-efficient configuration, or --tensor-parallel-size 8 to maximise throughput.Key consideration: MXFP4 quantization (used by GPT-OSS-120B) is optimized for Blackwell's native Tensor Core support. Running on Hopper (P5/P5e) is possible but uses different quantization kernels with potentially different performance characteristics.
Architecture

- Instance: Any NVIDIA GPU instance (P6, P5e, P5, G6e) with sufficient VRAM
- Serving: vLLM with OpenAI-compatible API (Chat Completions + Responses API)
- Access: IP-locked security group, SSM Session Manager for shell access
- Deployment: Launch instance → run Docker container → healthy in ~20 minutes
Prerequisites
- AWS account with GPU instance quota (P6, P5, or G6e depending on model size)
- AWS CLI configured
- A public subnet with internet access (for model download from HuggingFace)
- Security group with port 8000 open to your IP
Step 1: Get a GPU Instance
There are several ways to procure GPU capacity on EC2:
Option 1: On-Demand
The simplest path — launch an instance directly if you have quota. No commitment, pay by the second, but availability isn't guaranteed for GPU instances.
aws ec2 run-instances \ --instance-type p6-b200.48xlarge \ --image-id ami-0e2e1c9b9d71cc77f \ --subnet-id subnet-xxxxx \ --security-group-ids sg-xxxxx \ --iam-instance-profile Name=SSMAccessRole \ --block-device-mappings '[{"DeviceName":"/dev/sda1","Ebs":{"VolumeSize":500,"VolumeType":"gp3"}}]' \ --region us-east-1Option 2: Capacity Blocks
Reserve GPU capacity for a defined window (hours to weeks) at a fixed hourly rate. Best for planned benchmarks or time-bound workloads where you need guaranteed availability.
# Find available Capacity Blocksaws ec2 describe-capacity-block-offerings \ --instance-type p6-b200.48xlarge \ --capacity-duration-hours 24 \ --region us-east-1
# Purchase a blockaws ec2 purchase-capacity-block \ --capacity-block-offering-id cbr-xxxxx \ --instance-platform Linux/UNIX \ --region us-east-1Then launch into the reservation:
aws ec2 run-instances \ --instance-type p6-b200.48xlarge \ --capacity-reservation-specification \ "CapacityReservationTarget={CapacityReservationId=cr-xxxxx}" \ --image-id ami-0e2e1c9b9d71cc77f \ --subnet-id subnet-xxxxx \ --region us-east-1Option 3: On-Demand Capacity Reservations (ODCRs)
For longer-term dedicated capacity without Savings Plan commitment. Useful when you need guaranteed availability but want flexibility to start/stop.
Option 4: Spot Instances
Up to 90% discount but can be interrupted. Suitable for non-production benchmarking and experimentation — not recommended for customer-facing workloads.
aws ec2 request-spot-instances \ --instance-count 1 \ --type one-time \ --launch-specification '{ "ImageId": "ami-0e2e1c9b9d71cc77f", "InstanceType": "p6-b200.48xlarge", "SubnetId": "subnet-xxxxx" }' \ --region us-east-1AMI Selection
Use the latest Deep Learning OSS NVIDIA Driver AMI — it comes with Docker, NVIDIA drivers, and the container toolkit pre-installed:
# Resolves to the latest NVIDIA GPU AMI in any regionaws ssm get-parameter \ --name "/aws/service/deeplearning/ami/x86_64/oss-nvidia-driver-gpu-pytorch-2.12-ubuntu-24.04/latest/ami-id" \ --region us-east-1 --query "Parameter.Value" --output text⚠️ Do not use Neuron AMIs (designed for Trainium/Inferentia) — they lack NVIDIA drivers and the container toolkit. If you see "could not select device driver with capabilities: [[gpu]]", you're on the wrong AMI.
IAM Role for SSM Access
Attach a role with
AmazonSSMManagedInstanceCore so you can connect via Session Manager without SSH keys:# Create role (one-time)aws iam create-role --role-name SSMAccessRole \ --assume-role-policy-document '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"Service":"ec2.amazonaws.com"},"Action":"sts:AssumeRole"}]}'
aws iam attach-role-policy --role-name SSMAccessRole \ --policy-arn arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore
aws iam create-instance-profile --instance-profile-name SSMAccessRoleaws iam add-role-to-instance-profile --instance-profile-name SSMAccessRole --role-name SSMAccessRoleStep 2: Deploy vLLM
Once your instance is running, connect via SSM and start the model:
aws ssm start-session --target <INSTANCE_ID> --region us-east-1Then run:
mkdir -p /opt/hf-cache /opt/vllm-cache
sudo docker run -d --gpus all --ipc=host --init -p 8000:8000 \ -v /opt/hf-cache:/root/.cache/huggingface \ -v /opt/vllm-cache:/root/.cache/vllm \ vllm/vllm-openai:latest \ --model openai/gpt-oss-120b \ --tensor-parallel-size 4 \ --max-model-len 16384 \ --max-num-seqs 16 \ --gpu-memory-utilization 0.95 \ --enable-prefix-caching \ --enable-chunked-prefill \ --host 0.0.0.0 --port 8000 \ --served-model-name gpt-oss-120bFirst run takes ~15-20 minutes (image pull + model download + weight loading + CUDA graph warmup). Subsequent restarts with cached weights take ~3-5 minutes.
Running GPT-OSS-20B instead? Swap the model and reduce tensor parallelism — it fits on a single GPU:
sudo docker run -d --gpus all --ipc=host --init -p 8000:8000 \ -v /opt/hf-cache:/root/.cache/huggingface \ vllm/vllm-openai:latest \ --model openai/gpt-oss-20b \ --tensor-parallel-size 1 \ --max-model-len 32768 \ --enable-prefix-caching \ --host 0.0.0.0 --port 8000 \ --served-model-name gpt-oss-20bWatch progress:
sudo docker logs $(sudo docker ps -q) -fImportant notes:
-
--ipc=hostis required for vLLM v0.27.x — without it, you getRuntimeError: Insufficient space in /dev/shm
-
--enable-prefix-cachingis the single highest-impact setting for workloads with repeated system prompts
- Pin to a specific vLLM version (
v0.27.1) rather thanlatestfor reproducibility
- If your instance has NVMe ephemeral storage (P6 instances do), mount the HF cache there for faster model loading:
-v /opt/dlami/nvme/hf-cache:/root/.cache/huggingface
- Adjust
--tensor-parallel-sizebased on your instance: TP=4 for 4 GPUs, TP=8 for 8 GPUs
-
--max-model-len 16384caps context to 16k tokens for this example. The model natively supports up to 128k — increase to131072for production workloads if VRAM and KV cache budget allow.
Step 3: Invoke the Model
Once the health check passes (
curl http://<PUBLIC_IP>:8000/health ), the endpoint is fully OpenAI SDK-compatible. No API key is needed — access is controlled at the network layer via security group.Using the OpenAI Python SDK
pip install openaifrom openai import OpenAI
client = OpenAI( base_url="http://<PUBLIC_IP>:8000/v1", api_key="not-needed", # vLLM doesn't validate, but SDK requires the field)
# Responses APIresponse = client.responses.create( model="gpt-oss-120b", input=[ {"role": "system", "content": "You are a helpful assistant. Be concise."}, {"role": "user", "content": "Explain TCP in one sentence."}, ], max_output_tokens=100, temperature=0, reasoning={"effort": "low"},)print(response.output_text)Chat Completions API
response = client.chat.completions.create( model="gpt-oss-120b", messages=[ {"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "What is the capital of France?"}, ], max_tokens=50, temperature=0,)print(response.choices[0].message.content)Structured JSON Output
response = client.chat.completions.create( model="gpt-oss-120b", messages=[ {"role": "system", "content": "Extract entities from the user message."}, {"role": "user", "content": "John Smith ordered 3 widgets for $29.99"}, ], max_tokens=200, temperature=0, response_format={ "type": "json_schema", "json_schema": { "name": "entities", "strict": True, "schema": { "type": "object", "properties": { "name": {"type": "string"}, "quantity": {"type": "integer"}, "amount": {"type": "string"}, }, "required": ["name", "quantity", "amount"], "additionalProperties": False, }, }, },)print(response.choices[0].message.content)Streaming
stream = client.chat.completions.create( model="gpt-oss-120b", messages=[{"role": "user", "content": "Tell me a story"}], max_tokens=200, stream=True,)for chunk in stream: if chunk.choices[0].delta.content: print(chunk.choices[0].delta.content, end="")curl (for quick testing)
curl http://<PUBLIC_IP>:8000/v1/responses \ -H "Content-Type: application/json" \ -d '{ "model": "gpt-oss-120b", "input": "What is the capital of France?", "max_output_tokens": 50, "temperature": 0 }'Summary
That's it — from zero to running GPT-OSS-120B (or 20B) inference on EC2 in three steps. The OpenAI SDK works out of the box with zero code changes beyond pointing
base_url at your instance. Structured JSON output, streaming, the Responses API, and Chat Completions are all supported natively through vLLM's OpenAI-compatible server.For production use, consider adding a load balancer, autoscaling group, or migrating to a managed service like Amazon Bedrock or SageMaker. But for benchmarking, PoCs, or dedicated single-model inference — this is the fastest path to frontier-class performance on your own hardware.
Resources
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article