Build a Secure, Cost-Effective AI Sandbox on AWS EC2 GPU
Deploy a self-hosted LLM playground with Qwen, vLLM, and LiteLLM in Docker on an EC2 g5 instance — cut token bills and keep sensitive data inside your VPC.
The Problem
Running research experiments or building proof-of-concepts with agentic systems is exciting! The non-exciting part is that pulling API keys for proprietary models during these experiments quickly introduces two major roadblocks: spiraling token costs and data privacy concerns. Sending sensitive internal data or proprietary agent prompts to external APIs is often a critical concern in enterprise environments.
A Solution to this problem?
Among other considerations to solve this problem, this article provides a production-ready and cost-effective LLM playground. We will walk through the deployment of an open-source Qwen model using vLLM for high-throughput inference, containerised with Docker, hosted on an AWS EC2 GPU instance, and secured using a LiteLLM proxy. In this setup, LiteLLM serves as the secure gateway to track token usage and manage internal user access.
Architecture Overview
- Qwen: We specifically chose this open-source model family because it exhibits top-tier performance for coding, reasoning, and tool use (all of which are essential for agentic workflows). For this setup, we will use Qwen3-30B-A3B-Instruct-2507
- vLLM: This engine handles fast and optimized inference. Optimised LLM serving engines like this are important because they reduce memory waste, allowing for high-throughput, concurrent requests. We will serve the model with AWQ 4-bit quantization to fit comfortably within the GPU's VRAM.
- LiteLLM Proxy: This is a dedicated open-source gateway for LLMs. While we could skip it in this setup and expose the model directly with vLLM, doing so removes a very important security and observability layer. We could also use an NGINX proxy with static key validation logic, but LiteLLM works best for this use case because it provides LLM-specific security (access management, rate limiting, etc.) and observability (token usage, logging, etc.) right out of the box.
- Docker & AWS EC2: This pair provides the GPU computing power and the deployment environment for our setup.
Provisioning The Computing Power (AWS EC2 GPU Instance)
For running a quantised or smaller state-of-the-art Qwen model like Qwen3-30B-A3B-Instruct-2507 at 4-bit quantization (AWQ), an NVIDIA A10G or a G5 instance family does the job. Here are the important things to note:
- To get pre-installed CUDA drivers with the GPU instance, use the Deep Learning OSS Nvidia Driver AMI (Ubuntu 22.04 or later). It reduces a lot of installation headaches.
- I recommend using the
g5.xlargeinstance type, which features 1x NVIDIA A10G GPU with 24GB of VRAM. - For storage, allocate at least 100GB of gp3 EBS storage, as LLM weights are large and the inference KV cache requires sufficient space.
- Ensure Docker is configured to use the machine's GPU, because the model will be running inside a container. You will run these configuration commands once you SSH into the instance.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
# Verify GPU availability
nvidia-smi
# Install Docker Setup (just incase its not pre-installed)
sudo apt-get update
sudo apt-get install -y docker.io docker-compose
#Install NVIDIA Container Toolkit
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
&& curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list |
sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' |
sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt-get update
sudo apt-get install -y nvidia-container-toolkit
# Restart Docker to apply changes
sudo systemctl restart docker Configuring the LLM Gateway
LiteLLM is straightforward to deploy; you can check out their official documentation for more information. It relies on a simple YAML file to discover backends and enforce security constraints for the setup. You can create a
config.yaml file that looks like this:1
2
3
4
5
6
7
8
9
model_list:
- model_name: Qwen3-30B-A3B-Instruct-2507
litellm_params:
model: openai/Qwen/Qwen3-30B-A3B-Instruct-2507
api_base: http://vllm:8000/v1
api_key: "not-needed-for-internal-docker"
general_settings:
master_key: sk_master_secret_key_1234 # this is to protect admin routes for the gatewayThe Docker Compose File
To bring the setup together, we will deploy all components using a Docker Compose file. The
docker-compose.yaml file includes a database backend because LiteLLM requires one to manage unique user keys and store observability data.1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
version: '3.8'
services:
vllm:
image: vllm/vllm-openai:latest
container_name: vllm-qwen-engine
volumes:
- ~/.cache/huggingface:/root/.cache/huggingface
ipc: host
deploy:
resources:
reservations:
devices:
- driver: nvidia
count: 1
capabilities: [gpu]
command:
- "--model"
- "Qwen/Qwen3-30B-A3B-Instruct-2507"
- "--max-model-len"
- "8192"
- "--dtype"
- "bfloat16"
- "--quantization awq_marlin"
- "--enable-auto-tool-choice"
- "--tool-call-parser"
- "hermes"
litellm-db:
image: postgres:15-alpine
container_name: litellm-db
environment:
POSTGRES_DB: litellm
POSTGRES_USER: litellm_admin
POSTGRES_PASSWORD: secure_db_password
volumes:
- litellm_data:/var/lib/postgresql/data
litellm:
image: ghcr.io/berriai/litellm:main-latest
container_name: litellm-proxy
volumes:
- ./litellm_config.yaml:/app/config.yaml
ports:
- "4000:4000" # The gateway port your engineers will connect to
environment:
- DATABASE_URL=postgresql://litellm_admin:secure_db_password@litellm-db:5432/litellm
depends_on:
- litellm-db
- vllm
command: ["--config", "/app/config.yaml", "--port", "4000", "--detailed_debug"]
volumes:
litellm_data:To spin up the stack, you can run docker compose command:
1
docker-compose up -dManaging LiteLLM Keys and Observability
If all containers are in a healthy state, you can access the LiteLLM dashboard locally on port 4000. To understand its features extensively, I recommend checking out the docs. For this article, we will add a unique user and use the returned key to call our deployed Qwen model using the standard OpenAI interface.
Adding a User and Removing the a User:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
# Adding a new using to access the Qwen3-30B-A3B-Instruct-2507 model
curl -X POST http://localhost:4000/key/generate \
-H "Authorization: Bearer sk_master_secret_key_1234" \
-H "Content-Type: application/json" \
-d '{
"user_id": "femi@foobar.ai",
"duration": "30d",
"models": ["Qwen3-30B-A3B-Instruct-2507"],
"tpm_limit": 10000
}'
# LiteLLM will return a unique key e.g: sk_1234_femi_token_secret
# Deleting the key
curl -X POST http://localhost:4000/key/delete \
-H "Authorization: Bearer sk_master_secret_key_1234" \
-H "Content-Type: application/json" \
-d '{
"key": "sk_1234_femi_token_secret"
}'
Calling the LLM Through the LiteLLM Gateway and Unique
1
2
3
4
5
6
7
8
9
10
11
12
from openai import OpenAI
client = OpenAI(
api_key="sk_1234_femi_token_secret", # unique LiteLLM key
base_url="http://<YOUR_EC2_INTERNAL_IP>:4000" # LiteLLM proxy port
)
response = client.chat.completions.create(
model="Qwen3-30B-A3B-Instruct-2507",
messages=[{"role": "user", "content": "Run an agent loop check."}]
)
print(response.choices[0].message.content)All Set, Ready for Safe and Observable AI Experimentation 🚀
By combining Qwen, vLLM, and LiteLLM, we have engineered a modern, enterprise-ready AI sandbox for your home lab or internal engineering team. The LiteLLM setup can also be extended to work with external LLM providers, including OpenAI, Anthropic, and even AWS Bedrock.
With this stack, you can spin up high-frequency experimental scripts safely within your network boundaries without driving up corporate SaaS subscription bills. Administration retains total oversight over spend, rate limits, and the active developer lifecycle.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article