Architecting a Secure Low Cost Serverless LiteLLM Gateway with AWS Lambda MicroVMs and AWS CDK
Learn how to leverage the AWS CDK to deploy an auto-scaling, serverless LiteLLM inside AWS Lambda MicroVMs that centralizes governance, budget control, and security perimeters across all your LLM workloads.
Series: AWS OpenClaw (4 articles)
- 2Architecting a Secure Low Cost Serverless LiteLLM Gateway with AWS Lambda MicroVMs and AWS CDK This article
As generative AI initiatives transition from proof-of-concept to production, enterprise platform teams face a critical challenge: standardizing access to diverse Large Language Models (LLMs). Application teams often cross-route requests between different foundational models and backends to optimize for cost, performance, and specific capabilities. However, managing raw API keys, enforcing granular budgets, and securing egress traffic for dozens of internal applications quickly becomes an operational and security nightmare.
To address this, platform teams are adopting the LLM Gateway pattern. A gateway provides a single, secured perimeter where organization-wide concerns—such as authentication, rate limiting, logging, and cost governance—can be centralized.
In this post, we deep dive into a robust, serverless implementation of this pattern. We’ll examine how to build a highly available LLM gateway using the open-source LiteLLM router packaged within AWS Lambda MicroVMs, backed by Amazon Aurora Serverless v2, and defined entirely with infrastructure as code (IaC) via the AWS Cloud Development Kit (AWS CDK).
The Serverless LLM Gateway Architecture
Building this infrastructure manually would require significant effort, but the PrivateLiteLlmMicrovmStack deploys the entire ecosystem deterministically. The architecture is designed for defense-in-depth security and zero-maintenance scalability.
Ingress & Compute Topology
- API Perimeter: Inbound requests first hit Amazon API Gateway, which establishes the public access control and usage plan boundary.
- Ephemeral Compute Layer: API Gateway triggers an internal Lambda auth proxy function via an Amazon VPC Endpoint. This proxy coordinates traffic to the main application compute layer: LiteLLM running inside AWS Lambda MicroVM containers. The MicroVM runtime provides isolated execution suitable for hosting third-party LLM libraries.
- State & Data Management: To maintain state (user keys, budget tracking, routing logs), the Lambda MicroVMs connect to Amazon Aurora Serverless v2, which scales perfectly with zero-idle cost. Sensitive parameters, such as the initial LITELLM_MASTER_KEY, are securely fetched from AWS Secrets Manager, while rapid runtime coordination uses Amazon DynamoDB.
Balancing Costs with Flexible Provider Egress
A core architectural capability of this stack is its toggleable networking model, controlled via the publicMicrovm flag in cdk-settings.yaml. This setting directly handles the trade-offs between fixed infrastructure baselines and multi-cloud egress requirements:
| Dimension | publicMicrovm=true (Default Mode) | publicMicrovm=false (Private Mode) | Security / Cost Impact |
|---|---|---|---|
| MicroVM Subnet | AppPublic Subnet | AppPrivate (PRIVATE_WITH_EGRESS) | Private mode minimizes direct network exposure surface for connector ENIs. |
| NAT Gateways | None (natGateways: 0) | One NAT (natGateways: 1) | Public mode maintains the lowest fixed baseline cost; private mode introduces fixed hourly NAT infrastructure fees. |
| Public Internet Path | No direct NAT route for VPC-attached Lambda ENIs. | Yes (Private Subnet ➔ NAT Gateway ➔ Internet). | publicMicrovm=true is strictly optimized for AWS native paths (Bedrock); publicMicrovm=false is mandatory for outbound internet egress to external providers like Azure OpenAI or GCP Gemini. |
| AWS Private Access | Interface VPC Endpoints + S3 Gateway Endpoint. | Same. | Keeps high-value services (Bedrock, STS, KMS, Secrets, CloudWatch Logs) on private, internal VPC network paths in both modes. |
Enterprise-Grade Security: Two-Layer Authorization Architecture
Every standard client request must pass two independent, decoupled authorization boundaries before reaching any underlying model endpoint:
- Layer 1 (Perimeter Mitigation): Amazon API Gateway validates the client request via the x-api-key header. This token maps straight to an AWS API Gateway Usage Plan to enforce rate-limiting, burst throttling, and quotas at the cloud edge before compute execution begins.
- Layer 2 (Application Governance): The request passes into the Lambda MicroVM runtime where the secondary Authorization: Bearer <token> header is parsed. The LiteLLM application logic treats this as a "virtual key". It queries the Aurora Serverless v2 backend to evaluate fine-grained routing constraints, specific model allowlists, and enforce strict, database-driven cost budgets (e.g., capping a token at $10 daily).
Streamlining Internal Services with AWS IAM Auth (/iam/*)
While virtual bearer tokens are perfect for external applications or desktop endpoints like OpenClaw, forcing internal AWS microservices to handle static strings creates credential lifecycle overhead.
To solve this, the stack implements a zero-trust, tokenless machine-to-machine route over the /iam/* endpoint path.
Deep Dive: Lambda Proxy State Coordination
Because the Lambda MicroVM compute layer is highly ephemeral, state tracking across execution environments must happen programmatically outside of the runtime memory. The Lambda auth proxy intercepts requests and communicates with Amazon DynamoDB to coordinate live virtual machine IDs.
Resolving the "Stale Token" Bug
During active rotation or automatic recycling of a underlying Lambda MicroVM, early iterations of ephemeral proxies ran into an edge case where a cached authentication token was inadvertently reused against a newly created target VM id. This caused random, intermittent 403 Token authentication failed errors.
The resolution implemented in infra/cdk/lambda/microvm_proxy.py binds the cached auth token explicitly to a dedicated metadata field called token_microvm_id. At runtime, the proxy checks the active VM state; if the physical microvm_id has shifted, the proxy immediately flushes the DynamoDB cache item and forces a clean control-plane re-authentication.
Implementation & End-to-End Verification Blueprint
Deploying the stack and validating its two-layer authentication mechanism is straightforward.
During active rotation or automatic recycling of a underlying Lambda MicroVM, early iterations of ephemeral proxies ran into an edge case where a cached authentication token was inadvertently reused against a newly created target VM id. This caused random, intermittent 403 Token authentication failed errors.
The resolution implemented in infra/cdk/lambda/microvm_proxy.py binds the cached auth token explicitly to a dedicated metadata field called token_microvm_id. At runtime, the proxy checks the active VM state; if the physical microvm_id has shifted, the proxy immediately flushes the DynamoDB cache item and forces a clean control-plane re-authentication.
Implementation & End-to-End Verification Blueprint
Deploying the stack and validating its two-layer authentication mechanism is straightforward.
- Provisioning the Stack
Execute the programmatic wrapper script or run the AWS CDK construct directly to spin up the VPC, Aurora Serverless cluster, and API Gateway perimeters:
1
2
cd infra/cdk
./scripts/deploy-stack.sh --config cdk-settings.yaml --stack PrivateLiteLlmMicrovmStack- Programmatically Reconstructing System Endpoints
Once CloudFormation outputs are emitted, capture the API URLs and assemble the application's underlying cryptographic master keys from AWS Secrets Manager:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
STACK_NAME=PrivateLiteLlmMicrovmStack
AWS_REGION=us-east-1
# 1. Resolve public API endpoint
PUBLIC_API_URL=$(aws cloudformation describe-stacks \
--stack-name "$STACK_NAME" --region "$AWS_REGION" \
--query "Stacks[0].Outputs[?OutputKey=='PublicApiInvokeUrl'].OutputValue" --output text)
# 2. Extract divided master key parts from Secrets Manager
MASTER_KEY_SECRET_ARN=$(aws cloudformation describe-stacks \
--stack-name "$STACK_NAME" --region "$AWS_REGION" \
--query "Stacks[0].Outputs[?OutputKey=='LiteLlmMasterKeySecretArn'].OutputValue" --output text)
MASTER_JSON=$(aws secretsmanager get-secret-value \
--region "$AWS_REGION" --secret-id "$MASTER_KEY_SECRET_ARN" \
--query SecretString --output text)
# 3. Combine prefix and suffix programmatically to rebuild the admin string[cite: 8]
LITELLM_MASTER_KEY=$(python - <<'PY' "$MASTER_JSON"
import json,sys
v=json.loads(sys.argv[1])
print((v.get("prefix") or "") + (v.get("suffix") or ""))
PY
)
- Generating a Bound User Key
To configure a specific developer or client key, execute the automated provisioning script. This registers the identity in both the gateway edge and the application backend database simultaneously:
1
2
3
4
5
6
7
8
9
10
11
12
13
PUBLIC_PLAN_ID=$(aws cloudformation describe-stacks \
--stack-name "$STACK_NAME" --region "$AWS_REGION" \
--query "Stacks[0].Outputs[?OutputKey=='AwsGatewayUsagePlanId'].OutputValue" --output text)
# Provision an LLM-only token valid for 7 days, capped strictly to the nova-2-lite model[cite: 8, 11]
./scripts/create-api-key.sh \
--usage-plan-id "$PUBLIC_PLAN_ID" \
--alias engineer-app-key \
--duration 7d \
--models nova-2-lite \
--output-file .keys/engineer-key.txt
USER_KEY=$(cat .keys/engineer-key.txt)
- Executing an Authenticated Chat Request
Pass both the perimeter token (x-api-key) and the application bearer token (Authorization) to verify complete end-to-end routing:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
API_KEY_SECRET_ARN=$(aws cloudformation describe-stacks \
--stack-name "$STACK_NAME" --region "$AWS_REGION" \
--query "Stacks[0].Outputs[?OutputKey=='AwsGatewayApiKeySecretArn'].OutputValue" --output text)
API_GATEWAY_KEY=$(aws secretsmanager get-secret-value --region "$AWS_REGION" --secret-id "$API_KEY_SECRET_ARN" --query SecretString --output text | python -c 'import json,sys; print(json.load(sys.stdin)["apiKey"])')
curl -sS -X POST "${PUBLIC_API_URL%/}/chat/completions" \
-H "x-api-key: $API_GATEWAY_KEY" \
-H "Authorization: Bearer $USER_KEY" \
-H "Content-Type: application/json" \
--data-raw '{
"model": "nova-2-lite",
"messages": [{"role":"user","content":"Gateway verification complete."}]
}'
Production Operational Considerations & Pitfalls
- Database Cold Starts: When running completely serverless, sudden volumetric bursts can trigger minor latency spikes if the database tier is cold[cite: 8]. Ensure your Aurora Serverless v2 cluster maintains a minimum ACU configuration greater than 0.5 to keep connection pools active and eliminate 503 no_db_connection errors[cite: 8].
- MicroVM Image Stabilization: During deep infrastructure stack provisioning updates, the custom resource AWS::Lambda::MicrovmImage might occasionally report a NotStabilized state in your CloudFormation console. This is typically a transient state during container registration. The recommended remediation is to retry the deploy command independently of database or VPC adjustments.
Conclusion
By wrapping an open-source router like LiteLLM inside AWS Lambda MicroVMs, enterprise platform teams gain complete central control over costs, model constraints, and security without the heavy operational overhead of maintaining a permanent, non-serverless container fleet. With the AWS CDK orchestrating the entire lifecycle, deploying an enterprise-ready LLM gateway is entirely automated.
GitHub Repo: https://github.com/wongcyrus/litellm-aws-lambda-microvm-serverless
Series: AWS OpenClaw (4 articles)
- 2Architecting a Secure Low Cost Serverless LiteLLM Gateway with AWS Lambda MicroVMs and AWS CDK This article
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article