
Building an AI Gateway on ECS Fargate with per-application budgets
Deploy an AI gateway on Amazon ECS with AWS Fargate and give two applications separate access budgets. Using live Amazon Bedrock requests, verify that one app is denied after exhausting its budget while the other keeps working.
Series: AI-kido (1 article)
- 1Building an AI Gateway on ECS Fargate with per-application budgets This article
App A has spent its one-cent model budget. Its next request to Amazon Bedrock should be denied. App B uses the same model through the same endpoint, and its request should still succeed. A successful chat completion tells you very little about whether those two rules hold.
I built this gateway to test both rules on a real deployment. The applications use separate access, and the gateway tracks spend against each key. The tricky part is timing. A completion can put a key over its budget, and recorded spend can lag behind the requests that incurred it. The test needs to wait for that spend to appear, recognize a budget-specific denial, and then show that the second key still works.
Direct Bedrock access is a reasonable starting point when applications already manage their own AWS identities and usage policies. For this build, I wanted an OpenAI-compatible API and one place to assign model access and budgets to each application. AI Gateway runs on Amazon ECS with AWS Fargate, stores key and spend records in PostgreSQL, and calls Amazon Nova Micro through Bedrock Converse.
In the recorded run, App A was denied after AI Gateway reported $0.010573 in spend against its $0.01 budget. App B still completed a request. This walkthrough shows how to deploy the same gateway, reproduce the budget test, exercise retries independently of AWS, and remove the demo resources.
What happens to a request
Both demo clients connect to
127.0.0.1:4000 through an AWS Systems Manager port-forwarding session. They send the model alias nova-micro and their own bearer keys to /v1/chat/completions . AI Gateway maps the alias to bedrock/converse/amazon.nova-micro-v1:0 . The ECS task role is scoped to the Nova Micro foundation model for Bedrock invocation, so the clients never need the task's AWS credentials.
model_list: - model_name: nova-micro litellm_params: model: bedrock/converse/amazon.nova-micro-v1:0 aws_region_name: os.environ/BEDROCK_REGION
general_settings: master_key: os.environ/LITELLM_MASTER_KEY
litellm_settings: turn_off_message_logging: true
task.addToTaskRolePolicy(new iam.PolicyStatement({ actions: ['bedrock:InvokeModel', 'bedrock:InvokeModelWithResponseStream'], resources: [this.formatArn({ service: 'bedrock', account: '', resource: 'foundation-model', resourceName: 'amazon.nova-micro-v1:0', arnFormat: ArnFormat.SLASH_RESOURCE_NAME, })],}));
Fargate gives the long-running proxy and Admin UI a place to run. The task uses a public subnet and a public IPv4 address for outbound HTTPS, with no inbound security-group rule and no NAT gateway. The proxy binds to loopback. Aurora runs in isolated subnets with encryption at rest, required TLS, and PostgreSQL access from the task security group. The image pins LiteLLM version by digest and includes the RDS CA bundle. ECS injects the database credentials and master-key suffix from Secrets Manager. Session access and the master key are operator privileges. Each application gets only its virtual key.
1. Deploy the gateway
Use a disposable AWS environment in
us-east-1 . You'll need Node.js 24 or later, pnpm 12.18.0 , Python 3, Make, Git, AWS CLI v2, Docker with linux/amd64 support, and the Session Manager plugin. Your AWS profile needs CDK deployment permissions and, for verification, access to CloudFormation outputs, ECS task discovery, SSM sessions, and the demo's master-key secret. Check that your account can invoke Nova Micro in the region.
git clone https://github.com/dobeerman/aws-cb-race-3.gitcd aws-cb-race-3
export AWS_PROFILE=your-demo-profileaws sts get-caller-identity
make installmake checkmake synthmake bootstrap # Run once if this account and region aren't bootstrapped.make diffmake deploy
Review the diff before deploying. The stack outputs include the ECS cluster and service names, the master-key secret ARN, and the CloudWatch log group name.
2. Open the tunnel and make a real Bedrock call
In terminal A, open the tunnel and leave it running.
export AWS_PROFILE=your-demo-profilemake tunnel
The script finds the running task and uses the gateway container's runtime ID to start an SSM port-forwarding session. If ECS replaces the task, reopen the tunnel.
In terminal B, check readiness and run the smoke test.
export AWS_PROFILE=your-demo-profilecurl --noproxy '*' -i http://127.0.0.1:4000/health/readinessmake smoke
There's an easy failure to diagnose here. If the smoke test says it can't reach
/health/readiness , check the tunnel first. Deployment leaves that step to the operator. An HTTP 200 from readiness shows that the proxy is reachable. The smoke test goes further. It checks that anonymous model listing is rejected, creates a temporary nova-micro key with a $0.01 budget, reads back its policy, requires a nonempty Bedrock completion, and deletes the key.
3. Exhaust App A's budget and send a request from App B
With the tunnel still running, run the live budget test.
make test-budgets
The script creates two keys for
nova-micro , each with a 30-minute lifetime. It sends synthetic inputs through App A's key and uses returned token counts to estimate when to stop generating load. Then it polls AI Gateway's recorded spend and sends a small probe that must return a budget-specific error. An HTTP 429 by itself would be ambiguous because rate limiting can produce the same status. Finally, the script checks that App B still has zero spend before its first request, requires a successful completion, and waits for App B's own charge to appear. It attempts to delete both keys even if a check fails.
Budget target · Budget-error detection · Request limits and spend assertions · Key creation and cleanup
Application | Configured budget | Recorded spend shown by the run | Observed outcome |
|---|---|---|---|
App A | $0.01 | $0.010573 | Budget-specific denial after exhaustion |
App B | $0.10 | $0.000001 | Completion succeeded |
App A's recorded spend was 5.73% above its configured threshold at the point reported by the test. That is one observed run, with values taken from the gateway's spend records. The useful distinction is between denying further requests after recorded spend crosses a threshold and reserving the maximum possible cost before each request. This deployment demonstrated the former. The workload has a client-side request limit and an eight-token output limit per request. Those limits and the virtual-key budget cannot cap the AWS bill.

4. Test retries where you control the failures
It's difficult to reproduce precisely two throttles followed by a successful Bedrock response. The reliability suite uses the same pinned AI Gateway image and a local mock provider inside a Docker container with external networking disabled. It asserts the order of upstream requests, the client HTTP status, and the response marker. You can run it without AWS credentials, a database, or a tunnel.
make test-reliability
All four cases passed in the recorded run.
Case | Primary responses | Backup responses | Client result |
|---|---|---|---|
Baseline | 200 | — | 200 from primary |
Retry | 429, 429, 200 | — | 200 from primary |
Fallback | 503, 503, 503 | 200 | 200 from backup |
Exhaustion | 503, 503, 503 | 503, 503, 503 | 503 |
The backup route exists in the test configuration. The deployed gateway exposes one Bedrock model alias. The suite demonstrates how the pinned proxy handles the configured failures, with live Bedrock behavior covered separately by the smoke and budget tests.

To repeat the complete check, keep the tunnel open and run
make verify from terminal B. The recorded clean run passed stack destruction, redeployment, and verification, including three infrastructure tests, five unit tests, the smoke test, both budget assertions, and all four local reliability scenarios.
What this design costs and where it stops
The database keeps key and spend state outside the Fargate task. That state comes with a standing cost. The repository estimates about $69 for 730 hours in
us-east-1 , assuming one 0.5 vCPU / 2 GiB Fargate task, one public IPv4 address, and Aurora at its configured 0.5 ACU floor. This estimate excludes storage, I/O, logs, secrets, image storage, and inference. App A and App B can stop making requests while those infrastructure charges continue.
One task and one writer keep the demo small. The service can briefly go offline during an update. The proxy uses the generated database user for both runtime access and schema initialization. SSM is an operator access path, so a shared application service would need its own ingress and authorization design. The two-key test establishes independent spend accounting and enforcement within one deployment. Customer data isolation, recovery, and high availability would each need their own design and verification.
5. Clean up
Stop the tunnel in terminal A. In terminal B, confirm your AWS identity and destroy the demo stack.
aws sts get-caller-identitymake destroy
Confirm that
SpeedRunAiGateway reaches DELETE_COMPLETE in CloudFormation. The demo deletes the database and its key, budget, and spend history without a final snapshot. The shared CDKToolkit bootstrap stack and its ECR assets may remain. Check for other consumers before removing those shared resources.
Conclusion
The result I wanted was an ordinary failure that affected only one application. App A crossed its budget and received a denial. App B went through the same proxy to the same model and succeeded. The useful test was the request I sent after the failure.
The same API boundary gives the design room to grow. A provider such as OpenAI or Anthropic could be added behind the AI Gateway, with its credentials and routing managed at the gateway. If the endpoint, bearer-key authentication, model alias, and API behavior stay compatible, the applications can keep their existing code and access pattern. The live deployment verified Amazon Bedrock. Any additional provider route would need its own checks for request compatibility, spend accounting, and failure behavior.
Series: AI-kido (1 article)
- 1Building an AI Gateway on ECS Fargate with per-application budgets This article
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article