AWS Builder Center
Building an AI Gateway on ECS Fargate with per-application budgets

Building an AI Gateway on ECS Fargate with per-application budgets

Deploy an AI gateway on Amazon ECS with AWS Fargate and give two applications separate access budgets. Using live Amazon Bedrock requests, verify that one app is denied after exhausting its budget while the other keeps working.

Series: AI-kido (1 article)

  1. 1
    Building an AI Gateway on ECS Fargate with per-application budgets This article
App A has spent its one-cent model budget. Its next request to Amazon Bedrock should be denied. App B uses the same model through the same endpoint, and its request should still succeed. A successful chat completion tells you very little about whether those two rules hold.

I built this gateway to test both rules on a real deployment. The applications use separate access, and the gateway tracks spend against each key. The tricky part is timing. A completion can put a key over its budget, and recorded spend can lag behind the requests that incurred it. The test needs to wait for that spend to appear, recognize a budget-specific denial, and then show that the second key still works.

Direct Bedrock access is a reasonable starting point when applications already manage their own AWS identities and usage policies. For this build, I wanted an OpenAI-compatible API and one place to assign model access and budgets to each application. AI Gateway runs on Amazon ECS with AWS Fargate, stores key and spend records in PostgreSQL, and calls Amazon Nova Micro through Bedrock Converse.

In the recorded run, App A was denied after AI Gateway reported $0.010573 in spend against its $0.01 budget. App B still completed a request. This walkthrough shows how to deploy the same gateway, reproduce the budget test, exercise retries independently of AWS, and remove the demo resources.

What happens to a request

Both demo clients connect to  127.0.0.1:4000  through an AWS Systems Manager port-forwarding session. They send the model alias  nova-micro  and their own bearer keys to  /v1/chat/completions . AI Gateway maps the alias to  bedrock/converse/amazon.nova-micro-v1:0 . The ECS task role is scoped to the Nova Micro foundation model for Bedrock invocation, so the clients never need the task's AWS credentials.

model_list:
- model_name: nova-micro
litellm_params:
model: bedrock/converse/amazon.nova-micro-v1:0
aws_region_name: os.environ/BEDROCK_REGION

general_settings:
master_key: os.environ/LITELLM_MASTER_KEY

litellm_settings:
turn_off_message_logging: true

task.addToTaskRolePolicy(new iam.PolicyStatement({
actions: ['bedrock:InvokeModel', 'bedrock:InvokeModelWithResponseStream'],
resources: [this.formatArn({
service: 'bedrock',
account: '',
resource: 'foundation-model',
resourceName: 'amazon.nova-micro-v1:0',
arnFormat: ArnFormat.SLASH_RESOURCE_NAME,
})],
}));


Fargate gives the long-running proxy and Admin UI a place to run. The task uses a public subnet and a public IPv4 address for outbound HTTPS, with no inbound security-group rule and no NAT gateway. The proxy binds to loopback. Aurora runs in isolated subnets with encryption at rest, required TLS, and PostgreSQL access from the task security group. The image pins LiteLLM version by digest and includes the RDS CA bundle. ECS injects the database credentials and master-key suffix from Secrets Manager. Session access and the master key are operator privileges. Each application gets only its virtual key.


1. Deploy the gateway

Use a disposable AWS environment in  us-east-1 . You'll need Node.js 24 or later, pnpm  12.18.0 , Python 3, Make, Git, AWS CLI v2, Docker with  linux/amd64  support, and the Session Manager plugin. Your AWS profile needs CDK deployment permissions and, for verification, access to CloudFormation outputs, ECS task discovery, SSM sessions, and the demo's master-key secret. Check that your account can invoke Nova Micro in the region.


git clone https://github.com/dobeerman/aws-cb-race-3.git
cd aws-cb-race-3

export AWS_PROFILE=your-demo-profile
aws sts get-caller-identity

make install
make check
make synth
make bootstrap # Run once if this account and region aren't bootstrapped.
make diff
make deploy

Review the diff before deploying. The stack outputs include the ECS cluster and service names, the master-key secret ARN, and the CloudWatch log group name.


2. Open the tunnel and make a real Bedrock call

In terminal A, open the tunnel and leave it running.

export AWS_PROFILE=your-demo-profile
make tunnel

The script finds the running task and uses the gateway container's runtime ID to start an SSM port-forwarding session. If ECS replaces the task, reopen the tunnel.


In terminal B, check readiness and run the smoke test.

export AWS_PROFILE=your-demo-profile
curl --noproxy '*' -i http://127.0.0.1:4000/health/readiness
make smoke

There's an easy failure to diagnose here. If the smoke test says it can't reach  /health/readiness , check the tunnel first. Deployment leaves that step to the operator. An HTTP 200 from readiness shows that the proxy is reachable. The smoke test goes further. It checks that anonymous model listing is rejected, creates a temporary  nova-micro  key with a $0.01 budget, reads back its policy, requires a nonempty Bedrock completion, and deletes the key.


3. Exhaust App A's budget and send a request from App B

With the tunnel still running, run the live budget test.

make test-budgets

The script creates two keys for  nova-micro , each with a 30-minute lifetime. It sends synthetic inputs through App A's key and uses returned token counts to estimate when to stop generating load. Then it polls AI Gateway's recorded spend and sends a small probe that must return a budget-specific error. An HTTP 429 by itself would be ambiguous because rate limiting can produce the same status. Finally, the script checks that App B still has zero spend before its first request, requires a successful completion, and waits for App B's own charge to appear. It attempts to delete both keys even if a check fails.


Application
Configured budget
Recorded spend shown by the run
Observed outcome
App A
$0.01
$0.010573
Budget-specific denial after exhaustion
App B
$0.10
$0.000001
Completion succeeded

App A's recorded spend was 5.73% above its configured threshold at the point reported by the test. That is one observed run, with values taken from the gateway's spend records. The useful distinction is between denying further requests after recorded spend crosses a threshold and reserving the maximum possible cost before each request. This deployment demonstrated the former. The workload has a client-side request limit and an eight-token output limit per request. Those limits and the virtual-key budget cannot cap the AWS bill.




4. Test retries where you control the failures

It's difficult to reproduce precisely two throttles followed by a successful Bedrock response. The reliability suite uses the same pinned AI Gateway image and a local mock provider inside a Docker container with external networking disabled. It asserts the order of upstream requests, the client HTTP status, and the response marker. You can run it without AWS credentials, a database, or a tunnel.


make test-reliability

All four cases passed in the recorded run.

Case
Primary responses
Backup responses
Client result
Baseline
200
—
200 from primary
Retry
429, 429, 200
—
200 from primary
Fallback
503, 503, 503
200
200 from backup
Exhaustion
503, 503, 503
503, 503, 503
503

The backup route exists in the test configuration. The deployed gateway exposes one Bedrock model alias. The suite demonstrates how the pinned proxy handles the configured failures, with live Bedrock behavior covered separately by the smoke and budget tests.





To repeat the complete check, keep the tunnel open and run  make verify  from terminal B. The recorded clean run passed stack destruction, redeployment, and verification, including three infrastructure tests, five unit tests, the smoke test, both budget assertions, and all four local reliability scenarios.


What this design costs and where it stops

The database keeps key and spend state outside the Fargate task. That state comes with a standing cost. The repository estimates about $69 for 730 hours in  us-east-1 , assuming one 0.5 vCPU / 2 GiB Fargate task, one public IPv4 address, and Aurora at its configured 0.5 ACU floor. This estimate excludes storage, I/O, logs, secrets, image storage, and inference. App A and App B can stop making requests while those infrastructure charges continue.


One task and one writer keep the demo small. The service can briefly go offline during an update. The proxy uses the generated database user for both runtime access and schema initialization. SSM is an operator access path, so a shared application service would need its own ingress and authorization design. The two-key test establishes independent spend accounting and enforcement within one deployment. Customer data isolation, recovery, and high availability would each need their own design and verification.


5. Clean up

Stop the tunnel in terminal A. In terminal B, confirm your AWS identity and destroy the demo stack.

aws sts get-caller-identity
make destroy

Confirm that  SpeedRunAiGateway  reaches  DELETE_COMPLETE  in CloudFormation. The demo deletes the database and its key, budget, and spend history without a final snapshot. The shared  CDKToolkit  bootstrap stack and its ECR assets may remain. Check for other consumers before removing those shared resources.


Conclusion

The result I wanted was an ordinary failure that affected only one application. App A crossed its budget and received a denial. App B went through the same proxy to the same model and succeeded. The useful test was the request I sent after the failure.

The same API boundary gives the design room to grow. A provider such as OpenAI or Anthropic could be added behind the AI Gateway, with its credentials and routing managed at the gateway. If the endpoint, bearer-key authentication, model alias, and API behavior stay compatible, the applications can keep their existing code and access pattern. The live deployment verified Amazon Bedrock. Any additional provider route would need its own checks for request compatibility, spend accounting, and failure behavior.


Series: AI-kido (1 article)

  1. 1
    Building an AI Gateway on ECS Fargate with per-application budgets This article
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article