Managing Caller-Specific TPM Limits with Amazon Bedrock AgentCore Gateway
By Juan Pablo Garcia and Shruthi Rajoli
Imagine a team launching a new AI agent. Its traffic looks normal until a retry loop begins. One workload starts consuming a large share of the tokens available to a model. Other teams have changed nothing, but their requests begin to throttle because they use the same service-managed capacity.
This is the noisy-neighbor problem for generative AI. Amazon Bedrock quotas protect model capacity at the account and Region level, but several teams, tenants, or agents can still share that capacity. Without a caller-aware control, one traffic spike can affect everyone using the same model:
- Availability: one caller can contribute enough traffic to increase throttling for others.
- Fairness: there is no caller-level boundary to keep one workload from dominating the shared path.
- Governance: the application cannot express different limits for individual callers or models.
We wanted a smaller failure boundary: if one caller exceeds its allowance, reject that caller at the Gateway while another caller can continue using the same Gateway, target, model, and API.
The decision: enforce limits at the shared entry point
You could track tokens in each application. However, doing so requires a distributed counter, concurrency control, input-token estimates before each response, reconciliation with actual usage afterward, and a consistent caller identity across every application instance.
This example does not build that distributed system. Instead, it uses Amazon Bedrock AgentCore Gateway as the common enforcement point. The Gateway already validates callers and receives inference requests before they reach the model. It can apply token-per-minute (TPM) limits using identity from a validated JSON Web Token (JWT).
AgentCore Gateway supports request-rate limits, token-rate limits, and connection-rate limits. Token limits apply to inference targets and control token usage rather than only the number of requests. This matters because two requests can consume very different numbers of tokens.
The experiment uses two Amazon Cognito machine-to-machine clients, Caller0 and Caller1. Both send
bedrockmantle/openai.gpt-5.4 to the /inference/v1/responses path. The rules key traffic on the validated JWT sub claim. The test treats the access tokens as opaque values, so it does not decode or assert their subjects.1
2
3
4
5
6
7
8
9
10
11
flowchart TD
C0["Caller0<br />JWT sub A"] --> GW
C1["Caller1<br />JWT sub B"] --> GW
subgraph GW["Amazon Bedrock AgentCore Gateway"]
AUTH["Validate JWT"]
L1["Caller + model: 2,000 TPM"]
L2["Caller overall: 6,000 TPM"]
AUTH --> L1 --> L2
end
GW --> T["bedrock-mantle target"]
T --> M["openai.gpt-5.4"]The target uses the
bedrock-mantle connector and GATEWAY_IAM_ROLE for outbound credentials. The managed connector handles model discovery, model-ID translation, and path rewriting for the Bedrock Mantle endpoint.The implementation: two caller-aware limits
We configured two limits:
- 2,000 TPM per caller and model limits one caller's traffic to
openai.gpt-5.4. - 6,000 TPM per caller overall limits one caller across matching inference traffic.
Both rules use
$.context.jwt.sub, so the Gateway derives caller identity from the validated token instead of trusting a value supplied in the request body.1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
[
{
"id": "per-user-per-model-tpm",
"dimension_keys": ["qualifiedModelId", "$.context.jwt.sub"],
"entries": [{
"dimensions": {
"qualifiedModelId": "${model_id}",
"$.context.jwt.sub": "*"
},
"tokens": { "rate": 2000, "period": "minute" }
}]
},
{
"id": "per-user-overall-tpm",
"dimension_keys": ["$.context.jwt.sub"],
"entries": [{
"dimensions": { "$.context.jwt.sub": "*" },
"tokens": { "rate": 6000, "period": "minute" }
}]
}
]The fields work together:
dimension_keysdefine the values that identify a rate-limit bucket.entries[].dimensionsselect the traffic to which a rule applies.- The
*matches every resolved subject, while each unique resolvedsubgets its own bucket. - A request must satisfy both matching rules. The model-specific rule applies to
openai.gpt-5.4; the caller-only rule covers matching inference traffic across models.
In the repository,
config.json defines these limits, deploy.py renders them as CloudFormation resources, and test_ratelimit.py performs the sequential caller check. The current test uses one model and is designed for the 2,000 TPM caller-and-model rule to bind first. The automated test does not assert which rule caused a throttle, and it does not exercise the 6,000 TPM cross-model behavior.These controls are traffic-management ceilings. They do not reserve capacity, increase the underlying Bedrock quota, or guarantee throughput. Service-managed limits and model availability still apply.
The result: contain the noisy neighbor
After obtaining tokens for both callers, the script sends up to 12 requests as Caller0. If it classifies Caller0 as throttled, it sends the same request once as Caller1. In one manually recorded validation run, we observed:
| Request(s) | Caller | HTTP | Result |
|---|---|---|---|
| 1-6 | Caller0 | 200 | Requests completed |
| 7 | Caller0 | 429 | per-user-per-model-tpm reported |
| 1 | Caller1 | 200 | Request completed |
In that manually inspected 429 response,
limitKey named per-user-per-model-tpm and retryAfter was present. Caller1 then received HTTP 200 through the same Gateway, target, model, and API.The captured sequence is consistent with Caller0 encountering the model-specific limit while Caller1 could still make a request. The exact throttle point can vary because TPM is time-based and the Gateway estimates input tokens before forwarding a request, then records actual input and output usage after the response.
The automated test is intentionally a smoke test. It treats any HTTP 429, or a throttle-like HTTP 400 or 403 response, as Caller0 being throttled and then checks for Caller1 HTTP 200. It does not assert
limitKey, establish that the 429 came from the configured Gateway rule, or independently verify bucket identity.Production considerations
Before applying this pattern in production:
- For production deployments, store client secrets in a secrets service such as AWS Secrets Manager. This demo exposes them through CloudFormation outputs only to support automated testing; do not use this approach in production.
- Configure Gateway
AllowedScopes. Cognito defines and the test requestsagentcore-gateway/invoke, but the demo Gateway authorizer validatesAllowedClientsonly. - Use stable, bounded identity claims such as subject, tenant, workload, or service tier.
- Narrow the Gateway IAM role to the required models, Regions, resources, and actions.
- Honor
retryAfterwith bounded retries and jitter to avoid retry storms. - Treat rate limits as traffic management, not authorization. Rate limiting is fail-open when the service is unavailable or a dimension cannot be resolved.
Conclusion
Recall the problem: several callers share the same model quota, so a traffic spike from one caller can contribute to throttling for others. Instead of building a distributed token counter in every application, this example places caller-aware TPM controls at AgentCore Gateway.
In the manually recorded run, Caller0 received a 429 that named the per-caller, per-model rule, while Caller1 then succeeded on the same route. The sequence matched the intended behavior and illustrated three intended benefits:
- Availability: limit how much traffic one identity can contribute through the shared Gateway.
- Fairness: apply separate caller-aware limits using a validated identity.
- Governance: define policies by caller and model without adding shared counter state to each application.
The larger win is where this logic lives. The Gateway applies the policy in the path that already authenticates and routes inference traffic. From here, you can route one caller across several models to test the broader 6,000 TPM rule, or add request-rate and connection-rate limits when you also need to control request volume.
These limits remain traffic-management guardrails. They do not reserve model capacity or increase the underlying Bedrock quota, and rate limiting is fail-open. Continue to use authentication and authorization for access control, and plan capacity separately when the service-managed quota is too small.
References
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article