
AI Gateway Patterns on AWS
AI coding tools such as Claude Code are moving from individual use to organization-wide deployment on Amazon Bedrock. At that scale, teams need consistent access control, cost attribution, and observability. This article describes what Amazon Bedrock provides natively for these needs, the enterprise pattern of placing an LLM gateway in front of Bedrock, the gap that pattern addresses, and a reference implementation.
AI coding tools have moved quickly from individual experimentation to organization-wide adoption for developer productivity. Tools such as Claude Code now run across entire engineering teams, and many organizations route that traffic to models hosted on Amazon Bedrock so that inference runs inside their own AWS account.
This article looks at what changes when that deployment reaches scale, what Amazon Bedrock provides for governance and cost tracking on its own, why some organizations place an LLM gateway in front of Bedrock, and how a production implementation of that pattern fits together. It uses Claude Code as the running example because AWS publishes first-party guidance for running it on Bedrock. The same considerations apply to other AI coding tools.
Deploying developer AI tooling at scale
A single developer using an AI coding tool is a simple case. The tool sends prompts and code context to a model and returns a response. When the same tool is deployed across hundreds or thousands of developers, several operational concerns move to the foreground.
- Volume. Each active developer generates a continuous stream of model calls throughout the working day. Aggregate request and token volume grows with headcount and usage.
- Access control. The organization needs to decide which teams can use which models, and to apply that decision consistently.
- Cost attribution. Finance and platform teams need to see which teams, projects, or users are driving spend, and to read that from a reliable source.
- Observability. Platform teams need usage, latency, and error data to operate the service and to catch problems before developers do.
- Data handling. Security teams need to know where prompts and code context are processed and how that data is protected.
These concerns exist regardless of which tool or model is used. The question for a platform team is where each concern is handled: inside the model platform, inside the tool, or in a layer the organization builds and operates itself.
Why the gateway pattern is emerging
Across enterprise deployments, a common pattern has emerged: an LLM gateway placed between developers and the model providers. A gateway is a proxy that receives requests from many clients, applies organization-wide policy, and forwards each request to a backend model. The AWS Solutions Library publishes a reference implementation of this pattern, the Guidance for Multi-Provider Generative AI Gateway on AWS , built on the open-source LiteLLM proxy and deployed as a containerized service on Amazon ECS or Amazon EKS ( AWS Well-Architected Generative AI Lens ).
The gap the gateway pattern addresses is centralization. Without a shared layer, each of the concerns above is handled in a different place and often more than once.
- Access rules and model choices live in each tool's configuration on each developer's machine.
- Usage and cost data are spread across provider bills and tool dashboards.
- Rate limits and safety controls are applied per tool rather than per organization.
- Adding or changing a model provider means touching many clients.
A gateway consolidates these into one control point. It gives the platform team a single place to set model access, apply rate limits and guardrails, capture usage and cost data, and route across one or more providers. This is the same reason the pattern appears in the AWS multi-provider gateway guidance, where a single OpenAI-compatible endpoint fronts more than one hundred models.
The pattern has a cost. A gateway is production infrastructure that sits in the path of every request. It has to be deployed, scaled, monitored, and kept available. Whether that cost is justified depends on which of the concerns above the native platform already covers, which is the next question.
The Bedrock default: what is available
Amazon Bedrock handles several of these concerns directly. Running Claude Code against Bedrock without any additional layer already provides the following.
Data handling. Bedrock processes prompts and code context inside the customer's AWS account, under the customer's encryption keys, network path, and retention settings. The model providers do not have access to those accounts, so they cannot see prompts, completions, or Bedrock logs ( Bedrock data protection ). AWS publishes Guidance for Claude Code with Amazon Bedrock that federates developers through an identity provider into short-lived AWS credentials scoped to Bedrock, so long-lived keys are not stored on developer machines.
Access control. Access to Bedrock and to specific models is governed by IAM policy and, at the organization level, by service control policies that can deny models or Regions across accounts. Model access is enabled per account and per Region.
Observability. CloudWatch metrics report per-model invocation counts, token counts, and latency. Model Invocation Logging writes full request and response records to CloudWatch Logs, Amazon S3, or both.
Cost attribution. Bedrock provides more than one native mechanism for attributing spend. The right one depends on how traffic is shaped.
Mechanism | What it attributes by | Notes |
|---|---|---|
IAM identity tracking | The calling IAM identity, recorded on the Cost and Usage Report (version 2.0) | Works well when a service or tool calls under its own role. Role tags and session tags flow through as cost allocation data. Requires CUR 2.0 with caller identity enabled on the export. |
Application Inference Profiles | Tags on a profile that wraps a model | A clean fit for Anthropic Claude models. Covers the InvokeModel and Converse APIs. Profile creation can be automated as teams and models are added. |
Projects | The OpenAI-Project request header | Covers the Responses and Chat Completions APIs. |
Workspaces | The anthropic-workspace-id request header | Covers the Anthropic Messages API. Claude Code uses that API, so Workspaces can tag Claude Code traffic directly. Workspaces and Projects are the same underlying resource. |
Cost allocation tags must be activated in the Billing console before they appear in Cost Explorer and the Cost and Usage Report. Activation is not retroactive and can take up to 24 hours. That report is the record of actual billed amounts after discounts, which makes it the authoritative source for cost data.
For a deployment where each service or tool calls Bedrock under its own identity, these native mechanisms cover access control, observability, and cost attribution without additional infrastructure.
Where a gateway fits
A gateway adds capabilities that the native path does not provide on its own. The following are the common reasons an organization places one in front of Bedrock.
Capability | What the gateway adds |
|---|---|
Multi-provider access | A single endpoint that routes to Bedrock and to other providers or self-hosted models, so clients use one interface and one set of credentials. |
Central model control | One place to set the default model, allow specific models per team, and roll out a model change without editing every client. |
Rate limiting and guardrails | A single point to enforce request and token limits and to apply prompt-level safety controls uniformly across all traffic. |
Real-time budget enforcement | Logic that blocks or throttles requests when a team or key crosses a spend threshold, using the gateway's own usage counters. |
Custom cost dimensions | Per-user, per-project, or per-request cost data captured at the gateway, including dimensions that can change on every request. |
The last row interacts with how Bedrock attributes cost, and it is worth being precise about. Bedrock attributes spend to the identity that makes the call. When a gateway forwards requests from many developers under a single shared IAM role, Bedrock records that shared role as the caller ( IAM principal cost tracking ). Per-team and per-user detail is therefore not present on the Cost and Usage Report by default.
An organization that runs a gateway and needs per-team or per-user attribution has several options that work alongside Bedrock's native mechanisms.
- Route Claude Code traffic through Workspaces, which tag the traffic on the bill by workspace.
- Have the gateway assume a role per user and pass identity as session tags, so distinct identities appear on the report.
- Use one Application Inference Profile per team, tagged accordingly.
- Capture the additional dimensions at the gateway and join them with the Cost and Usage Report, keeping that report as the source of truth for billed amounts.
The choice depends on the granularity required and the effort the team is willing to take on. The point to plan for is that a shared gateway identity changes what appears on the bill, and the attribution approach should be decided before the deployment grows.
A production implementation
A production gateway for a client-side tool such as Claude Code has a consistent shape. Claude Code calls Bedrock directly from the developer's machine, so to insert a gateway the client is pointed at the proxy endpoint instead. The AWS multi-provider gateway guidance is one deployable starting point. The components below describe a representative deployment.
Components
- An identity provider federated to AWS, used to authenticate developers.
- The gateway proxy, deployed as a scalable, highly available service (for example, containers behind a load balancer).
- A credential store for any external provider keys, such as AWS Secrets Manager.
- A data store for gateway configuration, virtual keys, and usage counters.
- Amazon Bedrock as the primary model backend.
- A telemetry and cost pipeline that collects gateway logs and usage data.
Request flow
- A developer's Claude Code client sends a request to the gateway endpoint.
- The gateway authenticates the developer against the identity provider.
- The gateway applies policy: it checks the model allowlist, applies rate limits, and records usage for the authenticated developer.
- The gateway signs the request with SigV4 and forwards it to Amazon Bedrock under an IAM role.
- Bedrock runs inference and returns the response.
- The gateway returns the response to the client.
- Gateway logs and usage records flow to the telemetry and cost pipeline. Bedrock usage is recorded on the Cost and Usage Report under the gateway's IAM identity.
Developers Gateway (self-hosted) Amazon Bedrock ────────── ────────────────────── ──────────────┌──────────────┐ ┌─────────────────────────┐ ┌───────────────┐│ Claude Code │──── auth ────►│ authN via IdP │─ SigV4 ►│ Claude models ││ other tools │──── requests ►│ model allowlist │ └───────────────┘└──────────────┘ │ rate limiting │ │ │ usage + cost capture │ logs + │ cost │ observability │ usage ───►│ pipeline └─────────────────────────┘
Operational considerations
- Quotas. Bedrock runtime quotas are enforced per account, per Region, and per model, with separate requests-per-minute and tokens-per-minute limits. Estimate demand from active developers, requests per hour, and tokens per request, and request increases before rollout. A separate account for AI-tool inference keeps this traffic from competing with production workloads.
- Availability. The gateway sits in the request path, so it should run with multiple instances and health checks. Latency and error rate directly affect whether developers keep using the tool.
- Logging. Send gateway and Bedrock invocation logs to durable storage. Full request and response logs are sensitive, so encrypt them and restrict access.
- Cost attribution. Decide how per-team or per-user cost detail is recovered, using one of the approaches described above, and keep the Cost and Usage Report as the source of truth for billed amounts.
Summary
Deploying an AI coding tool such as Claude Code across an organization turns access control, cost attribution, observability, and data handling into operational requirements. Amazon Bedrock addresses these natively: inference runs in the customer's account, IAM governs access, CloudWatch and invocation logging provide observability, and several mechanisms attribute cost on the Cost and Usage Report. The gateway pattern centralizes these concerns into one control point and adds capabilities such as multi-provider routing, central model control, and custom cost dimensions, at the cost of running production infrastructure in the request path. A gateway also consolidates many developers behind one identity, which changes how cost appears on the bill and should be planned for. Understanding what the native platform already provides is the basis for deciding whether and how to add a gateway.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article