Choosing the Right LLM Gateway for Your Use Case
Many option on the market can do the job. What separates them is who carries the code.
Organizations often see a pattern appear when starting their first agentic workloads. One team builds a customer-facing assistant on Bedrock. A second team, in a different part of the organization, wires an internal tool to OpenAI with an API key in a config file. A third wants Claude for document processing. Within a small period, you have fragmented implementations to individually manage. Multiple authentication schemes, cost reports that do not reconcile, and a security review asking for an audit log of every prompt that nobody can produce. Each solved its own problem competently. The challenge is that governing access to models was never anyone's job, so it became everyone's.
In the eighteen months to September 2026, every major cloud provider, API gateway vendor, and observability company either shipped a dedicated AI gateway or bought one. The question is not whether to put something between your applications and your model providers but which choice best fits your needs, across four categories that make different bets about who should own the code.
What an LLM gateway is, and why your API gateway is not one
An LLM gateway sits between applications and model providers and mediates every call. Authentication, authorization, routing, policy, and logging happen there once, instead of inside each application. You likely already own an API gateway. Kong, Apigee, and Amazon API Gateway all sit in front of services and enforce policy. Why not put one in front of Bedrock and call it done? Those products were built for stateless request-response HTTP, and they treat the request body as an envelope to be delivered unopened. LLM traffic breaks that model in four ways:
- Responses stream. Server-Sent Events deliver tokens as the model produces them, so connections are long-lived and the response arrives in pieces.
- The routing key is inside the body. Which model to call is a JSON field, not a path or a header. A gateway that does not parse the body cannot route.
- Cost is measured in tokens. A requests-per-second limit says almost nothing about spend. One request can cost a hundredth of a cent or several dollars.
- The content needs inspection. Guardrails, PII detection, and prompt-injection defenses all require reading what is being sent and what comes back.
One more shift changes the shortlist. Over 2025 and 2026 these products stopped being LLM proxies and became agent infrastructure. They now mediate Model Context Protocol (MCP) tool calls and Agent-to-Agent (A2A) traffic, govern which tools an agent may invoke, and apply policy that depends on session state. MCP support is now the feature every vendor in this survey either ships or is visibly missing. If agents are on your roadmap, several otherwise good products drop out of contention.
Six problems a gateway solves
Inconsistent Authentication
Multiple teams, token-validation implementations, places for a bug to hide, and no single view of who can call which model. A gateway validates OAuth and JWT tokens from Entra ID, Okta, Cognito, or Auth0 in one place and enforces model-level role-based access there.
Quota exhaustion
Imagine a batch process with a bug loops against a large model over a weekend. Monday morning the shared quota is exhausted, the month's budget is gone, and the customer-facing application is throttled because of a report nobody reads. Token-level limits per user, team, and model turn this from an outage into an alert.
Compliance and Audit
In financial services, healthcare, and government, someone will eventually ask for an immutable record of every prompt and completion, evidence that harmful content was blocked before it reached the model, and proof that PII was caught. FINRA, PCI DSS, GLBA, and the EU AI Act all point in this direction. Retrofitting that across a dozen applications is a significant amount of work. At the gateway it is configuration.
Unexplainable Cost and Latency
Token counts per model, time to first token, p50/p90/p99 latency, and cost attributed to the calling team. Without these numbers a latency regression and a cost spike look identical, and both get blamed on the provider.
Switching providers means a rewrite
With direct integration, moving a workload from one model to another touches every application. With a gateway it is a routing rule. The same mechanism is what makes failover and model A/B testing possible.
Cognitive Overload
This is the least discussed benefit and often the largest. A developer points an existing OpenAI or Anthropic SDK at a URL and passes a JWT. No cloud SDK, no IAM credentials, no provider-specific API to learn. For a platform team supporting dozens of application teams, this is where most of the time gets saved.
Four Categories of Solutions
Managed cloud. AWS AgentCore Gateway. Purpose-built and operated by the provider. You configure it; they run it. Minimal operational burden, in exchange for coupling to one cloud.
Open source. LiteLLM, Kong AI Gateway 2.0, Portkey, Bifrost. Full flexibility and auditable source. You own operations, patching, and scaling.
Commercial. Solo.io agentgateway, Cloudflare AI Gateway. Packaged products with support contracts. Solo.io donated its core to the Linux Foundation, so the base is open source and the enterprise features are licensed.
Build your own. API Gateway plus Lambda in a sign-and-forward pattern. Total control, total ownership.
AWS AgentCore Gateway
AgentCore Gateway reached general availability in October 2025 and has steadily shipped improvements since: server-side tool execution, VPC egress, temporal policies and rate limiting. As of September 2026, AgentCore comprises 13 services, including Gateway, Runtime, Identity, Policy, Memory, and Observability. Three capabilities separate it from the rest of the field:
Authentication is declarative
The gateway validates JWTs natively (signature, expiry, audience, client ID, scopes, custom claims) against any OIDC-compliant identity provider. You describe the token you expect. There is no authorizer function to write, and none to patch when your identity provider rotates keys.
Policy is enforced outside the model
Cedar policies are evaluated by the gateway, not by the agent and not by the model. An adversarial prompt cannot talk a model into bypassing a policy the model never sees. The newer Dogwood language adds temporal conditions to Cedar: session-aware rate limits, running budgets, prerequisite steps, escalation triggers. Compare this to application-level guardrails, which are code your team wrote and the model is being asked to respect.
The same gateway carries agents
MCP tools, A2A, semantic tool discovery, and VPC Lattice integration are part of the product, alongside inference routing to Bedrock (50+ models), OpenAI, and Anthropic from a single endpoint. Adding agents later is a configuration change on infrastructure you already run.
Pricing is usage-based with no fixed component. The limitation is if your organization requires a gateway that can be deployed on another cloud or on-premises, additional solutions may need to be considered.
Open source: LiteLLM
LiteLLM has been the default open-source choice for two years. It has tens of thousands of GitHub stars, and presents an OpenAI-compatible API over more than 100 providers.
It runs as a Python ASGI proxy with PostgreSQL for configuration and spend tracking and Redis for rate limiting and caching. In June 2026 BerriAI began moving the hot path to Rust. The repository now describes itself as "Rust core with Python SDK," and third-party benchmarks put p50 overhead around 9ms.
The Gateway carries multiple tiers with different capabilities. The MIT core includes the gateway, virtual keys, and spend tracking. Enterprise Basic adds SSO past five users and semantic caching. Enterprise Premium adds air-gapped deployment and priority support.
In March 2026 the LiteLLM PyPI package was compromised. The malicious version exfiltrated Cloudflare, AWS, and GCP credentials at install time. It was caught and fixed, and the risk class is not specific to this project. But it is a concrete picture of what owning the supply chain means. When a dependency in your request path is compromised, your team is the incident response team, unless you are paying for enterprise support. Decide whether that is acceptable before the incident, not during it.
Open source: Kong AI 2.0, Portkey, and Bifrost
Kong AI Gateway 2.0 reached GA on September 1, 2026 after a private beta that opened in July. It is a separate product from Kong API Gateway with its own runtime, release cadence, and control plane, and it treats Models, MCP Servers, and Agents as first-class entities. Kong claims to be the only gateway covering LLM, MCP, and A2A in one product.
Portkey open-sourced its full enterprise gateway under MIT in March 2026. At that point it was processing over a trillion tokens and 120 million AI requests daily across 24,000+ organizations, with traffic growing 30 to 40% week over week. Palo Alto Networks announced its intent to acquire Portkey on April 30, 2026 and closed a month later; the product is being folded into Prisma AIRS. Evaluate it as a Palo Alto product.
Commercial: Solo.io agentgateway
In August 2025 Solo.io donated agentgateway to the Linux Foundation with backing from Microsoft, T-Mobile, Dell, CoreWeave, and Akamai. It has since been accepted as a growth-stage Agentic AI Infrastructure Foundation project, passed 1,000 GitHub stars in six months, and crossed 7 million downloads. Enterprise adds CEL-based policy, audit logging, budget enforcement, and tool governance (fingerprinting, versioning, sandboxing).
It is Kubernetes-native and runs alongside Istio ambient mesh. That is the strongest reason to choose it if you have that expertise, and the reason to look elsewhere if you do not.
Commercial: Cloudflare AI Gateway
Cloudflare's pitch is speed of adoption and edge reach. Setup is one line: change your base URL. The gateway runs across 300+ cities, fronts 350+ models, and the core features include analytics, caching, rate limiting, spend limits.
In August 2026 Cloudflare added identity-aware governance with spend limits per user and team denominated in dollars, not tokens. That is easier to explain to a finance partner than tokens per minute. Billing is unified: one Cloudflare invoice regardless of which providers you call, for a 5% fee on inference spend.
Two limitations settle this option for enterprises. There is no self-hosted or on-premises option; traffic transits Cloudflare's edge. And at scale, 5% of inference spend stops being a convenience fee and becomes a line item someone will ask about. There is also no MCP or A2A support today, which rules it out as an agent foundation.
Build your own: API Gateway plus Lambda
The reference pattern is the one Dynatrace developed for its own global AI features and published on the AWS Architecture Blog in November 2025: Route 53 for a custom domain, API Gateway as the entry point, a Lambda authorizer for JWT validation, and a Lambda integration that signs with SigV4 and forwards to Bedrock.
The advantage is procedural more than technical. API Gateway is probably already an approved service in your organization, which means you take one new service through enablement (Bedrock) instead of two. In a regulated enterprise where enablement runs weeks to months, that carries weight. You also get the full Bedrock API surface, including Converse, InvokeModel, Knowledge Bases, and Agents.
Cost and maintenance are two important items to consider. Two Lambda invocations sit in every request path. You own all of the code: JWT validation, SigV4 signing, guardrails, budget enforcement, observability. There is no multi-provider routing unless you build it and no path to MCP or A2A. API Gateway's 29-second integration timeout caps long-running inference.
On timing: Dynatrace built this because AgentCore did not exist yet. The blog post published one month after AgentCore reached GA. The pattern is sound; the question is whether the constraint that produced it still applies to you.
Comparison
| Dimension | AgentCore Gateway | LiteLLM | Solo.io | Cloudflare | DIY (API GW) |
|---|---|---|---|---|---|
| Auth | Declarative, native JWT | Virtual keys + JWT | MCP OAuth 2.1 | API keys + identity | Custom Lambda code |
| Client SDK | OpenAI / Anthropic | OpenAI-compatible | OpenAI-compatible | OpenAI-compatible | AWS SDK (boto3) |
| Multi-provider | Native (3) | Native (100+) | Native (all major) | Native (6+) | Bedrock only |
| MCP / A2A | Native | None | Native | None | None |
| Ops burden | Managed | Self-managed (PG + Redis) | Self-managed (K8s) | Managed (edge) | Full ownership |
| Agentic path | Native platform | Proxy only | Native platform | Proxy only | Not integrated |
How to decide
The axis that matters is ownership versus managed, not capability. Given enough engineering, every option here solves every problem here. Decide on what differs:
- How much code your team will maintain. DIY and open source mean writing and then maintaining auth, routing, guardrails, and observability as requirements change. Building it is the cheap part.
- Speed now against cost later. An already-approved service ships this quarter. A managed service costs less to run over the following three years. Which one you are optimizing for is a leadership question, not an engineering one.
- Whether this is a router or a platform. If MCP tools, A2A, managed memory, and policy engines are in your future, choose a platform now instead of a proxy you will replace.
- Enablement timeline. If the managed option is not yet approved, DIY is a legitimate stepping stone. Write down the migration path while you still remember why you chose it.
- Multi-cloud. LiteLLM, Portkey, Cloudflare, and Solo.io deploy anywhere. AgentCore does not. When this requirement is real, it decides the question by itself.
In short:
- AgentCore Gateway if you want zero custom auth code, a path to agents, per-user governance from JWT claims, and service enablement is achievable.
- LiteLLM or Portkey if multi-cloud is a hard requirement, you have the platform engineering depth for PostgreSQL and Redis, and source auditability matters.
- Kong or Solo.io if you already have a Kubernetes and service-mesh investment and want API and AI governance under one platform.
- Cloudflare if you want the fastest possible start, edge latency, dollar budgets, and cloud-only is acceptable.
- DIY if API Gateway is already approved, you are Bedrock-only, your team will own the code, and speed today outweighs cost over time.
Recommendation
If you are an enterprise running on AWS, use AgentCore Gateway. Not as a default to be argued out of, but because it is the only option in this survey where the three hardest problems (authentication, policy enforcement, and the path to agents) are solved by configuration instead of by code your team maintains.
Consider what the alternatives cost. The DIY pattern hands you two Lambda functions to own for the life of the platform: JWT validation that must track your identity provider, SigV4 signing, and whatever guardrails and budget logic you add on top. The open-source options replace that with a PostgreSQL and Redis deployment, a patch cadence, and a supply chain you are accountable for. While both are reasonable engineering choices, neither is free.
The policy argument is the one to put in front of your security review. Application-level guardrails and prompt instructions are things the model is asked to respect. Cedar policy at the gateway is enforced whether the model cooperates or not. For an organization with FINRA, PCI DSS, or EU AI Act exposure, that is the difference between a control and a suggestion.
Having a path that supports AI Agents matters even if you have no agents today. Every product without MCP and A2A support is a product you will migrate off when your first agentic workload arrives. AgentCore is the choice where that migration never happens.
Two situations change the answer. If multi-cloud deployment is a requirement, or AgentCore is not available to you, one of the alternatives may need to be considered. If AgentCore has not cleared service enablement in your organization, Dynatrace's API Gateway pattern is a sound way to start now; document the migration path the day you deploy it.
To begin, obtain governance requirements from security and compliance in writing before you compare products, because requirements eliminate more options than features do. Then start enablement, put a proof of concept behind one real application, and run a pilot with a named owner and a budget.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article