Building a Fault-Tolerant AI Gateway for Amazon Bedrock

How circuit breakers, chaos engineering, and graceful degradation keep your AI applications available when foundation models don't.

Mario Sageth

Sr. Technical Account Manager

Published Jun 15, 2026

Last modified Jun 15, 2026

Introduction

Foundation model APIs fail differently than traditional APIs. A REST endpoint either works or returns an error. An LLM endpoint can throttle gradually, spike latency from 200ms to 30 seconds without returning an error, deprecate a model version with 90 days notice, or silently degrade response quality under load. These failure modes break conventional retry logic and health checks.

Most LLM gateway guides focus on cost optimization and multi-model routing. This post focuses on what happens when things go wrong — and how to architect an AI gateway that degrades gracefully rather than failing catastrophically. I built a custom LLM gateway (codenamed Nadir) on Amazon EKS with Amazon Bedrock, specifically to handle these failure modes. The patterns described here apply to any LLM gateway architecture.

About Nadir: Nadir is a custom LLM gateway I built to demonstrate the resilience patterns in this post. It is not an official AWS service, AWS sample, or commercial product — it's a personal project shared in the builder community to illustrate how these patterns work together in a real implementation. The code is available as a reference architecture; the patterns themselves are what matter and can be applied with any LLM gateway framework (LiteLLM, Kong AI Gateway, or your own).

LLM-Specific Failure Modes

Before designing resilience, you need to understand how LLM APIs fail. These are fundamentally different from traditional API failures:

Failure Mode	How It Manifests	Why Traditional Retries Fail
Throttling (429)	Gradual — starts with occasional 429s, escalates to sustained rejection	Retrying immediately makes it worse; you need backoff + fallback
Latency degradation	P99 creeps from 2s to 15s without errors	Health checks pass, but users experience timeouts
Model deprecation	Model ID stops working after EOL date	No error until the day it breaks — requires proactive routing
Quota exhaustion	Daily/monthly token limit hit, all requests rejected	Not recoverable by retry — need a different model or cached response
Silent quality degradation	Model returns responses but quality drops under load	No error signal — requires output quality monitoring
Regional outage	Amazon Bedrock unavailable in one Region	Cross-region inference profiles or multi-region gateway needed

The key insight: most LLM failures are partial, not binary. Your gateway must handle the spectrum between "working perfectly" and "completely down."

Architecture: Resilient Gateway on Amazon EKS

Every component in this architecture has a failure mode and a corresponding resilience mechanism. Let's walk through each pattern.

Pattern 1: Circuit Breaker with Multi-Model Fallback

The circuit breaker pattern prevents cascading failures when a model is degraded. Instead of hammering a throttled endpoint (making throttling worse), the gateway detects failure and routes around it.

How It Works

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
class BedrockRouter:
    def __init__(self):
        self.circuit_breaker = {
            'primary':  {'failures': 0, 'state': 'closed'},  # Sonnet
            'fallback': {'failures': 0, 'state': 'closed'},  # Haiku
            'economy':  {'failures': 0, 'state': 'closed'},  # Nova Lite
        }

    async def complete(self, request, user_id, tenant_id):
        for model_tier in ['primary', 'fallback', 'economy']:
            if self._is_circuit_open(model_tier):
                continue  # Skip degraded model

            try:
                response = await self._invoke_bedrock(model_tier, request)
                self._record_success(model_tier)
                return response
            except ThrottlingException:
                self._record_failure(model_tier)
                continue  # Try next tier
            except ServiceUnavailableException:
                self._record_failure(model_tier)
                continue

        # All models failed — serve from cache or return 503
        return self._serve_degraded_response(request)

Configuration

1
2
3
4
5
6
7
8
9
10
11
12
13
routing:
  strategy: cost_optimized
  fallback_on:
    - throttling
    - overloaded
    - timeout
    - error_5xx

  circuit_breaker:
    enabled: true
    failure_threshold: 5      # Open after 5 consecutive failures
    timeout_seconds: 30       # Try again after 30s (half-open)
    half_open_requests: 3     # Test with 3 requests before closing

Why Three Tiers?

Tier	Model	Cost	Use Case
Primary	Claude Sonnet 4.6	$$$	Complex reasoning, code generation
Fallback	Claude Haiku 4.5	$$	Most conversational tasks
Economy	Amazon Nova Lite	$	Simple Q&A, classification

This isn't only cost optimization — it's a resilience strategy. When Sonnet is throttled, Haiku handles traffic. When both Anthropic models are degraded (rare but possible), Nova Lite keeps the application alive. The user experience degrades gracefully rather than failing completely.

Pattern 2: Semantic Caching as a Resilience Layer

Most teams think of caching as a cost optimization. In a resilient gateway, the cache is your last line of defense — when all upstream models are unavailable, cached responses keep the application functional.

How It Works

1
2
3
4
5
6
7
8
9
caching:
  enabled: true
  backend: redis
  ttl_seconds: 3600

  semantic:
    enabled: true
    similarity_threshold: 0.95
    embedding_model: amazon.titan-embed-text-v1

When a request arrives, the gateway generates an embedding using Amazon Titan and compares it against cached embeddings using cosine similarity. If a cached response is ≥95% similar, it's returned immediately — no Bedrock call needed.

Resilience Benefit

Under normal operation, semantic caching reduces Bedrock calls by 40-60%. Under failure conditions, it becomes critical:

Scenario	Without Cache	With Cache
Amazon Bedrock throttled (50% rejection)	50% of requests fail	Only novel requests fail; repeated queries succeed
Amazon Bedrock fully unavailable (2 min)	100% failure for 2 minutes	Only first-time queries fail; 40-60% of traffic served from cache
Redis degraded (500ms latency)	Rate limiter fails open, cache bypassed	Gateway falls through to Bedrock directly (graceful bypass)

Cache-Aside Under Failure

When Redis itself is degraded, the gateway doesn't fail — it bypasses the cache and routes directly to Amazon Bedrock. This "fail-open" behavior is critical: a cache failure should never cause a request failure.

Pattern 3: Chaos Engineering for AI Workloads

You can't trust resilience patterns you haven't tested. We use Chaos Mesh to inject failures and validate that the gateway degrades gracefully.

Test 1: Pod Termination (Validates PDB + HPA)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
# Terminate a random Nadir pod every 2 minutes
apiVersion: chaos-mesh.org/v1alpha1
kind: PodChaos
metadata:
  name: nadir-pod-terminate
spec:
  action: pod-kill
  mode: one
  selector:
    labelSelectors:
      app: nadir
  scheduler:
    cron: "*/2 * * * *"
  duration: "30s"

What we validate: PDB keeps minimum 2 pods alive. HPA replaces the terminated pod within 60 seconds. No requests are dropped (ALB drains connections before pod termination).

Test 2: Redis Latency Injection (Validates Graceful Bypass)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
# Add 500ms latency to all Redis connections
apiVersion: chaos-mesh.org/v1alpha1
kind: NetworkChaos
metadata:
  name: nadir-redis-latency
spec:
  action: delay
  delay:
    latency: "500ms"
    jitter: "50ms"
  direction: to
  target:
    selector:
      labelSelectors:
        app: redis
  duration: "5m"

What we validate: Rate limiter fails open (doesn't block legitimate requests). Cache lookups time out and are bypassed. Requests route directly to Bedrock. No user-visible errors — only increased latency and Bedrock cost.

Test 3: Amazon Bedrock Endpoint Block (Validates Circuit Breaker + Fallback)

1
2
3
4
5
6
7
8
9
10
11
12
13
# Block all traffic to Amazon Bedrock VPC endpoint
apiVersion: chaos-mesh.org/v1alpha1
kind: NetworkChaos
metadata:
  name: nadir-bedrock-block
spec:
  action: loss
  loss:
    loss: "100"
  direction: to
  externalTargets:
    - "bedrock-runtime.*.amazonaws.com"
  duration: "2m"

What we validate: Circuit breaker opens within 5 failures (~10 seconds). All three model tiers trip. Gateway serves cached responses for known queries. Novel queries receive 503 with Retry-After header. Recovery is automatic when Bedrock returns — circuit breaker enters half-open state and tests with 3 requests before closing.

Results from Chaos Testing

Test	Expected Behavior	Actual Result
Pod terminate (1 of 3)	Zero dropped requests	✅ Zero drops — PDB + graceful drain
Redis 500ms latency	Cache bypass, direct to Amazon Bedrock	✅ Latency +200ms, no errors
Amazon Bedrock full block	Cached responses served, 503 for novel	✅ 45% of traffic served from cache, 55% got 503
All three combined	Degraded but alive	✅ Cached responses only, P99 = 800ms

Pattern 4: Kubernetes Resilience Stack for LLM Traffic

LLM traffic has unique characteristics that require tuned Kubernetes configuration:

Long-lived connections (streaming responses can last 30+ seconds)
Bursty load (a single user can trigger 10x token consumption)
Memory-intensive (large request/response bodies, embedding vectors)

Pod Disruption Budget

1
2
3
podDisruptionBudget:
  enabled: true
  minAvailable: 2

During node upgrades or EKS Auto Mode consolidation, at least 2 pods always remain available. With 3 minimum replicas, this means only 1 pod can be disrupted at a time.

Horizontal Pod Autoscaler

1
2
3
4
5
6
7
8
9
10
11
12
13
autoscaling:
  enabled: true
  minReplicas: 3
  maxReplicas: 50
  targetCPUUtilizationPercentage: 65
  metrics:
    - type: External
      external:
        metric:
          name: nadir_requests_in_flight
        target:
          type: AverageValue
          averageValue: "20"

We scale on requests in flight, not just CPU. LLM requests are I/O-bound (waiting for Amazon Bedrock), so CPU stays low even when the pod is saturated with pending requests. Scaling on in-flight requests catches this.

Topology Spread

1
2
3
4
topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: topology.kubernetes.io/zone
    whenUnsatisfiable: DoNotSchedule

Pods are spread evenly across 3 Availability Zones. If an AZ fails, 2/3 of capacity remains — enough to serve traffic while EKS Auto Mode provisions replacement nodes.

EKS Auto Mode for Just-in-Time Nodes

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
# EKS Auto Mode NodePool — Graviton Spot for cost, On-Demand for baseline
apiVersion: karpenter.sh/v1
kind: NodePool
spec:
  template:
    spec:
      requirements:
        - key: kubernetes.io/arch
          operator: In
          values: ["arm64"]
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["spot", "on-demand"]
      nodeClassRef:
        name: nadir-nodes
  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized
    consolidateAfter: 60s

EKS Auto Mode provisions Graviton Spot instances for burst capacity and consolidates when load drops. The gateway is stateless — Spot interruptions are handled by PDB and graceful pod termination.

Observability: Detecting Degradation Before Failure

The most dangerous failures are the ones that don't trigger alerts. These metrics predict degradation before users notice. Grafana dashboards surface these in real-time from Prometheus, while Langfuse traces every LLM generation with prompt/response/token/cost data — letting you correlate model degradation with specific request patterns.

Key Metrics

Metric	Normal	Warning	Critical	Action
`nadir_circuit_breaker_state`	All closed	1 open	2+ open	Investigate Bedrock health
`nadir_cache_hit_ratio`	>40%	<30%	<15%	Cache warming issue or novel traffic spike
`nadir_request_latency_p99`	<3s	>5s	>10s	Model degradation or Redis latency
`nadir_fallback_rate`	<5%	>20%	>50%	Primary model throttled
`nadir_requests_in_flight`	<15/pod	>25/pod	>40/pod	Scale up or rate limit
`nadir_bedrock_throttle_count`	0	>10/min	>50/min	Request quota increase

CloudWatch Alarm Example

1
2
3
4
5
6
7
8
# Alert when fallback rate exceeds 20% for 5 minutes
- AlarmName: NadirHighFallbackRate
  MetricName: nadir_fallback_rate
  Threshold: 0.20
  ComparisonOperator: GreaterThanThreshold
  EvaluationPeriods: 5
  Period: 60
  TreatMissingData: breaching

The fallback rate is often one of the most important resilience metrics to monitor. A sustained >20% fallback rate means your primary model is degraded — even if no errors are being returned to users.

Design Decisions and Trade-offs

Decision	Trade-off	Why We Chose It
Fail-open rate limiter	Risk of cost spike during Redis failure	Availability > cost protection for short outages
3-tier model fallback	Lower quality on fallback	Degraded response > no response
95% similarity threshold	Occasional stale cache hits	High cache hit ratio > perfect freshness
30s circuit breaker timeout	Slow recovery after brief blips	Prevents thundering herd on recovery
Stateless pods	No session affinity	Simplifies scaling and failure recovery

Results

After 8 weeks of production operation with periodic chaos testing:

Metric	Target	Achieved
Availability	99.9%	99.95%
RTO (pod failure)	<60s	~30s
RTO (AZ failure)	<5 min	~2 min
RPO	0 (stateless)	0
Cache hit ratio	>40%	47%
Fallback rate (normal)	<5%	2.3%
P50 latency (cached)	<500ms	180ms
P99 latency (uncached)	<10s	4.2s

Getting Started

1
2
3
4
git clone https://gitlab.com/mesageth/custom_llm_gateway.git
cd gen-ai-nadir-gateway
export AWS_REGION=us-west-2
./scripts/deploy.sh dev

The automated deployment creates the full stack in ~20 minutes: VPC (3 AZs), EKS, Amazon ElastiCache for Redis OSS (Multi-AZ), DynamoDB, CloudFront, WAF, and the gateway application via Helm.

Prerequisites

AWS account with Amazon Bedrock model access (Claude Sonnet, Haiku, Nova Lite)
AWS CDK v2 installed
kubectl and Helm configured
Docker for building the gateway image

Resource Cleanup

When resources are no longer needed:

1
./scripts/undeploy.sh dev

This removes all deployed resources including the EKS cluster, Redis, DynamoDB tables, and networking components. CloudWatch log groups are retained for 30 days per the configured retention policy.

Conclusion

LLM reliability isn't about preventing failures — it's about designing systems that maintain useful behavior when failures inevitably occur. The patterns in this post — circuit breakers tuned for LLM failure modes, semantic caching as a resilience layer, chaos engineering for AI workloads, and Kubernetes configurations for long-lived streaming connections — transform an LLM gateway from a single point of failure into a fault-tolerant control plane.

The most important shift is mental: stop thinking of model fallback as "cost optimization" and start thinking of it as "disaster recovery." Your economy tier isn't a budget option — it's your last line of defense before returning a 503.

*Built following the AWS Well-Architected Framework — Reliability Pillar . Source code: https://gitlab.com/mesageth/custom_llm_gateway.git

# resilience # chaos-engineering # llm # amazon-bedrock # amazon-eks

Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.

Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article

Builder Center

Building a Fault-Tolerant AI Gateway for Amazon Bedrock

Mario Sageth

Introduction

LLM-Specific Failure Modes

Architecture: Resilient Gateway on Amazon EKS

Pattern 1: Circuit Breaker with Multi-Model Fallback

How It Works

Configuration

Why Three Tiers?

Pattern 2: Semantic Caching as a Resilience Layer

How It Works

Resilience Benefit

Cache-Aside Under Failure

Pattern 3: Chaos Engineering for AI Workloads

Test 1: Pod Termination (Validates PDB + HPA)

Test 2: Redis Latency Injection (Validates Graceful Bypass)

Test 3: Amazon Bedrock Endpoint Block (Validates Circuit Breaker + Fallback)

Results from Chaos Testing

Pattern 4: Kubernetes Resilience Stack for LLM Traffic

Pod Disruption Budget

Horizontal Pod Autoscaler

Topology Spread

EKS Auto Mode for Just-in-Time Nodes

Observability: Detecting Degradation Before Failure

Key Metrics

CloudWatch Alarm Example

Design Decisions and Trade-offs

Results

Getting Started

Prerequisites

Resource Cleanup

Conclusion