Build a local LiteLLM AI gateway to test model routing and cost attribution

Build a local LiteLLM gateway and test its load balancing, fallback, and per-user cost attribution capabilities across multiple model providers.

Gilberto Canales

Published Jun 22, 2026

Last modified Jun 22, 2026

Why an LLM gateway matters

Once you use more than one model or provider, every app ends up wiring its own keys, SDKs, retries, and cost tracking against each API, so switching or adding a model means touching code in every service. An LLM gateway removes that coupling: apps ask for a logical capability and the gateway picks the real model. That single indirection is what makes the hard things easy: evaluating new models on real traffic without client changes, fallbacks when a provider fails, load balancing across providers, and cost optimization with routing and spend control in one observable place. This guide brings that idea to life with a small, fully local demo.

What this test is and what it demonstrates

The goal is to test, end-to-end and locally, an AI Gateway: a single entry point through which all calls to language models flow, regardless of the provider (Groq, Gemini, Bedrock). Instead of each application talking directly to each API, they all point to the gateway (LiteLLM Proxy on :4000), which routes the request, spreads load across providers, retries with fallbacks, and logs every request. The underlying idea: adding or switching a model is editing configuration, not touching the consumer's code.

This demo focuses on two native LiteLLM capabilities, each exposed as a model group, a single logical name that clients call, behind which LiteLLM does the work:

Model group	Capability	What's behind it
`balanced-group`	Load balancing	Groq (Llama 3.1 8B) + Amazon Nova Micro, with equal weights (LiteLLM spreads traffic across both providers)
`fallback-group`	Fallback	Gemini 2.5 Flash as primary; if it fails, LiteLLM retries on Claude Haiku (Bedrock)

To generate real traffic, the stack spins up four client containers that run simultaneously and send requests at random intervals, each picking a random question from a fixed list:

Client	Calls	Exercises
`client-a`, `client-b`	`balanced-group`	Load balancing across Groq + Nova
`client-c`, `client-d`	`fallback-group`	Gemini primary → Haiku fallback

Each client calls only its assigned group, so the two capabilities are exercised in parallel under steady, realistic load. The clients ask for the logical group name (e.g. balanced-group), never a specific provider, so swapping or rebalancing the underlying models is just a config edit, with no client change.

Architecture at a glance

Four levels: the clients call a single endpoint (the gateway); the gateway owns the two model groups (load balancing / fallback are LiteLLM features, not client logic); each group fans out to the real LLMs across providers.

1
2
3
4
5
6
7
8
9
10
11
  LEVEL 1            LEVEL 2                     LEVEL 3                       LEVEL 4
  Clients            Gateway (LiteLLM)           Model groups (LiteLLM)        LLMs (providers)

  client-a ─┐                                  ┌──────────────────┐  ~50%   ┌─────────────────────┐
            ├──────► ┌───────────────────┐ ──► │  balanced-group  │ ──────► │ Groq · Llama 3.1 8B │
  client-b ─┘        │                   │     │ (load balancing) │ ──────► │ Bedrock · Nova Micro│
                     │   LiteLLM Proxy   │     └──────────────────┘  ~50%   └─────────────────────┘
  client-c ─┐        │      :4000        │     ┌──────────────────┐ primary ┌─────────────────────┐
            ├──────► │                   │ ──► │  fallback-group  │ ──────► │ Gemini 2.5 Flash    │
  client-d ─┘        └───────────────────┘     │ (primary→backup) │ ╌╌╌╌╌►  │ Bedrock · Claude Haiku
                                               └──────────────────┘  fail  └─────────────────────┘

Level 1 (clients): client-a/client-b call balanced-group; client-c/client-d call fallback-group. They only ever ask for the group name.
Level 2 (gateway): the LiteLLM Proxy receives every request and decides the real model.
Level 3 (model groups, managed by LiteLLM): balanced-group splits traffic ~50/50 across providers; fallback-group sends traffic to its primary and falls back only on failure.
Level 4 (LLMs): the real models. balanced-group lands on Groq Llama or Nova Micro; fallback-group uses Gemini and falls back to Claude Haiku only when Gemini fails.

Walkthrough

Prerequisites and .env
Bring up the stack
Model groups: load balancing and fallback
The LiteLLM UI: what you see and what you don't
Per-user consumption attribution
Consumption monitoring (by LLM and by user)
Shut down the environment

Appendix A: Security, keeping the gateway up to date

Scope. Everything runs locally with docker compose. This guide focuses on the docker compose cycle and on each gateway feature.

1. Prerequisites

Requirement	What for
Docker + Docker Compose	Orchestrate proxy, Postgres, and the 4 clients
Free port `4000`	The gateway uses it; check nothing else occupies it (`lsof -i :4000`)
Groq API key	Groq Llama 8B in `balanced-group` (free tier)
Gemini API key	Gemini 2.5 Flash, primary of `fallback-group` (free tier, 20 req/day)
AWS credentials	Bedrock models: Nova Micro (`balanced-group`) + Claude Haiku (`fallback-group` fallback)

`.env` file (repo root)

1
2
3
4
5
6
7
GROQ_API_KEY=...
GEMINI_API_KEY=...
LITELLM_MASTER_KEY=sk-master-local-dev
# Only if you'll use Bedrock models:
AWS_ACCESS_KEY_ID=...
AWS_SECRET_ACCESS_KEY=...
AWS_DEFAULT_REGION=...

Where each value comes from:

GROQ_API_KEY / GEMINI_API_KEY: generate them in each provider's console (Groq Console and Google AI Studio).
AWS_*: credentials for your AWS account (only if you'll use Bedrock). ⚠️ Below this block there's a security best-practices note on why these keys in .env are for local use only and what to use in production (IAM roles).
LITELLM_MASTER_KEY: you define it (no provider hands it to you). It's the proxy's admin key (it authenticates calls and the Admin UI). By LiteLLM convention it must start with the sk- prefix; the rest is free-form. sk-master-local-dev is just an example value for local use: in production use a long, random string and store it as a secret. This same value is what you'll use as Authorization: Bearer ... in this guide's curl commands and to log into the UI.

Compose reads these variables and injects them into the proxy container.

⚠️ Security: the AWS keys in .env are for this local demo only. Long-lived AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY are fine locally but not for production. There, use IAM roles for temporary, auto-rotated credentials: an EC2 instance profile, an ECS/Fargate task role, EKS IRSA / Pod Identity, or a Lambda execution role, per Security best practices in IAM . The AWS SDK (used by LiteLLM for Bedrock) then resolves credentials automatically via the default chain, so if you migrate to AWS, remove the keys from .env and docker-compose.yml.

⚠️ The IAM identity must be allowed to invoke each Bedrock model in config.yaml. The principal LiteLLM uses needs bedrock:InvokeModel (plus ...WithResponseStream for streaming) on every model the gateway can route to (here: Nova Micro and Claude Haiku). A missing model returns a 403 AccessDeniedException, which for balanced-group surfaces as a 500 (no fallback). Two Bedrock specifics: (1) cross-region inference profiles (e.g. us.amazon.nova-micro-v1:0) need two grants — the inference-profile ARN and the underlying foundation-model ARNs in every region it can route to; granting only the profile still 403s. (2) Each model must be enabled once on the Bedrock Model access page. See Identity-based policy examples and Prerequisites for inference profiles , and grant least privilege (only the models used, not bedrock:*).

2. Bring up the stack

The docker-compose.yml defines six pieces:

Service	Role
`litellm-db`	PostgreSQL 16, persistence of spend
`proxy`	LiteLLM Proxy (the gateway) on `:4000`
`client-a` / `client-b`	Traffic generators hitting `balanced-group`
`client-c` / `client-d`	Traffic generators hitting `fallback-group`

Project structure

Starting from an empty directory, you'll create these files (all the content is in this guide, there is nothing to clone). This is the final structure:

1
2
3
4
5
6
7
8
9
10
11
litellm-demo/
├── .env                        # keys and secrets (section 1)
├── docker-compose.yml          # stack orchestration
├── proxy/
│   ├── Dockerfile              # gateway image
│   └── config.yaml             # ⭐ model groups, fallbacks (the heart)
└── client/
    ├── Dockerfile.compose      # image of the traffic generators
    ├── requirements.txt        # client's Python dependencies
    ├── questions.txt           # 50 base questions (clients pick one at random)
    └── main.py                 # client logic (picks a question, calls its group)

Create the folders:

1
2
mkdir -p litellm-demo/proxy litellm-demo/client
cd litellm-demo

Next, the content of each file. Create them at the indicated paths.

The `docker-compose.yml` file

Create this file at the project root. It's the complete stack manifest:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
services:
  # --- PostgreSQL for LiteLLM (budget tracking) ---
  litellm-db:
    image: postgres:16-alpine
    environment:
      - POSTGRES_USER=litellm
      - POSTGRES_PASSWORD=litellm
      - POSTGRES_DB=litellm
    volumes:
      - litellm-pgdata:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U litellm"]
      interval: 5s
      timeout: 3s
      retries: 5

  # --- LiteLLM Proxy (API Gateway) ---
  proxy:
    build: ./proxy
    ports:
      - "4000:4000"
    environment:
      - GROQ_API_KEY=${GROQ_API_KEY}
      - GEMINI_API_KEY=${GEMINI_API_KEY}
      - AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID}
      - AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY}
      - AWS_SESSION_TOKEN=${AWS_SESSION_TOKEN}
      - AWS_DEFAULT_REGION=us-west-2
      - LITELLM_MASTER_KEY=${LITELLM_MASTER_KEY:-sk-master-local-dev}
      - DATABASE_URL=postgresql://litellm:litellm@litellm-db:5432/litellm
    depends_on:
      litellm-db:
        condition: service_healthy
    healthcheck:
      test: ["CMD", "python", "-c", "import urllib.request; urllib.request.urlopen('http://localhost:4000/health/liveliness')"]
      interval: 30s
      timeout: 10s
      retries: 5
      start_period: 40s

  # --- Client Tasks ---
  # client-a / client-b  -> balanced-group (Groq Llama 8B + Nova Micro, load balancing)
  # client-c / client-d  -> fallback-group (Gemini primary, Claude Haiku fallback)
  client-a:
    build:
      context: .
      dockerfile: client/Dockerfile.compose
    environment:
      - CLIENT_NAME=client-a
      - MODEL_GROUP=balanced-group
      - DURATION_MINUTES=${DURATION_MINUTES:-30}
      - PROXY_URL=http://proxy:4000
      - LITELLM_MASTER_KEY=${LITELLM_MASTER_KEY:-sk-master-local-dev}
      - INTERVAL_MIN=${INTERVAL_MIN:-10}
      - INTERVAL_MAX=${INTERVAL_MAX:-30}
    depends_on:
      proxy:
        condition: service_healthy

  client-b:
    build:
      context: .
      dockerfile: client/Dockerfile.compose
    environment:
      - CLIENT_NAME=client-b
      - MODEL_GROUP=balanced-group
      - DURATION_MINUTES=${DURATION_MINUTES:-30}
      - PROXY_URL=http://proxy:4000
      - LITELLM_MASTER_KEY=${LITELLM_MASTER_KEY:-sk-master-local-dev}
      - INTERVAL_MIN=${INTERVAL_MIN:-10}
      - INTERVAL_MAX=${INTERVAL_MAX:-30}
    depends_on:
      proxy:
        condition: service_healthy

  client-c:
    build:
      context: .
      dockerfile: client/Dockerfile.compose
    environment:
      - CLIENT_NAME=client-c
      - MODEL_GROUP=fallback-group
      - DURATION_MINUTES=${DURATION_MINUTES:-30}
      - PROXY_URL=http://proxy:4000
      - LITELLM_MASTER_KEY=${LITELLM_MASTER_KEY:-sk-master-local-dev}
      - INTERVAL_MIN=${INTERVAL_MIN:-10}
      - INTERVAL_MAX=${INTERVAL_MAX:-30}
    depends_on:
      proxy:
        condition: service_healthy

  client-d:
    build:
      context: .
      dockerfile: client/Dockerfile.compose
    environment:
      - CLIENT_NAME=client-d
      - MODEL_GROUP=fallback-group
      - DURATION_MINUTES=${DURATION_MINUTES:-30}
      - PROXY_URL=http://proxy:4000
      - LITELLM_MASTER_KEY=${LITELLM_MASTER_KEY:-sk-master-local-dev}
      - INTERVAL_MIN=${INTERVAL_MIN:-10}
      - INTERVAL_MAX=${INTERVAL_MAX:-30}
    depends_on:
      proxy:
        condition: service_healthy

volumes:
  litellm-pgdata:

Compose builds two images from custom Dockerfiles (proxy and the clients). You also need these two files:

proxy/Dockerfile, based on the official LiteLLM image and copies your config:

1
2
3
4
5
6
7
FROM ghcr.io/berriai/litellm:main-latest

COPY config.yaml /app/config.yaml

EXPOSE 4000

CMD ["--config", "/app/config.yaml", "--port", "4000", "--host", "0.0.0.0"]

client/Dockerfile.compose: the build context is the repo root; it copies the client/ folder (which includes questions.txt):

1
2
3
4
5
6
7
8
9
10
FROM python:3.13-slim

WORKDIR /app

COPY client/requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY client/ /app/client/

CMD python -m client.main ${CLIENT_NAME} ${DURATION_MINUTES}

`proxy/config.yaml`: the heart of the gateway

This file is mandatory (without it the proxy won't start) and defines the two model groups, the fallback chain, and settings. Its content is explained in section 3; here it is in full so the build works:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
# LiteLLM Proxy Configuration
# Docs: https://docs.litellm.ai/docs/proxy/configs

model_list:
  # --- balanced-group: load balancing across two providers (equal weights) ---
  # LiteLLM spreads traffic between Groq (Llama 8B) and Amazon Nova Micro using
  # the routing_strategy + weights. This is native gateway-side load balancing.
  - model_name: "balanced-group"
    litellm_params:
      model: "groq/llama-3.1-8b-instant"
      api_key: "os.environ/GROQ_API_KEY"
      weight: 1
  - model_name: "balanced-group"
    litellm_params:
      model: "bedrock/us.amazon.nova-micro-v1:0"
      aws_region_name: "us-west-2"
      weight: 1

  # --- fallback-group: primary + fallback across providers ---
  # Gemini is the primary; if it fails (quota/timeout/5xx), LiteLLM retries on
  # Claude Haiku (Bedrock). The fallback chain is defined under litellm_settings.
  - model_name: "fallback-group"
    litellm_params:
      model: "gemini/gemini-2.5-flash"
      api_key: "os.environ/GEMINI_API_KEY"
  - model_name: "fallback-group-fallback"
    litellm_params:
      model: "bedrock/global.anthropic.claude-haiku-4-5-20251001-v1:0"
      aws_region_name: "us-west-2"

litellm_settings:
  num_retries: 2
  request_timeout: 120
  routing_strategy: "simple-shuffle"
  fallbacks:
    - fallback-group: ["fallback-group-fallback"]
  # Note: no budget enforcement is applied (no blocking on spend).
  # Per-end-user consumption is recorded anyway (LiteLLM_DailyEndUserSpend /
  # LiteLLM_SpendLogs) for monitoring and cost attribution.

general_settings:
  master_key: "os.environ/LITELLM_MASTER_KEY"
  database_url: "os.environ/DATABASE_URL"

`client/requirements.txt`

1
2
3
openai
requests
python-dotenv

`client/questions.txt`: the base questions

A plain list of 50 questions, one per line. Each client picks one at random per request (no LLM involved in generating them). Here's an excerpt; use these or your own:

1
2
3
4
5
6
What is a container and how is it different from a virtual machine?
Explain the difference between TCP and UDP.
How does OAuth2 work at a high level?
Compare REST and GraphQL.
What is idempotency and why does it matter in APIs?
... (50 lines total)

`client/main.py`: the traffic generator

Each client reads questions.txt, picks a random question, and sends it to its assigned model group (set via the MODEL_GROUP env var in docker-compose.yml). It tags the request with its identity (x-litellm-end-user-id) for per-client monitoring:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
"""
Client Task. Traffic generator.
Picks a random question from questions.txt and sends it to its assigned model group
through the LiteLLM Proxy (OpenAI-compatible API).

Flow:
1. Pick a random question from questions.txt
2. Send it to the assigned model group (MODEL_GROUP) via the proxy
3. Wait a random interval and repeat

Each client is assigned ONE model group via the MODEL_GROUP env var:
- "balanced-group" : load balancing across Groq (Llama 8B) + Amazon Nova Micro
- "fallback-group" : Gemini primary with Claude Haiku (Bedrock) as fallback
"""

import sys
import os
import time
import random

from openai import OpenAI

# --- Configuration ---

PROXY_URL = os.getenv("PROXY_URL", "http://localhost:4000")
LITELLM_MASTER_KEY = os.getenv("LITELLM_MASTER_KEY", "sk-master-local-dev")
MODEL_GROUP = os.getenv("MODEL_GROUP", "balanced-group")

QUESTIONS_FILE = os.path.join(os.path.dirname(os.path.abspath(__file__)), "questions.txt")

def load_questions() -> list:
    """Loads the base questions from questions.txt (one per line)."""
    with open(QUESTIONS_FILE, encoding="utf-8") as f:
        return [line.strip() for line in f if line.strip()]

def create_client(api_key: str) -> OpenAI:
    """Creates an OpenAI client pointing at the proxy."""
    return OpenAI(base_url=f"{PROXY_URL}/v1", api_key=api_key)

def execute_query(client: OpenAI, prompt: str, model_group: str, user_id: str) -> dict:
    """Sends the query to the proxy against the assigned model group."""
    start = time.time()
    try:
        response = client.chat.completions.create(
            model=model_group,
            messages=[{"role": "user", "content": prompt}],
            user=user_id,
            extra_headers={"x-litellm-end-user-id": user_id},
            extra_body={"metadata": {"trace_user_id": user_id}},
        )
        latency = (time.time() - start) * 1000
        return {
            "success": True,
            "model": response.model,
            "tokens": response.usage.total_tokens,
            "latency_ms": latency,
        }
    except Exception as e:
        return {"success": False, "error": str(e)[:150]}

def run_client(client_name: str, model_group: str, duration_minutes: float = 10,
               interval_min: int = 10, interval_max: int = 30):
    """Main loop of the client task."""
    user_id = client_name  # propagated via x-litellm-end-user-id for per-client monitoring
    client = create_client(api_key=LITELLM_MASTER_KEY)
    questions = load_questions()
    print(f"[{client_name}] start | group={model_group} | questions={len(questions)} | {duration_minutes} min")

    start_time = time.time()
    end_time = start_time + (duration_minutes * 60)
    query_count = 0
    total_tokens = 0
    errors = 0

    while time.time() < end_time:
        query_count += 1
        prompt = random.choice(questions)

        result = execute_query(client, prompt, model_group, user_id)
        if result["success"]:
            total_tokens += result["tokens"]
            print(f"[{client_name}] {result['model']} | {result['tokens']} tok | {result['latency_ms']:.0f}ms")
        else:
            errors += 1
            print(f"[{client_name}] error: {result['error'][:60]}")

        wait = random.uniform(interval_min, interval_max)
        if time.time() + wait > end_time:
            break
        time.sleep(wait)

    print(f"[{client_name}] done | queries={query_count} | tokens={total_tokens} | errors={errors}")

if __name__ == "__main__":
    client_name = sys.argv[1] if len(sys.argv) > 1 else os.getenv("CLIENT_NAME", "client-a")
    duration = float(sys.argv[2]) if len(sys.argv) > 2 else float(os.getenv("DURATION_MINUTES", "10"))

    interval_min = int(os.getenv("INTERVAL_MIN", "10"))
    interval_max = int(os.getenv("INTERVAL_MAX", "30"))

    run_client(client_name, MODEL_GROUP, duration_minutes=duration,
               interval_min=interval_min, interval_max=interval_max)

With the files created (.env, docker-compose.yml, proxy/Dockerfile, proxy/config.yaml, client/Dockerfile.compose, client/requirements.txt, client/questions.txt, client/main.py) you can now bring up the stack.

Bring up the full stack

This builds the images, starts the proxy + Postgres, and launches the 4 clients, which generate traffic for 30 minutes:

1
DURATION_MINUTES=30 docker compose up --build

The proxy takes ~40s to become healthy (the healthcheck uses /health/liveliness) before the clients start. You can verify it from another terminal:

1
2
curl -s http://localhost:4000/health/liveliness    # -> "I'm alive!"
docker compose ps                                   # proxy + db in healthy state

Useful variables to modulate the traffic:

Variable	Default	Effect
`DURATION_MINUTES`	30	How long the clients generate queries
`INTERVAL_MIN` / `INTERVAL_MAX`	10 / 30	Seconds to wait between queries (raise both to spread spend over time)

3. Model groups: load balancing and fallback

Everything lives in proxy/config.yaml. Instead of exposing individual models, this demo defines two model groups: a single logical name per group that clients call. The gateway does the rest. This is the core of the demo: both groups are native LiteLLM capabilities.

3.1 `balanced-group`: load balancing

Two deployments share the same model_name, so LiteLLM spreads traffic across both providers according to routing_strategy and the weights (here, equal → ~50/50):

1
2
3
4
5
6
7
8
9
10
  - model_name: "balanced-group"
    litellm_params:
      model: "groq/llama-3.1-8b-instant"     # Groq
      api_key: "os.environ/GROQ_API_KEY"
      weight: 1
  - model_name: "balanced-group"
    litellm_params:
      model: "bedrock/us.amazon.nova-micro-v1:0"   # Amazon Nova Micro (Bedrock)
      aws_region_name: "us-west-2"
      weight: 1

client-a and client-b call balanced-group. Each request lands on Groq or Nova; the client never knows which. Useful to distribute load, dodge a single provider's rate limits, or A/B between models. The split is controlled entirely by weight (set 3/1 for ~75/25, etc.).

See which deployment answered. The response body's model field shows the logical name (balanced-group), not the real one. The real deployment is in the response header x-litellm-model-id (and x-litellm-model-api-base appears when Groq answers). You'll also see the real model per request in the UI Logs tab (section 4).

3.2 `fallback-group`: fallback across providers

Gemini is the primary; if it fails (quota, timeout, 5xx), LiteLLM transparently retries on Claude Haiku (Bedrock). The fallback target is a separate model_name, wired under litellm_settings.fallbacks:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
  - model_name: "fallback-group"
    litellm_params:
      model: "gemini/gemini-2.5-flash"               # primary
      api_key: "os.environ/GEMINI_API_KEY"
  - model_name: "fallback-group-fallback"
    litellm_params:
      model: "bedrock/global.anthropic.claude-haiku-4-5-20251001-v1:0"   # fallback
      aws_region_name: "us-west-2"

litellm_settings:
  num_retries: 2
  routing_strategy: "simple-shuffle"
  fallbacks:
    - fallback-group: ["fallback-group-fallback"]

client-c and client-d call fallback-group. The client always asks for the same logical name and always gets a response, even when Gemini is down. This is especially easy to observe because Gemini's free tier is limited to ~20 requests/day: once exhausted, Gemini returns 429 and every request after that is served by Haiku, with no client-side change.

3.3 Balancing strategies (reference)

routing_strategy controls how LiteLLM picks among the deployments of a group. This demo uses simple-shuffle (weighted random). Other native options:

Strategy	How it chooses
`simple-shuffle` (default)	Weighted random by `weight` (or `rpm`/`tpm`), minimal overhead
`latency-based-routing`	The deployment with the lowest recent latency
`usage-based-routing-v2`	The one with most available capacity (rate-limit aware)
`least-busy`	The deployment with fewest in-flight requests
`cost-based-routing`	The lowest cost-per-token among candidates
`mirror_traffic` (shadow testing)	Mirrors a % of live traffic to a second deployment in the background, without affecting the client's response — to test a new model against real traffic before giving it any real share

The full reference (strategies, weights, per-group routing, traffic mirroring) is in the official docs: Router - Load Balancing .

4. The LiteLLM UI: what you see and what you don't

LiteLLM ships its own Admin UI at localhost:4000/ui. How to log in:

If you didn't define UI_USERNAME/UI_PASSWORD (this demo's case), the default login is:
- Username: admin
- Password: your master key (sk-master-local-dev)
If you prefer your own credentials, add them to the proxy service in docker-compose.yml (UI_USERNAME and UI_PASSWORD) and reload the proxy; then you use those values.
The UI lives in the same process as the proxy: there's nothing extra to bring up.

4.1 Why you do NOT see here what you configured in the files

This is the most common question when opening the UI for the first time. On the Models + Endpoints page you'll see your two model groups (balanced-group, fallback-group), but as read-only: you can't edit or delete them from the UI, and the fallback chain / routing_strategy you put in config.yaml don't appear as manageable items.

The reason: this demo defines everything in config.yaml and does not enable store_model_in_db. The UI can only manage (create/edit/delete) the objects that live in the database; what comes from the YAML it shows for reference, marked as db_model=false. This is the expected behavior of a config-as-code setup:

Configuration is the source of truth and lives in git. The UI is a read-only window onto that config + a monitoring console over consumption. It's not where you edit the gateway in this demo (that's done in the YAML and the proxy is reloaded).

If you wanted to manage models from the UI, you'd have to enable store_model_in_db: true and accept the dual source of truth (store_model_in_db docs ).

4.2 What the UI IS good for in this demo

Configuration (models, fallbacks, routing): shown read-only; edited in the YAML. By design (config-as-code).
Consumption by user and by model: seen and explored natively on the Usage page; this is the monitoring of section 6.
For detailed traces, routing debugging, and quality evaluation, you can plug in an external observability tool such as Langfuse via LiteLLM callbacks (optional; out of scope for this demo).

4.3 Send an ad-hoc request and inspect it in the UI

Beyond the autonomous clients, you can hit the gateway yourself with a single curl and then see exactly how it was handled. The endpoint is OpenAI-compatible (/v1/chat/completions), and you ask for a model group (balanced-group or fallback-group), and the gateway resolves it to a real model.

1
2
3
4
5
6
7
8
9
# balanced-group -> load-balanced across Groq (Llama 8B) and Nova Micro
curl -s http://localhost:4000/v1/chat/completions \
  -H "Authorization: Bearer sk-master-local-dev" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "balanced-group",
    "user": "manual-test",
    "messages": [{"role": "user", "content": "What is an API gateway?"}]
  }' | python -m json.tool

See it in the UI. Open localhost:4000/ui → Logs tab. Each request you sent appears as a row showing the real model that answered (e.g. groq/llama-3.1-8b-instant or bedrock/...nova-micro... for balanced-group), tokens, cost, latency, and status. This is how you confirm, per request, which model resolved it and what it cost, natively, without Langfuse.

5. Per-user consumption attribution

Before looking at the monitoring, it's worth understanding how the gateway knows which client to attribute each request to. The goal here is not to block on spend, but to attribute consumption to each client (client-a/b/c/d) for reporting and chargeback.

Requirement: the compose's Postgres (DATABASE_URL, already configured in the proxy service) persists the spend. Without it, consumption is lost when the proxy restarts.

The client sends its identity on every request (client/main.py, execute_query):

1
2
3
4
5
6
7
response = client.chat.completions.create(
    model=model,
    messages=[{"role": "user", "content": prompt}],
    user=user_id,                                       # standard OpenAI field
    extra_headers={"x-litellm-end-user-id": user_id},   # end-user identity
    extra_body={"metadata": {"trace_user_id": user_id}},
)

The proxy resolves the end-user by priority:

Priority	Method	Example
1	Header `x-litellm-end-user-id`	`-H 'x-litellm-end-user-id: client-a'`
2	`user` field in the body	`"user": "client-a"`
3	`metadata.user_id`	`"metadata": {"user_id": "client-a"}`

No code to touch: attribution works with the local-mode master key. That client-a/b/c/d is what you'll see in the section 6 monitoring.

Attribution vs. enforcement. LiteLLM can assign budgets and enforce them (block requests once a key, team, or end-user exceeds a spend limit). It's a native capability. This demo intentionally doesn't enable it: it only measures and attributes consumption per client, without blocking anyone. If you wanted hard limits, you'd configure budgets in LiteLLM; see Budgets & Rate Limits .

6. Consumption monitoring

All monitoring is done in the LiteLLM UI (section 4), with no commands. Open localhost:4000/ui and go to the Usage page (in the left menu, or directly at localhost:4000/ui/?login=success&page=new_usage).

6.1 Consumption by user (`client-a/b/c/d`)

On the Usage page, open the "Usage View" selector (top) and choose "Customer Usage".
In "Filter by customer", select client-a, client-b, client-c, client-d (or leave it empty to see all).
You'll see the "Spend Per Customer" widget: one row per client with its spend, successful/failed requests, and tokens. That's the chargeback view: how much each client spent. Since client-a/b hit balanced-group and client-c/d hit fallback-group, you can compare the spend profile of each capability side by side.

⚠️ Don't confuse it with "Spend Per User". There's another widget called Spend Per User (in the Global Usage view) that shows the internal user of the API key (in this demo it shows as default_user_id), not your clients. Your client-a/b/c/d are end-users/customers, and only appear in Customer Usage → Spend Per Customer.

⚠️ If the view shows $0, check the date range. The "Select Time Range" selector filters by day, but LiteLLM groups spend by UTC date. If you work in a timezone behind UTC (e.g. GMT-6) and generated traffic in the afternoon/evening, that consumption falls into the next UTC day's bucket and a range "up to today" leaves it out. Fix: extend the End date one day forward. (Note: LiteLLM's timezone only changes budget resets, not this grouping, so it doesn't solve this case.)

6.2 Consumption by model / LLM

On the same Usage page:

The "Top Models" widget (or the "Model Activity" tab) shows spend and tokens per model (gemini/gemini-2.5-flash, groq/llama-3.1-8b-instant, Nova Micro, Claude Haiku). Answers "which model is costing me the most?".
For balanced-group, you should see traffic split across Groq Llama 8B and Nova Micro (the load balancing). For fallback-group, most traffic on Gemini, and Claude Haiku picking up whenever Gemini hits its free-tier limit (the fallback in action).

6.3 Other cuts

Tag Activity / Endpoint Activity (tabs in the same view): consumption by tag or by endpoint, for additional attribution.

For this demo's monitoring goal (consumption by user and by model), the LiteLLM UI is sufficient on its own. For detailed traces and quality evaluation, plug in an external observability tool such as Langfuse via LiteLLM callbacks.

7. Shut down the environment

1
2
3
4
5
# Stop containers, keeping the consumption history (spend)
docker compose down

# Also delete the Postgres volume (resets the consumption history)
docker compose down -v

Walkthrough summary

Capability	Where it's demonstrated
Multi-provider model groups	`proxy/config.yaml` → `model_list`
Load balancing (`balanced-group`)	`proxy/config.yaml` → two deployments + `weight` + `routing_strategy`
Fallback (`fallback-group`)	`proxy/config.yaml` → `litellm_settings.fallbacks`
LiteLLM UI (read-only config + monitoring)	`localhost:4000/ui`
Per-user consumption attribution	`x-litellm-end-user-id` header (`client/main.py`)
Consumption monitoring (user and model)	LiteLLM UI → Usage page
Advanced observability (traces, eval)	External tool (e.g. Langfuse) via LiteLLM callbacks (optional)

Appendix A: Security, keeping the gateway up to date

LiteLLM is a fast-moving open-source project and, like any dependency, has had security fixes over time (standard supply-chain hygiene, nothing specific to AWS or Bedrock). Before exposing this gateway beyond your machine:

Pin an explicit version, not :main-latest in proxy/Dockerfile, and upgrade deliberately after reviewing the project's release notes and security advisories.
Keep the proxy local while you can. Don't expose port 4000 to a network. If you later deploy it (ECS/Fargate, EKS, etc.), put it behind proper authentication and network controls, and apply least privilege on the API keys.

Track advisories and fixed versions via the project's GitHub releases and security advisories pages.

Closing thoughts

The whole point of this exercise fits in one sentence: the model is a configuration detail, not a client dependency. Once the gateway sits in front of your traffic, clients only ask for a logical capability (balanced-group, fallback-group) and never learn which provider answered, so adding a provider, rebalancing, or removing a model is a model_list edit with no client changes; fallbacks and weighted routing live at the edge and every client inherits them (the fallback-group keeps answering through Claude Haiku when Gemini hits its free-tier limit); and per-client and per-model spend are observable natively in the LiteLLM UI. From here the natural next steps are to pin a specific LiteLLM version, move secrets to IAM roles when you deploy beyond your laptop, and explore traffic mirroring to evaluate a new model against real traffic. If you already consume models directly from one or more providers, a gateway is a low-friction way to add Amazon Bedrock to the mix and test fallback and load balancing without touching application code.

Note: this is a personal proof of concept shared for learning purposes. Validate any configuration against the official documentation before using it in production.

# ai # amazon-bedrock # ai-for-developers # llm

Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.

Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article

Builder Center

Build a local LiteLLM AI gateway to test model routing and cost attribution

Gilberto Canales

Why an LLM gateway matters

What this test is and what it demonstrates

Architecture at a glance

Walkthrough

1. Prerequisites

.env file (repo root)

2. Bring up the stack

Project structure

The docker-compose.yml file

proxy/config.yaml: the heart of the gateway

client/requirements.txt

client/questions.txt: the base questions

client/main.py: the traffic generator

Bring up the full stack

3. Model groups: load balancing and fallback

3.1 balanced-group: load balancing

3.2 fallback-group: fallback across providers

3.3 Balancing strategies (reference)

4. The LiteLLM UI: what you see and what you don't

4.1 Why you do NOT see here what you configured in the files

4.2 What the UI IS good for in this demo

4.3 Send an ad-hoc request and inspect it in the UI

5. Per-user consumption attribution

6. Consumption monitoring

6.1 Consumption by user (client-a/b/c/d)

6.2 Consumption by model / LLM

6.3 Other cuts

7. Shut down the environment

Walkthrough summary

Appendix A: Security, keeping the gateway up to date

Closing thoughts

`.env` file (repo root)

The `docker-compose.yml` file

`proxy/config.yaml`: the heart of the gateway

`client/requirements.txt`

`client/questions.txt`: the base questions

`client/main.py`: the traffic generator

3.1 `balanced-group`: load balancing

3.2 `fallback-group`: fallback across providers

6.1 Consumption by user (`client-a/b/c/d`)