Token Consumption Monitoring for LLM Workloads on Amazon EKS
Four approaches to monitoring LLM token consumption on Amazon EKS — from Bedrock IAM attribution to distributed agent tracing with the OTel Operator.
Intro
As LLM-powered agents move from prototypes to production on Amazon EKS, teams face a new cost management challenge: token consumption is unpredictable, varies 100x between requests, and is invisible to traditional monitoring.
You have agents, chatbots, RAG pipelines, or direct model calls running on EKS. You want to know how many tokens each request consumes, which teams or services drive the most spend, and where tokens are wasted. You don't want to modify application code to get this visibility.
This post covers four approaches to token monitoring on EKS — ranging from zero-touch billing attribution to deep agent-level tracing — with minimal or no application code changes. Approaches 1, 2, and 4 require no image modifications; Approach 3 requires the telemetry library to be pre-installed in the container image (an image dependency, not a code change).
The Problem
LLM workloads have a unique observability gap. Traditional metrics (CPU, memory, request count) don't capture the primary cost driver — token consumption. A single API call can cost anywhere from $0.001 to $0.50 depending on context length, and an agent loop might make 5-20 such calls per user request.
Without token-level visibility, cost attribution happens at the billing cycle level — after the spend has already occurred.
What we want to capture (per LLM call):
- Input tokens, output tokens
- Model used
- Latency (time to first token, total duration)
- Which service/team/user initiated the call
- For agents: which reasoning step or tool call triggered it
Architecture Overview
Architecture Overview — Token monitoring approaches on EKS
The diagram shows the target state: Agent pods emit telemetry via StrandsTelemetry or OpenLLMetry , LiteLLM Proxy adds cost tracking, ADOT Collector ships everything to CloudWatch GenAI Observability . Bedrock IAM attribution feeds billing separately. The rest of this post covers how to get there.
How a Request Flows Through the System
Sequence diagram — request flow through agent, proxy, and collector
Each user request triggers a reasoning loop: the agent calls the LLM (via proxy), processes the response, optionally calls tools, and repeats. Telemetry is exported in parallel — agent spans and proxy cost spans flow to ADOT, then to CloudWatch. Bedrock IAM attribution reaches billing automatically without any instrumentation.
Approach 1: LLM Gateway Proxy
How it works: Route all LLM API traffic through a proxy that logs token usage transparently. The application doesn't know it's being monitored.
Implementation: Deploy LiteLLM Proxy (or a similar LLM gateway) as a Kubernetes Deployment. Point your workloads to the proxy endpoint instead of calling the LLM provider directly. The key requirement for any gateway is OTLP export of token metrics — other options include custom API Gateway configurations or service mesh sidecars.
1
2
3
4
5
6
7
8
# Agent pod env (the only "change" — an endpoint override)
# Exact variable depends on your SDK/framework:
# boto3: AWS_ENDPOINT_URL_BEDROCK_RUNTIME
# OpenAI-compatible clients: OPENAI_BASE_URL
# LiteLLM SDK: LITELLM_PROXY_BASE_URL
env:
- name: AWS_ENDPOINT_URL_BEDROCK_RUNTIME
value: "http://litellm-proxy:4000"
Note: This env var redirects all bedrock-runtime API calls (InvokeModel, Converse). If your pod also calls Bedrock Knowledge Bases or Bedrock Agents APIs, use SDK-levelendpoint_urlconfig on individual clients instead of the blanket env var.
What the proxy captures:
- Token counts (input/output) from every response
- Cost per request (auto-calculated from model pricing)
- Per-team attribution via virtual API keys
- Latency, errors, rate limits
- Full OTel traces (with
LITELLM_OTEL_V2=true)
Per-team budgets:
1
2
3
4
5
6
7
8
9
10
11
# LiteLLM config
general_settings:
max_budget: 500.0
budget_duration: "monthly"
# Per-team limits via virtual keys
team_settings:
- team_id: "research-agents"
max_budget: 200.0
- team_id: "customer-support"
max_budget: 100.0
When budget is exceeded → 429 response → the client fails immediately instead of accumulating debt.
Pros:
- Framework-agnostic (works with any LLM client — Boto3, OpenAI SDK, LangChain, Strands, raw HTTP)
- Zero code changes (just endpoint env var)
- Real-time budget enforcement
- Multi-provider support (Bedrock + OpenAI + Anthropic behind one endpoint)
Limitations:
- Additional network hop (added latency depends on cluster network; typically low single-digit ms)
- Infrastructure to maintain (deployment, HA, monitoring)
- Doesn't see application internals (agent reasoning, tool calls)
When to use: You want cost control and attribution across any LLM workloads regardless of framework.
Security: CVE-2026-42271 — command injection, CVSS 8.7. Affected versions: 1.74.2–1.83.6. Pin your deployment to ≥1.83.7.
Approach 2: OTel Auto-Instrumentation (Operator)
How it works: The OpenTelemetry Operator injects instrumentation libraries into pods via mutating webhook. For Python workloads, it adds packages that monkey-patch LLM SDK calls to emit spans with token counts.
Implementation:
- Install the ADOT EKS Add-On (which includes the OTel Operator)
- Create an Instrumentation CRD that includes GenAI instrumentation packages
- Annotate workload pods
1
2
3
4
5
6
7
8
9
10
11
12
apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
name: genai-instrumentation
spec:
exporter:
endpoint: http://adot-collector:4317
propagators:
- tracecontext
- baggage
python:
image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-python:latest
Annotate pods:
1
2
3
metadata:
annotations:
instrumentation.opentelemetry.io/inject-python: "genai-instrumentation"
Example: Distributed trace in Jaeger (agent + proxy in a single trace, 2 tool calls):
1
2
3
4
5
6
7
8
9
10
POST /invoke (17.2s) [agent]
├── invoke_agent Strands Agents [in:2675, out:875] [agent]
│ ├── chat (1.7s) [in:659, out:80] [agent]
│ │ └── Received Proxy Server Request [in:659, out:80] [litellm-proxy]
│ ├── execute_tool search_web (2ms) [agent]
│ ├── chat (9.2s) [in:779, out:418] [agent]
│ │ └── Received Proxy Server Request [in:779, out:418] [litellm-proxy]
│ ├── execute_tool summarize (3ms) [agent]
│ └── chat (6.3s) [in:1237, out:377] [agent]
│ └── Received Proxy Server Request [in:1237, out:377] [litellm-proxy]
The Operator injects HTTPX instrumentation that propagates
traceparent headers on every outgoing call — linking agent reasoning spans with proxy cost spans automatically. No code changes required.
Distributed trace in Jaeger — agent and LiteLLM proxy spans in a single trace
What the instrumentation captures depends on installed packages:
| Package | What it intercepts | Tokens visible |
|---|---|---|
| opentelemetry-instrumentation-openai | OpenAI SDK calls | ✅ input/output tokens |
| opentelemetry-instrumentation-bedrock | boto3 bedrock-runtime | ✅ input/output tokens |
| opentelemetry-instrumentation-langchain | LangChain chains/agents | ✅ + chain steps |
| Base (no GenAI packages — default) | HTTP, DB, gRPC only | ❌ no token visibility |
Critical point: The OTel Operator is a delivery mechanism. Without GenAI-specific instrumentation packages in the init container image, it only produces HTTP-level spans — no tokens, no model info.
Pros:
- Standard OTel ecosystem (vendor-neutral backends)
- Sees inside the application (per-SDK-call granularity)
- Declarative (Kubernetes-native CRDs)
- No proxy infrastructure
- Frameworks that use the global TracerProvider (e.g., Strands Agents) emit full agent spans automatically — no activation needed
- HTTPX instrumentation enables distributed trace linking between agent and LLM proxy
Limitations:
- Requires GenAI packages in the auto-instrumentation image (may need custom image)
- Framework-specific (different packages for OpenAI vs Bedrock vs LangChain)
- Doesn't provide budget enforcement
- Experimental status for most GenAI instrumentations
When to use: You want distributed traces spanning agent and proxy with zero code changes, or you're already using OTel Operator for general APM.
Approach 3: Framework-Native Telemetry
How it works: Some agent frameworks (Strands Agents, LangChain) and libraries (OpenLLMetry) have built-in OTel support that emits rich spans — not just token counts, but agent reasoning cycles, tool executions, and memory operations. Enable them in your workloads to get the deepest level of visibility.
What's available:
| Library | Covers | What it emits |
|---|---|---|
| StrandsTelemetry (strands-agents[otel]) | Strands Agents | Agent invocation, reasoning cycles, tool calls, per-call tokens |
| OpenLLMetry (traceloop-sdk) | OpenAI, Anthropic, Bedrock, LangChain, LlamaIndex, CrewAI, 20+ others | LLM calls with tokens, vector DB queries, chain steps |
| opentelemetry-instrumentation-langchain | LangChain/LangGraph | Chain runs, retriever calls, LLM calls |
Example: Strands Agent trace in Jaeger (2-tool-call query, zero code changes):
1
2
3
4
5
6
7
8
9
invoke_agent Strands Agents (14.7s) [total: 3,227 tokens]
├── execute_event_loop_cycle (2.9s)
│ └── chat (2.9s) [in:661, out:76] → decides to search
├── execute_tool search_web (1ms)
├── execute_event_loop_cycle (7.6s)
│ └── chat (7.6s) [in:777, out:309] → summarizes results
├── execute_tool summarize (2ms)
├── execute_event_loop_cycle (4.3s)
│ └── chat (4.3s) [in:1127, out:277] → final response
Each
chat span carries gen_ai.usage.input_tokens and gen_ai.usage.output_tokens. The token accumulation per cycle (661 → 777 → 1127) shows context growing as the agent reasons — this is exactly the visibility needed to identify token-wasteful patterns.Activation options (pick one):
| Method | Code changes | Best for |
|---|---|---|
| Add 2 lines to your entrypoint | Minimal (2 lines) | Single app, you own the code |
| OTel Operator Instrumentation CRD | None (annotation) | If GenAI packages are in the init image |
| sitecustomize.py via ConfigMap | None (mount + env var) | Multiple apps, don't control source code |
| Kyverno policy | None (label-based) | At scale, dozens of agent deployments |
Example: ConfigMap-based activation (no code changes):
Python automatically executes any
sitecustomize.py found on PYTHONPATH at interpreter startup — before your application code runs. By mounting one via ConfigMap, we inject telemetry initialization into the container without modifying it. Ensure your base image doesn't ship its own sitecustomize.py, or import the original at the top of yours to chain them.1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
apiVersion: v1
kind: ConfigMap
metadata:
name: agent-otel-init
data:
sitecustomize.py: |
import os
if os.environ.get("AGENT_OTEL_ENABLED") != "true":
pass
else:
try:
from strands.telemetry.config import StrandsTelemetry
StrandsTelemetry().setup_otlp_exporter()
except ImportError:
try:
from traceloop.sdk import Traceloop
Traceloop.init(disable_batch=False)
except ImportError:
pass
Add to Deployment:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
env:
- name: AGENT_OTEL_ENABLED
value: "true"
- name: PYTHONPATH
value: "/opt/otel-init"
- name: OTEL_EXPORTER_OTLP_ENDPOINT
value: "http://adot-collector:4317"
volumeMounts:
- name: otel-init
mountPath: /opt/otel-init
volumes:
- name: otel-init
configMap:
name: agent-otel-init
Pros:
- Deepest visibility (agent internals, not just LLM calls)
- Multiple activation methods — from 2 lines of code to fully declarative
- Toggleable per-environment via env var
Limitations:
- Python only
- Requires instrumentation library pre-installed in container image (
strands-agents[otel]ortraceloop-sdk) - Framework-specific (different libraries for different frameworks)
Scaling up: For organizations with dozens of agent deployments, automate the ConfigMap mount and env var injection with a Kyverno ClusterPolicy that targets all pods with a specific label (e.g.,
app.kubernetes.io/type: llm-agent). This eliminates per-Deployment YAML edits.When to use: You want full reasoning traces, not just token counts. Works best when combined with Approach 1 (gateway) for cost attribution.
Approach 4: Bedrock-Native Cost Attribution (No Instrumentation)
How it works: If all your LLM calls go to Amazon Bedrock, AWS provides cost attribution at the billing level — no proxy, no OTel, no instrumentation.
What you get:
- IAM Principal Attribution — every InvokeModel/Converse call tagged with caller identity (user/role) automatically
- Application Inference Profiles — attach team/project/environment tags to inference endpoints
- CloudWatch metrics —
InputTokenCount,OutputTokenCountper model (1-minute resolution) - CountTokens API — estimate token usage before sending requests (free, no charges)
Setup: Tag your IAM roles with cost allocation tags (team, project). Enable in Billing Console. Optionally, use CountTokens in your application to estimate costs before sending requests.
Pros:
- Zero infrastructure (no proxy, no collector, no ConfigMaps)
- Works with any language/framework calling Bedrock
- CloudWatch metrics provide per-model token counts at 1-minute resolution
- CountTokens API is free — enables client-side cost estimation
Limitations:
- Scope: Bedrock API calls only
- Attribution granularity: IAM role / inference profile
- Budget control: CloudWatch Alarms (reactive, not blocking)
- Visibility scope: token counts and cost (for agent-level tracing, see Amazon Bedrock AgentCore )
When to use: Bedrock-only workloads where you need cost breakdown by team and per-model usage metrics without additional infrastructure.
Comparison
| Gateway (LiteLLM) | OTel Operator | Framework Telemetry | Bedrock Native | |
|---|---|---|---|---|
| Setup | Env var only | Annotation only | Env var + ConfigMap | None |
| Token granularity | Per-request | Per-call + agent steps | Per-call + agent steps | Aggregate (per-model, 1-min) |
| Cost attribution | Per-team, real-time | Per-service | Per-service | Per-IAM-role (billing) |
| Budget enforcement | Yes (hard 429) | No | No | Alerts only |
| Agent reasoning visibility | No | Yes (if framework uses global TracerProvider) | Yes | No |
| Distributed tracing | Proxy spans only | Automatic (HTTPX propagation) | Separate traces per service | N/A |
| Multi-provider | Yes | Yes | Yes | Bedrock only |
| Infrastructure | Proxy deployment | OTel Operator | ConfigMap | None |
| Language support | Any | Python, Node.js, Java | Python | Any |
| Image modification | None | None | Library pre-installed | None |
Combining Approaches
These aren't mutually exclusive. A production setup might combine:
- Bedrock IAM attribution for cost reporting by team or role
- LiteLLM Proxy for real-time budget enforcement and per-request cost tracking
- OTel Operator for distributed traces spanning agent and proxy
- Framework telemetry for agent reasoning visibility without the Operator dependency
With Approach 2 (OTel Operator), agent and proxy spans merge into a single distributed trace automatically — see the example in Approach 2 above. Without the Operator (Approach 3 alone), traces remain separate per-service. In that case, use
model_kwargs for per-agent attribution:1
2
3
4
5
6
7
8
model = LiteLLMModel(
model_id="...",
client_args={"api_base": "http://litellm:4000", "api_key": os.environ["LITELLM_API_KEY"]},
model_kwargs={
"user": "research-bot",
"metadata": {"team": "research", "environment": "production"},
},
)
LiteLLM records these on every span as
llm.user and metadata.requester_metadata — filterable and groupable in any OTel backend without distributed trace linking.Observability Backend
All approaches export via OpenTelemetry Protocol (OTLP). On EKS, deploy the ADOT Collector (EKS Add-On) and route to your backend:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
apiVersion: opentelemetry.io/v1alpha1
kind: OpenTelemetryCollector
metadata:
name: adot-collector
spec:
mode: deployment
config:
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
exporters:
awsxray: {}
awsemf:
namespace: GenAI/Agents
service:
pipelines:
traces:
receivers: [otlp]
exporters: [awsxray]
metrics:
receivers: [otlp]
exporters: [awsemf]
Supported backends:
| Backend | Managed | Self-hosted |
|---|---|---|
| CloudWatch GenAI Observability | ✅ | — |
| Amazon Managed Prometheus + Grafana | ✅ | — |
| Langfuse | Cloud option | ✅ (MIT) |
| Arize Phoenix | Cloud option | ✅ |
Summary
| If you need... | Use... |
|---|---|
| Cost breakdown per team (Bedrock only) | Bedrock IAM Attribution |
| Real-time per-request cost + budget enforcement | LiteLLM Gateway |
| Token counts without proxy overhead | OTel Operator with GenAI packages |
| Full agent reasoning traces | Framework telemetry (StrandsTelemetry / OpenLLMetry) |
| Distributed traces spanning agent + proxy | OTel Operator (auto-injects HTTPX propagation) |
| All of the above | Combine them |
No approach requires modifying application source code. Approaches 1, 2, and 4 need no image changes; Approach 3 assumes the telemetry library is already in the image. Start with the one that matches your immediate need, add layers as visibility requirements grow.
Companion code: All approaches are implemented in a demo — llm-token-monitoring-eks on GitHub .
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article