Building LLM Observability on Amazon EKS with Langfuse, AWS Bedrock, and Weaviate

Learn how to build an LLM observability platform on Amazon EKS using Langfuse, AWS Bedrock, Weaviate, PostgreSQL, Kong, KEDA, and Terraform

Yevgeni Shapiro

Head Of Cloud | Driving Cloud-Native Innovation | Platform Engineering

Published Oct 1, 2026

Last modified Oct 1, 2026

Generative AI applications are moving from experimentation into production, but operating an LLM-powered application introduces a new set of challenges.

Traditional application monitoring can tell us whether an API is available, how much CPU a Kubernetes pod is consuming, or whether a database is responding. It does not necessarily explain why an LLM generated a particular answer, which documents were retrieved, how many tokens were consumed, or where latency was introduced in a RAG pipeline.

This is where LLM observability becomes important.

In this article, we explore Langfuse, an open-source LLM observability platform, and demonstrate how it can fit into an AWS architecture based on:

Amazon EKS
AWS Bedrock
Weaviate
PostgreSQL
Amazon S3
Kong API Gateway
KEDA
Prometheus, Grafana, and Loki
Terraform and Helm

The goal is to build an observable, scalable RAG platform where infrastructure telemetry and LLM-specific telemetry complement each other.

Architecture

The architecture separates infrastructure observability from LLM observability.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
                           Users
                             |
                             v
                    +----------------+
                    |  Kong Gateway  |
                    +-------+--------+
                            |
                            v
                    +----------------+
                    |   RAG API      |
                    |                |
                    | Langfuse SDK   |
                    +-------+--------+
                            |
              +-------------+-------------+
              |                           |
              v                           v
       +-------------+              +-------------+
       |  Weaviate   |              | AWS Bedrock|
       | Vector DB   |              |    LLM     |
       +-------------+              +-------------+
              |                           |
              +-------------+-------------+
                            |
                            v
                    +---------------+
                    |    Langfuse   |
                    |               |
                    | Traces        |
                    | Prompts       |
                    | Evaluations   |
                    | Costs         |
                    +-------+-------+
                            |
                            v
                    +---------------+
                    |  PostgreSQL   |
                    +---------------+

        Infrastructure Observability
        --------------------------------
        Prometheus -> Grafana
        Kubernetes Logs -> Loki

        Infrastructure
        --------------------------------
        Amazon EKS
        Terraform
        Helm
        KEDA
        Argo CD / Argo Events
        Amazon S3 / MinIO

The key design principle is that Langfuse does not replace traditional Kubernetes monitoring.

Instead:

Prometheus/Grafana/Loki answer:
"Is my platform healthy?"

Langfuse answers:
"What happened inside my LLM application?"

Why LLM Observability Matters

Consider a RAG request:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
User question
     |
     v
Embedding
     |
     v
Vector search
     |
     v
Retrieved documents
     |
     v
Prompt construction
     |
     v
AWS Bedrock
     |
     v
Generated answer

A traditional application metric might show:

1
2
3
4
HTTP 200
Latency: 3.8 seconds
CPU: 42%
Memory: 1.2 GB

This is useful, but incomplete.

An LLM observability platform can provide a much deeper view:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
Trace
 |
 +-- Retrieval
 |    +-- Weaviate query
 |    +-- 5 documents retrieved
 |
 +-- Prompt
 |    +-- Prompt version: v17
 |
 +-- Bedrock
 |    +-- Model
 |    +-- Input tokens
 |    +-- Output tokens
 |    +-- Latency
 |
 +-- Generation
      +-- Answer
      +-- Evaluation score

This makes it possible to investigate problems at the application and model level.

What is Langfuse?

Langfuse is an open-source platform designed for LLM engineering and observability.

It provides capabilities including:

LLM tracing
Prompt management
Token usage tracking
Cost monitoring
Evaluations
Scores
Datasets
Debugging
Model/application analytics

One of its most important capabilities is tracing.

A trace represents an end-to-end operation, while observations within that trace can represent individual operations such as retrieval, prompt processing, or model generation.

For a RAG application, this creates a useful relationship:

1
2
3
4
5
6
7
8
9
Trace
 |
 +-- Retrieval
 |
 +-- Prompt
 |
 +-- LLM Generation
 |
 +-- Evaluation

Integrating Langfuse with Amazon EKS

Amazon EKS provides the Kubernetes foundation for the platform.

A typical deployment can be managed using Terraform and Helm.

For example:

1
2
3
4
5
6
7
8
terraform/
├── eks.tf
├── langfuse.tf
├── postgresql.tf
├── weaviate.tf
├── kong.tf
├── keda.tf
└── monitoring.tf

Terraform manages the infrastructure and Helm manages Kubernetes applications.

This approach provides reproducibility and makes it easier to promote the same architecture between development, staging, and production environments.

PostgreSQL

Langfuse requires persistent storage for its application data.

PostgreSQL can be deployed inside the EKS cluster for development and smaller environments, while production architectures may use Amazon RDS for PostgreSQL.

A simplified architecture is:

1
2
3
4
5
6
                 Langfuse
                    |
                    v
              PostgreSQL
                    |
             Persistent Data

For production workloads, using a managed database such as Amazon RDS can reduce operational overhead and provide AWS-managed backup, maintenance, and availability capabilities.

Weaviate for RAG

The vector database is responsible for storing embeddings and retrieving relevant content.

In this architecture, Weaviate can store objects such as:

1
2
3
4
5
6
7
DocumentChunk
├── content
├── title
├── source
├── document_id
├── chunk_id
└── namespace

A RAG request becomes:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
Question
   |
   v
Embedding
   |
   v
Weaviate
   |
   v
Relevant chunks
   |
   v
Prompt
   |
   v
AWS Bedrock

Langfuse can trace this process so developers can correlate retrieved context with the generated response.

AWS Bedrock as the LLM Layer

AWS Bedrock provides access to foundation models through AWS APIs.

This allows the RAG application to keep the model layer separate from the rest of the application.

1
2
3
4
5
6
7
8
9
10
RAG API
   |
   v
Langfuse instrumentation
   |
   v
AWS Bedrock
   |
   v
Foundation Model

This separation is valuable when applications evolve from one model to another.

For example, teams can evaluate different models against the same dataset and compare:

latency
token consumption
cost
response quality
evaluation scores

Instrumenting the RAG Application

The Langfuse SDK can be integrated into the application layer.

Conceptually, the application creates a trace for each request:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
trace = langfuse.trace(
    name="rag-query",
    user_id=user_id
)

retrieval = trace.span(
    name="vector-search"
)

# Query Weaviate

generation = trace.generation(
    name="bedrock-generation",
    model=model_name,
    input=prompt
)

# Call AWS Bedrock

generation.end(
    output=response
)

trace.update(
    output=response
)

The exact SDK APIs should be aligned with the Langfuse version used by the application.

The important architectural principle is to instrument the application workflow, rather than only monitoring the underlying Kubernetes infrastructure.

Tracing a RAG Request

A production trace might conceptually look like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
Trace: customer-question-12345
|
+-- Retrieval
|    |
|    +-- Weaviate
|         |
|         +-- 5 chunks
|
+-- Prompt
|    |
|    +-- prompt-v17
|
+-- Generation
|    |
|    +-- AWS Bedrock
|         |
|         +-- Input tokens
|         +-- Output tokens
|         +-- Latency
|
+-- Evaluation
     |
     +-- Relevance score
     +-- Answer quality score

This provides significantly more context when debugging a production RAG system.

Prompt Management

Prompts are application code.

Changing a prompt can change application behavior even when the underlying model remains unchanged.

For this reason, prompt versioning becomes important.

For example:

1
2
3
4
5
6
customer-support-prompt
|
+-- v1
+-- v2
+-- v3
+-- v4

A trace can then be associated with the prompt version used to generate the response.

This allows teams to investigate questions such as:

Did the new prompt improve response quality?
Did token usage increase?
Did latency change?
Did retrieval context become more effective?
Did evaluation scores change?

Evaluating RAG Applications

Observability alone does not tell us whether an answer is good.

A production RAG platform should combine tracing with evaluation.

Possible evaluation dimensions include:

1
2
3
4
5
6
7
                    RAG Evaluation
                         |
          +--------------+--------------+
          |              |              |
          v              v              v
     Retrieval       Answer         Grounding
     Relevance       Quality        / Context

Teams can create evaluation datasets containing representative questions and expected behavior.

The same dataset can then be used to compare application versions.

For example:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
             RAG Version
                 |
       +---------+---------+
       |                   |
       v                   v
    Version A           Version B
       |                   |
       +---------+---------+
                 |
                 v
            Evaluation
                 |
       +---------+---------+
       |         |         |
       v         v         v
    Quality   Latency     Cost

This creates a foundation for regression testing of AI applications.

Cost and Token Observability

LLM applications introduce a new operational dimension: token consumption.

A request may involve:

embedding tokens
retrieved context
prompt tokens
completion tokens
multiple model calls

A seemingly small application can therefore generate significant LLM usage at scale.

Tracing token usage allows teams to correlate:

1
2
3
4
5
6
7
8
9
10
11
12
13
User request
     |
     +-- Retrieved context
     |
     +-- Prompt size
     |
     +-- Input tokens
     |
     +-- Output tokens
     |
     +-- Model latency
     |
     +-- Cost

This makes optimization much more data-driven.

For example, excessive retrieval context may increase prompt size without improving answer quality.

Kubernetes Observability vs. LLM Observability

A production platform should use both.

Layer	Technology	Primary purpose
Infrastructure metrics	Prometheus	Kubernetes/application metrics
Dashboards	Grafana	Infrastructure visualization
Logs	Loki	Kubernetes/application logs
LLM traces	Langfuse	LLM request tracing
LLM evaluation	Langfuse	Quality measurement
Vector search	Weaviate	RAG retrieval
LLM	AWS Bedrock	Model inference
API Gateway	Kong	API traffic management
Autoscaling	KEDA	Event/metric-driven scaling

The resulting observability model is:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
                    AI Application
                         |
          +--------------+--------------+
          |                             |
          v                             v
    Infrastructure                 LLM Layer
          |                             |
    +-----+-----+                 +-----+-----+
    |     |     |                 |     |     |
    v     v     v                 v     v     v
Prometheus Grafana Loki        Traces Prompts Evaluations
                                  |
                                  v
                              Langfuse

Scaling with KEDA

KEDA can be used to scale Kubernetes workloads based on application or event-driven metrics.

For an AI platform, this can be useful when workloads are asynchronous.

For example:

1
2
3
4
5
6
7
8
9
10
11
12
13
S3 Event
   |
   v
Argo Events
   |
   v
RAG Processing Job
   |
   v
Embedding
   |
   v
Weaviate

KEDA can scale workers based on workload demand.

Langfuse can then provide visibility into the LLM operations generated by those workers.

This creates a useful separation:

1
2
3
4
5
6
7
8
KEDA
  -> How many workers should run?

Prometheus
  -> How healthy is Kubernetes?

Langfuse
  -> What are the AI workloads doing?

Event-Driven RAG

An event-driven architecture can connect Amazon S3, Argo Events, Kubernetes Jobs, Weaviate, and Langfuse.

For example:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
                Amazon S3
                    |
              Object Created
                    |
                    v
               Argo Events
                    |
                    v
             Kubernetes Job
                    |
        +-----------+-----------+
        |                       |
        v                       v
   Document Parser          Embedding
                                |
                                v
                            Weaviate
                                |
                                v
                          Ready for RAG

When a user subsequently queries the system:

1
2
3
4
5
6
7
8
9
10
11
12
13
User
 |
 v
Kong
 |
 v
RAG API
 |
 +--> Weaviate
 |
 +--> AWS Bedrock
 |
 +--> Langfuse

This provides an end-to-end architecture for document ingestion and AI inference.

Security Considerations

LLM observability introduces additional security considerations because traces may contain sensitive information.

Before deploying Langfuse in production, consider:

What user data is included in traces?
Are prompts stored?
Are model responses stored?
Which teams can access traces?
How long should traces be retained?
Are credentials stored securely?
Is traffic encrypted?
Is the Langfuse endpoint private or public?
What data should be redacted?

Secrets should be managed using appropriate Kubernetes and AWS mechanisms rather than hard-coded into application manifests.

For AWS environments, teams can also consider integrating Kubernetes workloads with AWS identity mechanisms instead of distributing long-lived AWS credentials.

Exposing Langfuse Through Kong

If Langfuse needs to be exposed externally, Kong can provide the API gateway layer.

A simplified flow is:

1
2
3
4
5
6
7
8
9
10
Internet
   |
   v
AWS Load Balancer
   |
   v
Kong
   |
   v
Langfuse

Kong can provide capabilities such as:

TLS termination
routing
authentication integration
rate limiting
observability
API policies

For internal deployments, Langfuse can instead remain accessible only through internal Kubernetes networking or a private ingress architecture.

Infrastructure as Code

Terraform can be used to make the platform reproducible.

A possible project structure is:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
terraform/
│
├── providers.tf
├── variables.tf
├── locals.tf
│
├── eks.tf
├── langfuse.tf
├── postgresql.tf
├── weaviate.tf
├── minio.tf
├── kong.tf
├── keda.tf
│
└── outputs.tf

This allows the complete AI platform to be recreated consistently.

Helm can then manage application-level configuration:

1
2
3
4
5
6
7
8
9
10
11
12
13
Terraform
   |
   +-- EKS
   |
   +-- PostgreSQL
   |
   +-- Weaviate
   |
   +-- Langfuse
   |
   +-- Kong
   |
   +-- Monitoring

Production Architecture

A more complete production architecture could look like:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
                              Users
                                |
                                v
                         +-------------+
                         |    Kong     |
                         +------+------+
                                |
                                v
                         +-------------+
                         |   RAG API   |
                         |             |
                         | Langfuse SDK|
                         +------+------+ 
                                |
                 +--------------+--------------+
                 |                             |
                 v                             v
          +-------------+                +-------------+
          |  Weaviate   |                | AWS Bedrock |
          | Vector DB   |                | Foundation  |
          +-------------+                |   Models    |
                                         +-------------+

                                |
                                v
                         +-------------+
                         |  Langfuse   |
                         +------+------+
                                |
                                v
                         +-------------+
                         | PostgreSQL  |
                         +-------------+

       -----------------------------------------------------

                         EKS Platform

       Terraform | Helm | KEDA | Argo CD | Argo Events

       Prometheus | Grafana | Loki

This architecture separates concerns while maintaining a common Kubernetes platform.

Operational Workflow

A useful production workflow is:

1. Deploy

Terraform provisions the infrastructure.

2. Configure

Helm deploys Langfuse, Weaviate, PostgreSQL, and supporting services.

3. Instrument

The RAG application integrates the Langfuse SDK.

4. Observe

Production requests generate traces.

5. Evaluate

Representative datasets are used to evaluate application behavior.

6. Optimize

Teams analyze:

retrieval quality
prompt efficiency
model latency
token consumption
application cost

7. Deploy improvements

Application and prompt changes are deployed through the existing CI/CD process.

This creates an iterative AI engineering lifecycle:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
Build
  |
  v
Deploy
  |
  v
Observe
  |
  v
Evaluate
  |
  v
Optimize
  |
  v
Deploy Again

Conclusion

Building production-grade generative AI applications requires more than deploying an LLM.

A complete platform needs visibility across infrastructure, APIs, vector search, prompts, model calls, tokens, latency, cost, and answer quality.

Langfuse provides the LLM-specific observability layer that complements Kubernetes monitoring tools such as Prometheus, Grafana, and Loki.

Combined with Amazon EKS, AWS Bedrock, Weaviate, PostgreSQL, Kong, KEDA, Terraform, and Helm, it can form a flexible foundation for building observable RAG and GenAI platforms on AWS.

The resulting architecture provides a clear separation of responsibilities:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
AWS EKS
  -> Compute and orchestration

Weaviate
  -> Retrieval

AWS Bedrock
  -> Generative AI

Langfuse
  -> LLM observability and evaluation

Prometheus/Grafana/Loki
  -> Infrastructure observability

Kong
  -> API management

KEDA
  -> Event-driven scaling

Terraform/Helm
  -> Infrastructure and application automation

For teams moving generative AI workloads from prototypes into production, this combination creates an important foundation for observable, measurable, and continuously improvable AI applications.

# llm # amazon-bedrock # amazon-bedrock-agents # vector-database # api-gateway

Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.

Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article

Builder Center

Building LLM Observability on Amazon EKS with Langfuse, AWS Bedrock, and Weaviate

Yevgeni Shapiro

Architecture

Why LLM Observability Matters

What is Langfuse?

Integrating Langfuse with Amazon EKS

PostgreSQL

Weaviate for RAG

AWS Bedrock as the LLM Layer

Instrumenting the RAG Application

Tracing a RAG Request

Prompt Management

Evaluating RAG Applications

Cost and Token Observability

Kubernetes Observability vs. LLM Observability

Scaling with KEDA

Event-Driven RAG

Security Considerations

Exposing Langfuse Through Kong

Infrastructure as Code

Production Architecture

Operational Workflow

1. Deploy

2. Configure

3. Instrument

4. Observe

5. Evaluate

6. Optimize

7. Deploy improvements

Conclusion