AWS Builder Center

Builder Center

Running lakeFS on Amazon EKS with Amazon S3 and PostgreSQL

Running lakeFS on Amazon EKS with Amazon S3 and PostgreSQL

Deploy lakeFS on Kubernetes with Amazon S3 as the data lake backend and PostgreSQL for metadata. Learn how Git-like branching, versioning, and rollback can be integrated into a cloud-native data platform.

Yevgeni Shapiro

Head Of Cloud | Driving Cloud-Native Innovation | Platform Engineering
Modern data platforms need more than scalable object storage. As data lakes grow, teams also need reliable version control, reproducibility, collaboration, and data recovery.
lakeFS brings Git-like operations to data lakes, allowing teams to work with data using concepts such as branches, commits, merges, and reverts.
By running lakeFS on Amazon EKS, using Amazon S3 as the underlying data lake and Amazon RDS for PostgreSQL for metadata, organizations can build a scalable and cloud-native data management architecture.

Architecture

The architecture consists of three primary components:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
Amazon EKS
┌─────────────────┐
│ │
│ lakeFS │
│ Deployment │
│ │
└────────┬────────┘
│
┌─────────┴─────────┐
│ │
▼ ▼
┌─────────────┐ ┌─────────────┐
│ Amazon S3 │ │ Amazon RDS │
│ Data Lake │ │ PostgreSQL │
└─────────────┘ └─────────────┘
The responsibilities are separated:
  • Amazon EKS runs lakeFS.
  • Amazon S3 stores the actual data objects.
  • Amazon RDS for PostgreSQL stores lakeFS metadata.
  • lakeFS provides the version-control layer between data applications and S3.

Why lakeFS?

Traditional object storage provides durability and scalability, but managing multiple versions of a data lake can become difficult.
For example, a data pipeline might modify thousands or millions of objects. If a pipeline introduces an unexpected change, restoring the data to a known-good state can be challenging.
lakeFS addresses this by introducing Git-like data operations.
Instead of treating the data lake as a collection of independent objects, teams can work with logical versions of the data.
Common operations include:
  • Branch
  • Commit
  • Merge
  • Revert
  • Tag
  • Diff
  • Isolated development environments
This makes lakeFS particularly useful for data engineering, analytics, machine learning, and AI workloads.

Amazon EKS

Amazon EKS provides the Kubernetes platform for running lakeFS.
A Kubernetes deployment provides several operational benefits:
  • Containerized deployment
  • Horizontal scalability
  • Kubernetes-native configuration
  • Integration with ingress controllers
  • Integration with AWS IAM
  • Automated pod scheduling and recovery
  • Infrastructure-as-code deployment with Terraform or Helm
A typical deployment can include:
1
2
3
4
5
6
7
8
9
10
Amazon EKS
│
├── lakeFS Deployment
│ ├── lakeFS Pod
│ └── lakeFS Service
│
├── Ingress / Load Balancer
│
└── Kubernetes Secrets
└── Database and AWS configuration
For production environments, lakeFS can be exposed through an AWS load balancer or Kubernetes ingress controller.

Amazon S3 as the Data Lake

Amazon S3 is the primary storage layer.
The actual data remains in S3 while lakeFS provides the version-control abstraction on top of it.
A simplified flow looks like this:
1
2
3
4
5
6
7
8
9
10
11
12
Data Application
│
▼
lakeFS
│
▼
Amazon S3
│
├── Raw Data
├── Processed Data
├── Analytics Data
└── ML/AI Data
This separation is important because lakeFS does not require the data itself to be moved into a traditional database.
S3 continues to provide the scalability, durability, lifecycle management, and integration with AWS analytics services.

PostgreSQL Metadata Store

lakeFS also requires a metadata database.
In this architecture, Amazon RDS for PostgreSQL provides the database layer.
1
2
3
4
5
6
7
8
9
lakeFS
│
▼
Amazon RDS PostgreSQL
│
┌────────┴────────┐
│ │
Metadata Repository
& State Information
The PostgreSQL database is used for lakeFS metadata and state rather than storing the data lake itself.
Using Amazon RDS provides managed database capabilities such as:
  • Automated backups
  • High availability options
  • Monitoring
  • Maintenance management
  • Encryption
  • Security-group integration
The database should normally be deployed in private subnets and accessed only by the EKS workloads that require it.

Data Flow

A typical workflow looks like this:
1
2
3
4
5
6
7
8
9
10
Data Pipeline
│
▼
lakeFS
│
┌─────┴─────┐
│ │
▼ ▼
Amazon S3 PostgreSQL
Data Lake Metadata
When a data pipeline writes data through lakeFS, the objects are stored in S3 while lakeFS manages the logical state and version information.
This allows teams to create isolated branches for experimentation.
For example:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
main
│
├── production data
│
└── feature/new-pipeline
│
├── test changes
├── validate data
└── commit
│
▼
merge
│
▼
main
This workflow can significantly reduce the risk associated with testing changes directly against production datasets.

Data Engineering Workflow

Consider a data engineering team developing a new ETL pipeline.
Without data versioning, the team might need to create copies of datasets before experimenting.
With lakeFS, the workflow can instead be:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
1. Create branch
│
▼
2. Run data pipeline
│
▼
3. Validate results
│
├── Failed → Revert
│
└── Passed
│
▼
Commit
│
▼
Merge
The team can experiment without immediately changing the production branch.

Machine Learning and AI Workloads

lakeFS can also be useful for machine learning and AI platforms.
Training datasets frequently change over time. If a model was trained using a particular dataset, reproducing that training run requires knowing exactly which version of the data was used.
A versioned data workflow can look like:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
Data Sources
│
▼
Amazon S3
│
▼
lakeFS
│
┌─────┴─────┐
│ │
▼ ▼
Dataset v1 Dataset v2
│ │
└─────┬─────┘
▼
ML Training
│
▼
Model
This can improve reproducibility by associating model-training workflows with specific data versions.

Security Architecture

Security should be designed across the Kubernetes, AWS, storage, and database layers.
A production architecture can use:
1
2
3
4
5
6
7
8
9
10
11
12
AWS Account
│
┌────────┴────────┐
│ │
Amazon EKS Amazon RDS
│ │
lakeFS Pod PostgreSQL
│
IAM Role
│
▼
Amazon S3
Recommended controls include:
  • Private EKS worker nodes where appropriate
  • Private RDS deployment
  • Security groups restricting database access
  • IAM-based access to S3
  • Encryption at rest
  • TLS for network communication
  • Kubernetes Secrets or an external secrets solution
  • Least-privilege IAM policies
  • Network policies where appropriate
  • Centralized logging and monitoring
For AWS access, IAM Roles for Service Accounts (IRSA) or EKS Pod Identity can be used to provide AWS permissions to workloads without embedding long-lived AWS credentials in containers.

High Availability

For production environments, the architecture can be expanded beyond a single lakeFS pod.
1
2
3
4
5
6
7
8
9
10
11
12
Load Balancer
│
┌──────────┴──────────┐
│ │
lakeFS Pod 1 lakeFS Pod 2
│ │
└──────────┬──────────┘
│
┌───────┴───────┐
│ │
Amazon S3 Amazon RDS
PostgreSQL
Kubernetes can distribute lakeFS replicas across availability zones.
Amazon RDS can provide the database availability layer, while S3 provides highly durable object storage.
The exact configuration should be based on the workload, availability requirements, and lakeFS deployment model.

Observability

Running lakeFS on EKS also makes it possible to integrate it into an existing Kubernetes observability stack.
A typical platform could include:
1
2
3
4
5
6
7
8
9
10
11
lakeFS
│
┌───────────┼───────────┐
▼ ▼ ▼
Metrics Logs Traces
│ │ │
▼ ▼ ▼
Prometheus Loki OpenTelemetry
│
▼
Grafana
Operational metrics can help teams monitor:
  • Request rates
  • Latency
  • Errors
  • Resource utilization
  • Database performance
  • Kubernetes pod health
  • S3-related activity
Centralized logging is especially useful when troubleshooting data pipeline failures or repository operations.
# data-engineering# data-lakes# data-warehouses# amazon-s3# data-management
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article