AWS Builder Center

Builder Center

Building a Cloud-Native Data Integration Platform with Airbyte on AWS

Building a Cloud-Native Data Integration Platform with Airbyte on AWS

Explore how to deploy Airbyte on AWS to build scalable ELT pipelines, connecting APIs, databases, SaaS applications, and files with Amazon S3, Redshift, RDS, and modern analytics and AI platforms.

Yevgeni Shapiro

Head Of Cloud | Driving Cloud-Native Innovation | Platform Engineering
Modern applications generate data across databases, APIs, SaaS platforms, files, and operational systems. As organizations adopt data lakes, warehouses, analytics platforms, and AI workloads, moving this data reliably becomes a critical part of the architecture.
Airbyte is an open-source data integration platform designed for building Extract and Load (EL) and Extract, Load, Transform (ELT) pipelines. It provides connectors for moving data from APIs, databases, SaaS applications, and files into databases, data warehouses, and data lakes.
When combined with AWS services, Airbyte can provide a flexible data movement layer for building cloud-native data platforms.
This article explores an architecture for running Airbyte on AWS, with Amazon EKS providing the Kubernetes platform and AWS storage, database, security, and analytics services supporting the surrounding infrastructure.

Architecture Overview

A common architecture can be represented as:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
DATA SOURCES
┌───────────────────────────────────────────────────────────────┐
│ │
│ APIs SaaS PostgreSQL MySQL Files │
│ │ │ │ │ │ │
└───┴────────────┴─────────────┴─────────────┴───────────┴───────┘
│
▼
┌──────────────────┐
│ Airbyte │
│ │
│ Connectors │
│ EL / ELT │
│ Sync Engine │
└────────┬─────────┘
│
┌──────────────┼──────────────┐
│ │ │
▼ ▼ ▼
┌────────────┐ ┌────────────┐ ┌──────────────┐
│ Amazon S3 │ │ Redshift │ │ RDS / Aurora │
│ Data Lake │ │ Warehouse │ │ Databases │
└─────┬──────┘ └────────────┘ └──────────────┘
│
▼
┌───────────────┐
│ Analytics / │
│ ML / AI │
│ Workloads │
└───────────────┘
The architecture separates data movement from data processing and analytics.
Airbyte is responsible primarily for extracting and loading data, while downstream AWS services can be used for transformation, analytics, machine learning, and AI workloads.

1. Data Sources

The first layer contains systems that generate or store operational data.
Typical sources include:
  • REST APIs
  • SaaS applications
  • PostgreSQL
  • MySQL
  • MongoDB
  • Application databases
  • Event and business systems
  • CSV and JSON files
  • Object storage
  • Internal services
Airbyte's connector model allows different sources to be integrated through a common synchronization framework.
For example:
1
2
3
4
5
6
7
8
CRM
│
├── Customers
├── Accounts
└── Opportunities
│
▼
Airbyte
At the same time, application databases can provide transactional data:
1
2
3
4
5
6
7
8
9
PostgreSQL
│
├── users
├── payments
├── orders
└── transactions
│
▼
Airbyte
This allows multiple data producers to feed a centralized data platform.

2. Airbyte as the Data Movement Layer

Airbyte sits between data producers and data destinations.
Its primary responsibility is to move data between systems.
A simplified pipeline looks like:
1
2
3
4
5
6
7
8
9
10
11
12
13
Source
│
▼
Extract
│
▼
Airbyte Connector
│
▼
Load
│
▼
Destination
Airbyte connections define:
  • Source
  • Destination
  • Data streams
  • Sync frequency
  • Replication behavior
  • Selected tables or resources
  • Data synchronization configuration
This provides a standardized approach to integrating many different data sources.

3. Deploying Airbyte on Amazon EKS

For organizations already using Kubernetes, Amazon EKS provides a natural platform for running Airbyte.
The deployment can be structured as:
1
2
3
4
5
6
7
8
9
10
11
12
13
AWS
│
└── Amazon EKS
│
├── Airbyte
│ ├── Server
│ ├── Worker
│ ├── Connector workloads
│ └── Internal services
│
├── PostgreSQL
│
└── Supporting components
Amazon EKS provides:
  • Kubernetes orchestration
  • High availability
  • Workload scheduling
  • Resource isolation
  • Horizontal scaling
  • Integration with AWS networking
  • IAM integration
  • Monitoring and logging capabilities
For production deployments, Airbyte should be treated as a platform workload rather than simply a standalone application.

4. Storage Architecture

A data integration platform requires persistent storage for application state and synchronization metadata.
A production architecture can use AWS-managed storage services.
For example:
1
2
3
4
5
6
7
8
9
10
11
Airbyte
│
├──────────► Amazon RDS PostgreSQL
│ Metadata
│ Configuration
│ Application state
│
└──────────► Amazon S3
Data Lake
Staging
Raw Data
Using managed AWS services can reduce the operational burden associated with running databases and object storage directly inside Kubernetes.

5. Amazon S3 as the Data Lake

Amazon S3 can serve as the foundation for a centralized data lake.
A possible structure is:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
s3://company-data/
│
├── raw/
│ ├── customers/
│ ├── orders/
│ ├── payments/
│ └── transactions/
│
├── processed/
│ ├── customers/
│ ├── orders/
│ └── transactions/
│
└── curated/
├── analytics/
├── reporting/
└── ml/
The raw layer contains data close to its original form.
The processed layer contains transformed and standardized datasets.
The curated layer contains data optimized for analytics, reporting, and downstream applications.
This separation makes the data platform easier to manage and evolve.

6. Data Warehouse Integration

Airbyte can also load data into analytical databases such as Amazon Redshift.
A typical flow is:
1
2
3
4
5
6
7
8
9
10
Operational Systems
│
▼
Airbyte
│
▼
Amazon Redshift
│
▼
BI / Analytics
This architecture is useful when organizations require structured analytical workloads with SQL-based reporting and business intelligence.
Airbyte focuses on data movement while Redshift provides the analytical processing layer.

7. RDS and Aurora Integration

AWS relational database services can also be used as destinations.
Examples include:
  • Amazon RDS for PostgreSQL
  • Amazon RDS for MySQL
  • Amazon Aurora PostgreSQL
  • Amazon Aurora MySQL
For example:
1
2
3
4
5
6
7
8
9
10
SaaS Application
│
▼
Airbyte
│
▼
Aurora PostgreSQL
│
▼
Application / Analytics
This can be useful for operational data synchronization and downstream application workloads.

8. ELT Architecture

One of the major advantages of this architecture is the separation between data movement and data transformation.
Instead of transforming data during extraction:
1
2
3
4
5
6
7
Source
│
▼
Transform
│
▼
Load
a modern ELT architecture typically follows:
1
2
3
4
5
6
7
8
9
10
11
12
13
Source
│
▼
Extract
│
▼
Load
│
▼
Data Platform
│
▼
Transform
The destination platform performs transformations using the capabilities of the data warehouse, lake, or processing engine.
This allows organizations to preserve raw data and perform transformations independently.

9. Airbyte and Modern Data Lakes

Airbyte can also become part of a larger data lake architecture.
For example:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
┌──────────────┐
│ APIs / SaaS │
└──────┬───────┘
│
┌──────────────┐ │
│ Databases │────────┤
└──────────────┘ │
▼
┌──────────────┐
│ Airbyte │
└──────┬───────┘
│
▼
┌──────────────┐
│ Amazon S3 │
│ Raw Zone │
└──────┬───────┘
│
▼
┌──────────────┐
│ Processing │
│ / Transform │
└──────┬───────┘
│
▼
┌──────────────┐
│ Curated Data │
└──────────────┘
The S3 data lake can then become a shared foundation for analytics, machine learning, and AI applications.

10. Security Architecture

Security should be designed into the platform from the beginning.
A production deployment can use multiple AWS security capabilities.

IAM

Use AWS IAM to control access to AWS resources.
Where possible, workloads running on EKS should use IAM roles for service accounts or EKS Pod Identity rather than static AWS credentials.
This allows workloads to receive only the permissions they require.
For example:
1
2
3
4
5
6
7
8
Airbyte
│
▼
EKS Pod Identity / IAM
│
├── S3 access
├── Secrets Manager access
└── Other required AWS APIs

11. Secrets Management

Database credentials, API tokens, and other sensitive configuration should not be stored directly in application configuration.
A common architecture is:
1
2
3
4
5
6
7
8
9
10
AWS Secrets Manager
│
▼
External Secrets / CSI
│
▼
Amazon EKS
│
▼
Airbyte
Secrets can include:
  • Database credentials
  • API credentials
  • OAuth credentials
  • Destination credentials
  • Encryption keys
  • Service tokens
This provides centralized secret management and reduces credential exposure.

12. Network Architecture

A production Airbyte deployment should typically run inside a controlled VPC.
A simplified architecture is:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
AWS VPC
┌────────────────────────────────────────────────────┐
│ │
│ Public Subnets │
│ │
│ Load Balancer │
│ │ │
│ ▼ │
│ ┌──────────────────────────────┐ │
│ │ Amazon EKS │ │
│ │ │ │
│ │ Airbyte │ │
│ └──────────────────────────────┘ │
│ │
│ Private Subnets │
│ │
│ RDS / Aurora │
│ │
└────────────────────────────────────────────────────┘
Security groups and network policies can restrict communication between workloads.
For sensitive environments, data sources and destinations should remain private whenever possible.

13. High Availability

Production data integration platforms should be designed to tolerate failures.
Amazon EKS can distribute workloads across multiple Availability Zones.
For example:
1
2
3
4
5
6
7
Amazon EKS
│
┌─────────────┼─────────────┐
▼ ▼ ▼
AZ-A AZ-B AZ-C
│ │ │
Worker Worker Worker
This architecture reduces dependency on a single node or Availability Zone.
Managed services such as Amazon RDS and S3 can further improve platform resilience.

14. Scaling Airbyte

Data synchronization workloads can vary significantly.
For example:
1
2
3
4
5
6
7
8
9
10
11
12
13
Low workload
│
▼
Few syncs
│
▼
Higher workload
│
▼
Multiple concurrent syncs
│
▼
More connector workloads
Kubernetes allows the underlying compute capacity to scale according to workload requirements.
Scaling considerations include:
  • Number of concurrent connections
  • Number of records
  • Connector resource requirements
  • Sync frequency
  • CPU usage
  • Memory consumption
  • Network throughput
  • Destination write capacity
For large deployments, workload isolation and appropriate resource requests and limits become especially important.

15. Observability

Data pipelines need more than infrastructure monitoring.
A production platform should monitor both platform health and data pipeline health.
Useful metrics include:

Infrastructure

  • CPU
  • Memory
  • Disk
  • Network
  • Pod restarts
  • Node health

Pipeline

  • Sync success rate
  • Sync duration
  • Records transferred
  • Failed connections
  • Connector errors
  • Data freshness
  • Replication lag
A possible observability stack is:
1
2
3
4
5
6
7
8
9
10
11
12
Airbyte / EKS
│
├── Metrics
├── Logs
└── Events
│
▼
Monitoring Platform
│
┌─────┴─────┐
▼ ▼
Grafana Alerts
CloudWatch and Kubernetes-native observability tooling can be integrated depending on the organization's monitoring strategy.

16. Data Reliability

Moving data is only one part of a reliable data platform.
Organizations should also consider:
  • Schema changes
  • Duplicate records
  • Missing records
  • Failed synchronization jobs
  • API rate limits
  • Database connectivity
  • Destination capacity
  • Data freshness
  • Backfills
A reliable pipeline should provide visibility into failures and allow synchronization jobs to be retried or investigated.

17. Handling Schema Evolution

Source systems frequently change.
For example:
1
2
3
4
5
customers
│
├── id
├── name
└── email
Later, the source application adds:
1
2
3
4
5
6
customers
│
├── id
├── name
├── email
└── customer_status
The data platform must be able to handle these changes without unexpectedly breaking downstream consumers.
Schema evolution should therefore be considered as part of the overall data architecture.

18. Airbyte with Infrastructure as Code

For production environments, Airbyte infrastructure can be managed using Infrastructure as Code.
A common AWS platform stack could include:
1
2
3
4
5
6
7
8
9
Terraform
│
├── VPC
├── EKS
├── IAM
├── S3
├── RDS
├── Secrets Manager
└── Airbyte Helm deployment
This provides:
  • Repeatable deployments
  • Environment consistency
  • Version-controlled infrastructure
  • Automated provisioning
  • Easier disaster recovery
  • Infrastructure review through Git workflows
An organization can maintain separate environments such as:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
dev
│
├── Airbyte
├── S3
└── RDS

staging
│
├── Airbyte
├── S3
└── RDS

production
│
├── Airbyte
├── S3
└── RDS

19. GitOps Deployment

Airbyte can also be incorporated into a GitOps-based platform.
For example:
1
2
3
4
5
6
7
8
9
10
Git Repository
│
▼
Argo CD
│
▼
Amazon EKS
│
▼
Airbyte
This allows Kubernetes manifests and Helm configuration to be managed through Git.
A typical workflow becomes:
1
2
3
4
5
6
7
8
9
10
11
12
13
Developer
│
▼
Git Commit
│
▼
Pull Request
│
▼
GitOps Controller
│
▼
Amazon EKS
This approach provides an auditable deployment process and helps standardize platform operations.

20. Data Platform Integration

Airbyte should generally be considered one component of a larger data platform.
A more complete architecture could look like:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
DATA SOURCES
│
┌───────────────┼───────────────┐
│ │ │
APIs Databases SaaS
│ │ │
└───────────────┼───────────────┘
│
▼
┌─────────────┐
│ Airbyte │
│ Data Move │
└──────┬──────┘
│
▼
┌─────────────┐
│ Amazon S3 │
│ Data Lake │
└──────┬──────┘
│
┌──────────┼──────────┐
▼ ▼ ▼
Transform Warehouse Catalog
│ │ │
└──────────┼──────────┘
│
┌─────────┴─────────┐
▼ ▼
Analytics AI / ML
Airbyte handles the data movement layer, while other components provide transformation, governance, analytics, and AI capabilities.

21. Airbyte for AI and ML Platforms

Modern AI applications require access to high-quality and continuously updated data.
Airbyte can help populate the data layer used by downstream AI systems.
For example:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
Operational Data
│
▼
Airbyte
│
▼
Amazon S3
│
▼
Data Processing
│
▼
Curated Datasets
│
├──────────────► Analytics
│
├──────────────► ML Training
│
└──────────────► RAG / AI Applications
This creates a connection between traditional data integration and modern AI infrastructure.
Fresh operational data can be continuously moved into the data platform, where it can subsequently be processed and made available to AI workloads.

22. Operational Considerations

Before deploying Airbyte in production, several areas should be evaluated.

Capacity

Estimate:
  • Number of connections
  • Records per synchronization
  • Synchronization frequency
  • Concurrent workloads
  • Expected data growth

Security

Define:
  • IAM permissions
  • Network boundaries
  • Secrets management
  • Encryption
  • Access control

Reliability

Plan for:
  • Connector failures
  • Source outages
  • Destination outages
  • API rate limits
  • Schema changes
  • Kubernetes node failures

Cost

Monitor:
  • EKS compute
  • Database resources
  • S3 storage
  • Data transfer
  • Logging
  • Monitoring
  • Destination infrastructure

23. Example End-to-End Architecture

Putting the components together:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
AWS CLOUD
┌──────────────────────────────────────────────────────────────────┐
│ │
│ Amazon VPC │
│ │
│ ┌────────────────────────────────────────────────────────────┐ │
│ │ Amazon EKS │ │
│ │ │ │
│ │ ┌──────────────────────────────────────────────────────┐ │ │
│ │ │ Airbyte │ │ │
│ │ │ │ │ │
│ │ │ API / UI → Server → Workers → Connectors │ │ │
│ │ └───────────────────────┬──────────────────────────────┘ │ │
│ │ │ │ │
│ └──────────────────────────┼─────────────────────────────────┘ │
│ │ │
│ ┌────────────────┼─────────────────┐ │
│ │ │ │ │
│ ▼ ▼ ▼ │
│ Amazon S3 RDS / Aurora Amazon Redshift │
│ Data Lake Metadata DB Warehouse │
│ │ │
│ ▼ │
│ Data Processing │
│ │ │
│ ┌─────┴──────┐ │
│ ▼ ▼ │
│ Analytics AI / ML │
│ │
└──────────────────────────────────────────────────────────────────┘

▲ ▲ ▲
│ │ │
REST APIs Databases SaaS
This architecture creates a clear separation between source systems, data movement, storage, processing, and consumption.
# data-management# data-lakes# data-engineering# data-warehouses# data-integration
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article