Scalable AWS ETL Data pipeline
A scalable end-end data ETL pipeline flow use case using AWS services.
Series: AWS Data and AI (15 articles)
- 1Scalable AWS ETL Data pipeline This article
In this article I am sharing a scalable model Data Pipeline I built for my personal case study.
- Kinesis Data Stream can be used to consume data from multiple input sources.
- Kinesis Data Firehouse can be configured to move the data from above Kinesis Data Stream to S3 bucket. Data is stored in S3 bucket.
- AWS Glue crawler is configured to read data in above S3 bucket and populate AWS Glue Data Catalog with metadata about the data.
- It can crawl multiple data stores in a single run and create or update tables in your Data Catalog.
- These tables are then used as sources and targets for Extract, Transform, and Load (ETL) jobs in AWS Glue.
- The data cleaned(processed) can be further moved to another S3 bucket as a clean data.
- This data can be further moved to AWS redshift which is a data warehouse.
- This data can be further consumed for multiple activities based on Business use case.
Series: AWS Data and AI (15 articles)
- 1Scalable AWS ETL Data pipeline This article
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article