As a computer science graduate student interested in data engineering, I wanted to understand how AWS services connect to build a simple data pipeline.
Three services helped me understand the basics: Amazon S3, AWS Glue, and Amazon Athena.
The Basic Pipeline
Raw Data → Amazon S3 → AWS Glue → Amazon Athena
Amazon S3
S3 provides the storage layer. You can store CSV, JSON, Parquet, logs, and other datasets without maintaining your own storage infrastructure.
AWS Glue
AWS Glue helps discover, catalog, and transform data.
A Glue crawler can scan data stored in S3 and add its metadata to the Glue Data Catalog, making the dataset easier for other AWS services to use.
Amazon Athena
Athena lets you query data stored in S3 using SQL.
For example:
SELECT customer_id, SUM(amount) AS total_spend
FROM transactions
GROUP BY customer_id;
This was useful because I could apply SQL skills directly to cloud data without setting up a traditional database server.
What I Learned
The biggest thing I learned was the role each service plays:
S3 → Storage
Glue → Cataloging and transformation
Athena → SQL querying
Understanding these three components made AWS data engineering much easier for me to approach.
What’s Next?
I want to explore how this simple pipeline changes when you add automation, orchestration, data quality checks, monitoring, and larger datasets.
If you work with AWS data engineering, what would you recommend learning next?
Loading article