AWS Builder Center

Builder Center

From CSV to Insights: Building a Data Engineering Pipeline with Amazon S3, AWS Glue, and Amazon Athena

From CSV to Insights: Building a Data Engineering Pipeline with Amazon S3, AWS Glue, and Amazon Athena

Build your first end-to-end data pipeline on AWS, from raw CSV files to actionable insights using Amazon S3, AWS Glue, Parquet, and Amazon Athena.

Introduction

When I started studying Data Engineering on AWS, I noticed something quickly: learning individual services is useful, but understanding how they work together is where things start to make sense.
Knowing what Amazon S3, AWS Glue, and Amazon Athena are is one thing.
Building a pipeline where each service has a specific role is another.
In this hands-on project, I will build a simple data pipeline that takes raw sales data from a CSV file, stores it in Amazon S3, transforms it with AWS Glue, converts it to Parquet, and makes it available for analysis through Amazon Athena.
The final architecture looks like this:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
┌──────────────┐
│ CSV File │
└──────┬───────┘
│
▼
┌──────────────┐
│ Amazon S3 │
│ Raw │
└──────┬───────┘
│
▼
┌──────────────┐
│ AWS Glue │
│ ETL │
└──────┬───────┘
│
▼
┌──────────────┐
│ Amazon S3 │
│ Processed │
│ Parquet │
└──────┬───────┘
│
▼
┌──────────────┐
│ Athena │
│ SQL │
└──────┬───────┘
│
▼
Data Insights
The goal is not to build a production-ready platform. Instead, the goal is to understand the fundamentals behind a cloud-based data pipeline and, more importantly, understand why these services work well together.

The Use Case

Let's imagine that we receive a daily sales file from an e-commerce application.
The source system generates a CSV like this:
1
2
3
4
5
6
id_venda,produto,categoria,quantidade,valor,data
1,Notebook,Electronics,1,3500,2026-09-01
2,Mouse,Electronics,2,120,2026-09-01
3,Office Chair,Furniture,1,850,2026-09-02
4,Keyboard,Electronics,3,250,2026-09-02
5,Desk,Furniture,1,1200,2026-09-03
At first, this looks like a simple file.
But if this process happens every day, we quickly have a data engineering problem.
We need to answer questions such as:
  • Where should we store the original files?
  • How should we transform the data?
  • What format should we use for analytics?
  • How can analysts query the data using SQL?
  • How can the pipeline scale as the amount of data increases?
  • What happens if the transformation logic changes later?
This is where the pipeline architecture becomes useful.

Step 1 — Creating the Data Lake with Amazon S3

The first component of the pipeline is Amazon S3.
I will use S3 as the storage layer for the data lake.
Instead of putting everything into a single directory, I will separate the data into logical layers:
1
2
3
4
5
6
7
sales-data-lake/
│
├── raw/
│ └── sales.csv
│
└── processed/
└── sales.parquet
The raw layer contains the original data exactly as it was received.
The processed layer contains the transformed version of the dataset.
This separation is important.
If something goes wrong during processing, we don't want to lose the original data.
For example, imagine that we later discover that the transformation logic was incorrect.
Because the original file is still available in the raw layer, we can fix the transformation and process the data again.
This is a simple example of an important Data Engineering principle:
Keep your raw data so you can reproduce your transformations.

Step 2 — Uploading the Raw Data

The first version of our pipeline is intentionally simple.
We upload the CSV file to:
1
s3://sales-data-lake/raw/sales.csv
At this point, no transformation has happened.
The data is simply being stored.
This gives us a clear boundary:
1
2
3
4
5
Source Data
│
▼
Amazon S3
Raw
Amazon S3 is not processing our data here.
It is simply providing durable, scalable storage.

Step 3 — Processing the Data with AWS Glue

Now we need to transform our raw data.
This is where AWS Glue comes in.
AWS Glue is a serverless data integration service that can be used to discover, transform, and move data.
For this project, we will use a Glue job to:
  1. Read the CSV from S3.
  2. Convert the data types.
  3. Apply transformations.
  4. Write the result as Parquet.
  5. Store the processed data back in S3.
A simplified Glue job could look like this:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
from pyspark.sql import SparkSession
from pyspark.sql.functions import col

spark = SparkSession.builder \
.appName("SalesETL") \
.getOrCreate()

input_path = "s3://sales-data-lake/raw/"
output_path = "s3://sales-data-lake/processed/"

df = spark.read \
.option("header", True) \
.csv(input_path)

df = df.withColumn(
"quantidade",
col("quantidade").cast("int")
)

df = df.withColumn(
"valor",
col("valor").cast("double")
)

df.write \
.mode("overwrite") \
.parquet(output_path)

spark.stop()
There are several things happening here.
First, Glue reads the CSV from S3.
Then we explicitly convert fields such as quantidade and valor to numeric data types.
Finally, we write the transformed dataset as Parquet.
The resulting flow is:
1
2
3
4
5
6
7
8
9
10
11
S3 Raw
│
▼
AWS Glue
│
├── Read
├── Transform
└── Write
│
▼
S3 Processed

Why Convert CSV to Parquet?

This is one of the most important decisions in this pipeline.
CSV is convenient.
It is human-readable, easy to generate, and supported almost everywhere.
But it is not an ideal format for analytical workloads.
Parquet is a columnar storage format.
Instead of storing data primarily row by row, Parquet organizes the data by columns.
Imagine that our dataset contains:
1
2
3
4
5
6
id_venda
produto
categoria
quantidade
valor
data
If an analytical query only needs:
1
2
categoria
valor
a columnar format can avoid reading unnecessary columns.
This can reduce the amount of data scanned and improve query performance.
For services such as Amazon Athena, reducing scanned data can also help reduce query costs.
So the transformation is not simply:
1
CSV → Parquet
It is really:
1
2
3
Raw Data
↓
Analytics-Optimized Data
That is an important Data Engineering concept.

Step 4 — Querying the Data with Amazon Athena

Now we have our processed data stored in:
1
s3://sales-data-lake/processed/
But storing data is only part of the problem.
We also want to analyze it.
This is where Amazon Athena comes in.
Athena allows us to run SQL queries directly against data stored in Amazon S3.
We can create an external table describing our dataset:
1
2
3
4
5
6
7
8
9
10
CREATE EXTERNAL TABLE vendas (
id_venda INT,
produto STRING,
categoria STRING,
quantidade INT,
valor DOUBLE,
data DATE
)
STORED AS PARQUET
LOCATION 's3://sales-data-lake/processed/';
Now we can query the data using SQL.
For example, let's find the total revenue by category:
1
2
3
4
5
6
SELECT
categoria,
SUM(valor * quantidade) AS receita_total
FROM vendas
GROUP BY categoria
ORDER BY receita_total DESC;
We can also find the best-selling products:
1
2
3
4
5
6
SELECT
produto,
SUM(quantidade) AS unidades_vendidas
FROM vendas
GROUP BY produto
ORDER BY unidades_vendidas DESC;
Or analyze revenue over time:
1
2
3
4
5
6
SELECT
data,
SUM(valor * quantidade) AS receita_total
FROM vendas
GROUP BY data
ORDER BY data;
Now we have completed the entire journey:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
CSV
│
▼
Amazon S3
│
│ Raw Data
▼
AWS Glue
│
│ Transformation
▼
Amazon S3
│
│ Parquet
▼
Amazon Athena
│
│ SQL
▼
Insights

What I Learned from This Pipeline

The most valuable part of this project was not learning how to create an S3 bucket or write an Athena query.
It was understanding the responsibility of each component.

Amazon S3

Storage
It gives us a durable location for our raw and processed data.

AWS Glue

Processing
It allows us to transform and prepare the data for analytical workloads.

Amazon Athena

Analytics
It allows us to query the processed data using SQL without managing a database server.
The architecture works because each service has a clear responsibility.

What Could We Improve?

This pipeline is intentionally simple.
A real production pipeline would require much more.
For example, we could introduce:

Data Quality

What happens if a record contains:
1
quantidade = "ABC"
or:
1
valor = NULL
We would need validation rules and potentially a quarantine layer for invalid records.

Partitioning

As the dataset grows, we could partition the processed data by date:
1
2
3
4
5
6
processed/
├── year=2026/
│ ├── month=09/
│ │ ├── day=01/
│ │ ├── day=02/
│ │ └── day=03/
This can reduce the amount of data scanned by Athena when queries filter by date.

Incremental Processing

Instead of processing the entire dataset every time, we could process only newly arrived files.

Schema Evolution

What happens if the source system adds a new column?
Our pipeline should be able to detect and handle schema changes instead of silently producing incorrect data.

Monitoring and Observability

We could monitor:
  • Glue job failures
  • Processing duration
  • Number of records processed
  • Data quality failures
  • Pipeline execution status

Orchestration

As the pipeline becomes more complex, we could introduce orchestration so that each step runs in the correct order and failures can be handled automatically.

The Bigger Picture

This small project changed the way I look at AWS services.
It is easy to approach cloud learning like this:
1
2
3
4
5
Learn S3
Learn Glue
Learn Athena
Learn another service
Learn another service
But a better question is:
How can I combine these services to solve a real data problem?
Once we start thinking that way, the architecture becomes much easier to understand.
S3 is not just "object storage."
Glue is not just "an ETL service."
Athena is not just "a SQL query tool."
Together, they can form a complete data flow:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
DATA ENGINEERING PIPELINE

┌─────────────┐
│ Data Source │
└──────┬──────┘
│
▼
┌─────────────┐
│ S3 │
│ Raw │
└──────┬──────┘
│
▼
┌─────────────┐
│ Glue │
│ ETL │
└──────┬──────┘
│
▼
┌─────────────┐
│ S3 │
│ Parquet │
└──────┬──────┘
│
▼
┌─────────────┐
│ Athena │
│ SQL │
└──────┬──────┘
│
▼
┌─────────────┐
│ Insights │
└─────────────┘
That is where learning AWS starts to feel like actual engineering.

Conclusion

We started with something extremely simple: a CSV file containing sales data.
By connecting three AWS services, we turned that file into a small analytical data platform.
Amazon S3 gave us scalable storage.
AWS Glue transformed the raw data.
Amazon Athena allowed us to query the processed data using SQL.
The architecture is simple, but the concepts behind it are fundamental to Data Engineering.
And this is probably the biggest lesson I took from the project:
Don't just learn what a cloud service does. Build something that makes you understand why you need it.
From here, the pipeline can evolve into something much more robust with data quality, partitioning, incremental processing, schema evolution, orchestration, monitoring, and other production-grade practices.
But every complex data platform starts with a simple flow.
Ingest. Transform. Store. Query. Learn.
# data-engineering# amazon-s3# aws-glue# etl# pipeline
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article