Your First Serverless Data Pipeline on AWS: S3 + Glue + Athena for Beginners

A step-by-step walkthrough to land CSV files in S3, catalogue them with a Glue Crawler, query with Athena SQL and convert to partitioned Parquet - no servers required.

Srinivas S

Published Oct 8, 2026

Last modified Oct 8, 2026

When I started learning data engineering on AWS, every tutorial seemed to jump straight to Spark clusters and streaming. But most real pipelines start much simpler: files land in a bucket, get catalogued, and become queryable with SQL.

In this article, I'll walk through a beginner-friendly, serverless pipeline using three services:

Amazon S3 – where raw and cleaned data lives
AWS Glue Data Catalog + Crawler – discovers the schema automatically
Amazon Athena – runs SQL directly on files in S3, and converts them to Parquet

No servers, no clusters, and you only pay for what you use.

The architecture

1
2
3
CSV files ──► S3 (raw/) ──► Glue Crawler ──► Data Catalog ──► Athena SQL
                                                               │
                                       S3 (curated/, Parquet) ◄┘  (CTAS)

We'll use a simple sales.csv file with columns: order_id, customer_id, amount, order_date.

Step 1: Create a bucket and upload raw data

Bucket names are globally unique, so add a suffix of your own.

1
2
aws s3 mb s3://my-sales-pipeline-demo-123
aws s3 cp sales.csv s3://my-sales-pipeline-demo-123/raw/sales/sales.csv

Tip: Keep a clear folder layout from day one – raw/ for untouched source files and curated/ for cleaned, query-optimised data. It makes permissions, lifecycle rules and debugging much easier later.

Step 2: Catalogue the data with a Glue Crawler

Create a database in the Glue Data Catalog, then a crawler that scans the raw folder and infers the schema.

1
2
3
4
5
6
7
8
9
aws glue create-database --database-input '{"Name":"sales_db"}'

aws glue create-crawler \
  --name sales-raw-crawler \
  --role AWSGlueServiceRole-demo \
  --database-name sales_db \
  --targets '{"S3Targets":[{"Path":"s3://my-sales-pipeline-demo-123/raw/sales/"}]}'

aws glue start-crawler --name sales-raw-crawler

The IAM role needs the AWSGlueServiceRole managed policy plus read access to your bucket. Once the crawler finishes, you'll see a table called sales in sales_db.

Step 3: Query it with Athena

In the Athena console, set a query result location (for example s3://my-sales-pipeline-demo-123/athena-results/) and run:

1
2
3
4
5
6
7
SELECT customer_id,
       COUNT(*)    AS orders,
       SUM(amount) AS total_spent
FROM sales_db.sales
GROUP BY customer_id
ORDER BY total_spent DESC
LIMIT 10;

That's it – you're running SQL on a CSV file sitting in S3.

Step 4: Convert to Parquet with CTAS

Athena charges by the amount of data scanned. CSV is row-based and uncompressed, so every query reads everything. Converting to partitioned Parquet usually cuts scan size (and cost) dramatically.

1
2
3
4
5
6
7
8
9
10
11
12
13
CREATE TABLE sales_db.sales_curated
WITH (
  format = 'PARQUET',
  write_compression = 'SNAPPY',
  external_location = 's3://my-sales-pipeline-demo-123/curated/sales/',
  partitioned_by = ARRAY['order_month']
) AS
SELECT order_id,
       customer_id,
       CAST(amount AS DOUBLE) AS amount,
       order_date,
       date_format(date_parse(order_date, '%Y-%m-%d'), '%Y-%m') AS order_month
FROM sales_db.sales;

Note: the partition column must be the last column in the SELECT.

Now compare the "Data scanned" figure when you query sales_curated filtered on order_month – you'll see the difference straight away.

What I learned

Schema-on-read is powerful – you don't need a database to get value from files.
File format matters more than you think – Parquet + partitions is the cheapest optimisation you'll ever make.
Folder structure is design – raw/ and curated/ zones are the foundation of a lakehouse.
Start serverless – you can add Glue ETL jobs, Step Functions or EventBridge triggers later when you need automation.

Clean up

To avoid surprise charges, delete what you created:

1
2
3
aws glue delete-crawler --name sales-raw-crawler
aws glue delete-database --name sales_db
aws s3 rb s3://my-sales-pipeline-demo-123 --force

Next steps

Trigger the crawler automatically when new files arrive using EventBridge
Replace CTAS with a scheduled Glue ETL job for incremental loads
Visualise the curated table in Amazon QuickSight

🙌 A small request

I'm working towards my AWS Builder Center badges, and community engagement really helps me get there. If you found this useful:

👍 Like this article
💬 Comment with your questions, or how you'd extend this pipeline – I read and reply to every comment
📖 Check out my previous article: Agents Meet the Lakehouse: 5 AWS Trends Data Engineers Should Watch in Late 2026 – likes and comments there are much appreciated too!

Thanks for reading and supporting a fellow builder. Happy building! 🚀

# amazon-s3 # aws-glue # amazon-athena # data-engineering # serverless

Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.

Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article