AWS Builder Center

Builder Center

Building a Serverless Data Pipeline on AWS: From Raw CSV to Dashboard in Under a Day

Hypothetical scenario, created for learning purposes.

The problem

Imagine Casa do Café, a fictional online store that receives daily sales CSV files from three different marketplaces. Today, someone on the team downloads the files, merges everything into a spreadsheet, and builds a report by hand. The process takes hours, breaks whenever a column is renamed, and doesn't scale.
The goal of this article is to replace that with an automated, low-cost pipeline that has no servers to manage.

Architecture

The flow uses five services:
  1. Amazon S3: stores data in layers (raw/, curated/).
  2. AWS Lambda: triggers processing when a new file arrives.
  3. AWS Glue: converts CSV to Parquet and maintains the data catalog.
  4. Amazon Athena: queries the data with SQL, directly on S3.
  5. Amazon EventBridge: schedules runs and connects events.
The design follows a layered data lake approach: raw data is never modified, and each subsequent step produces a cleaner version.

Step by step

1. Organize the bucket

1
2
3
s3://casa-do-cafe-datalake/
├── raw/sales/marketplace=A/year=2026/month=10/day=07/sales.csv
└── curated/sales/year=2026/month=10/
Partitioning by date and source reduces the amount of data scanned per query and, as a result, lowers Athena costs.

2. React to incoming files

An S3 event triggers a Lambda function, which validates the file name and format and starts the Glue job:
python
1
2
3
4
5
6
7
8
9
10
11
12
13
import boto3

glue = boto3.client("glue")

def handler(event, context):
for record in event["Records"]:
key = record["s3"]["object"]["key"]
if not key.endswith(".csv"):
continue
glue.start_job_run(
JobName="sales-csv-to-parquet",
Arguments={"--input_key": key},
)

3. Transform with Glue

The job does three things: standardizes column names across marketplaces, converts types (dates and monetary values), and writes the result as Parquet in the curated/ layer. The columnar format typically shrinks file sizes considerably and speeds up analytical queries.

4. Query with Athena

Once the Glue Crawler updates the catalog, analysis becomes plain SQL:
sql
1
2
3
4
5
6
7
SELECT marketplace,
date_trunc('month', sale_date) AS month,
SUM(total_amount) AS revenue
FROM curated_sales
WHERE year = 2026
GROUP BY 1, 2
ORDER BY month, revenue DESC;

Lessons learned

  • Never overwrite the raw layer. When a transformation bug shows up, reprocessing from the original data is straightforward.
  • Validate the schema at ingestion. A file with a renamed column should fail early and raise an alert, not contaminate the curated layer.
  • Partition thoughtfully. Too many partitions create lots of small files and hurt performance.
  • Monitor costs from day one. Set up CloudWatch alarms and tag your resources so you know what each pipeline costs.

Next steps

To evolve the solution, consider exploring:
  • Amazon QuickSight for dashboards connected to Athena;
  • AWS Step Functions to orchestrate steps with more robust error handling;
  • Data quality checks (for example, with AWS Glue Data Quality) before publishing to the curated layer.

Conclusion

With managed services, you can go from a manual process to a reliable pipeline with costs that scale with usage. What matters most isn't the tool itself, but the discipline of organizing data in layers, validating early, and automating what repeats.
What other data challenges have you faced while building pipelines on AWS? Share them in the comments.
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article