Build a Serverless Data Lake on AWS
Ingest, catalog, and query data cost-effectively with S3, Glue, and Athena.
A data lake on AWS doesn't require a data warehouse cluster running 24/7 — S3, Glue, and Athena together provide storage, schema management, and SQL querying entirely serverlessly, paying only for storage and the queries you actually run.
The pieces
- S3 — raw and processed data storage, organized into logical zones (raw/, processed/, curated/).
- AWS Glue Crawler — scans data in S3, infers schema, and registers tables in the Glue Data Catalog automatically, so you don't hand-write schema definitions.
- Glue ETL jobs (Spark or Python Shell) — transform raw data into cleaned, partitioned, columnar formats.
- Athena — runs standard SQL directly against S3 data using the Glue Catalog's schema, with no cluster to provision.
A typical flow
1
Raw JSON logs → S3 (raw/) → Glue Crawler (infers schema) → Glue ETL job (clean + partition + convert to Parquet) → S3 (curated/) → Athena (SQL queries)Registering a crawler
bash
1
2
3
4
5
aws glue create-crawler \
--name raw-logs-crawler \
--role $GLUE_ROLE_ARN \
--database-name data_lake \
--targets '{"S3Targets":[{"Path":"s3://my-lake/raw/"}]}'Partitioning for performance
Organize curated data by a query-relevant column, typically date —
s3://my-lake/curated/year=2026/month=09/day=15/ — so Athena can skip entire partitions that don't match a query's WHERE clause instead of scanning the full dataset every time.Why this beats a traditional warehouse for many workloads
No cluster to size, patch, or pay for when idle. Storage (S3) and compute (Athena, billed per query) scale and cost independently. It's a strong fit for infrequent, ad-hoc, or bursty analytical workloads; a steady, high-concurrency BI workload with many simultaneous users may still be better served by a provisioned warehouse like Redshift, where the fixed cost of a warm cluster pays for itself against Athena's per-query pricing at high query volume.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article