Why Your Data Lake Gets Expensive Before It Gets Big
Learn how partitioning, Parquet file sizes, and compaction affect Amazon S3 data lake performance and Amazon Athena query costs.
A data lake can become inefficient long before its storage volume becomes impressive.
One common reason is a decision that seems harmless during ingestion: writing every small batch as a separate file.
Imagine a pipeline that receives thousands of small events throughout the day and writes each batch to Amazon S3 as a Parquet file. The data is stored successfully, but as the number of objects grows, analytical queries have to handle more files and metadata.
This is where data layout becomes an engineering decision, not just a storage detail.
The architecture behind the problem
Consider a simple analytical setup:
- Amazon S3 stores the data lake.
- AWS Glue Data Catalog maintains table metadata.
- Amazon Athena queries the data directly from S3.
- AWS Glue ETL can transform data and compact small files when needed.
Each service has a distinct responsibility. However, none of them eliminates the need to design the dataset around its workload.
Three decisions worth revisiting
1. Partition for real query patterns
Partitioning by date can help Athena scan only the relevant data when queries filter by date.
But adding too many partition keys can fragment the dataset into small files and create additional metadata overhead.
The right partition strategy should reflect the filters used by common queries, rather than every dimension available in the data.
2. Choose a suitable file format
Apache Parquet is a columnar format that allows analytical engines to read only the required columns and take advantage of compression.
However, switching to Parquet alone does not guarantee good performance. A dataset containing thousands of tiny Parquet files can still suffer from file-management and query-planning overhead.
3. Compact small files when it makes sense
Compaction combines multiple small files into fewer, larger files while preserving their data structure.
AWS Glue supports managed compaction for Apache Iceberg tables in supported configurations. For other datasets, a scheduled ETL process may be more appropriate.
The trade-off is that compaction consumes compute resources and introduces additional processing. It should address a measured problem, not become another job running without a clear purpose.
How would I investigate the problem?
Before changing the architecture, I would examine:
- The number and size distribution of files in each partition.
- The amount of data scanned by representative Athena queries.
- The partition keys used by frequent queries.
- Query execution times and the overhead of maintaining the dataset.
These measurements help determine whether the primary issue is excessive file counts, poor partition design, inefficient queries, or a combination of factors.
The takeaway
A data lake is not optimized simply because it uses S3, Parquet, and partitioning.
Performance and cost depend on how those components work together with the ingestion pattern and analytical workload.
The best architecture is the one that improves the queries and operations that actually matter, without introducing unnecessary complexity.
Question for the community: If you inherited a slow Athena workload over an S3 data lake, what would you investigate first: file sizes, partition design, or query patterns? What evidence would make you choose one over the others?
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article