Building a Data Lake on AWS with Lake Formation, Glue, and Athena
The architecture used in the lab was relatively simple but demonstrated an important pattern used in cloud data platforms. The movie dataset was stored in an Amazon S3 bucket. The S3 location was registered with Lake Formation, allowing it to become a designated storage location for the data lake.
Data is only useful when it can be stored, organized, accessed, and analyzed efficiently. As the volume of data grows, simply keeping files in an Amazon S3 bucket is not enough. There needs to be a way to manage the data, understand its structure, control access to it, transform it into useful formats, and query it efficiently.
I recently completed an AWS Builder Lab focused on building a data lake using AWS Lake Formation, and the lab provided a practical look at how several AWS services can work together to create a simple data analytics pipeline.
The main services involved were Amazon S3, AWS Lake Formation, AWS Glue, and Amazon Athena. The workflow started with movie data stored as CSV files in Amazon S3, moved through cataloging and transformation, and ended with querying the data using Athena.
The lab objectives included creating a data lake and database, crawling the data with AWS Glue, querying it with Athena, and transforming the dataset from CSV into Apache Parquet format.
Understanding the Architecture
The architecture used in the lab was relatively simple but demonstrated an important pattern used in cloud data platforms.
The movie dataset was stored in an Amazon S3 bucket. The S3 location was registered with Lake Formation, allowing it to become a designated storage location for the data lake.
AWS Glue was then used to discover the structure of the data and create metadata in the AWS Glue Data Catalog. Amazon Athena could subsequently use that catalog to query the underlying data without requiring a traditional database server.
The overall flow looked like this:
Amazon S3 → Lake Formation → AWS Glue Data Catalog → Athena
A second part of the workflow introduced data transformation:
CSV data → AWS Glue ETL → Parquet → Amazon S3 → AWS Glue Catalog → Athena
The lab architecture demonstrates how storage, governance, metadata management, transformation, and analytics can be separated into different services while still working together.
Exploring the Source Data in Amazon S3
The first step was to examine the resources that had already been provisioned for the lab.
Inside the Amazon S3 bucket, the dataset was stored under a
data/ folder, with the movie CSV file located in the movies_csv/ subfolder.The dataset contained movie information such as:
- Year
- Title
- Directors
- Rating
- Genres
- Rank
- Running time
- Actors
The sample data included movies such as Rush, Prisoners, and The Hunger Games: Catching Fire. The dataset covered movies from 1920 to 2018.
At this stage, the data was simply stored as files. There was no structured database table that an analyst could query directly.
This is where the data lake architecture became useful.
Registering the Data with AWS Lake Formation
The next step was to configure AWS Lake Formation.
Lake Formation provides capabilities for setting up and managing data lakes, including registering data locations and controlling access to data.
I registered the relevant Amazon S3 location as a data lake storage location. The lab used a predefined
LakeFormationServiceRole for this purpose. The role had permissions required to interact with the S3 data location, including actions such as getting, putting, deleting, and listing objects.One important part of this process was understanding that simply storing data in S3 does not automatically create a useful analytical structure.
The data still needs metadata.
For that reason, the next step was to create a database in the AWS Glue Data Catalog.
Creating the Data Catalog Database
The database created during the lab was:
movies-dbAWS Glue Data Catalog stores metadata about data sources, databases, tables, schemas, and locations. A database provides a logical collection of tables that represent the underlying datasets.
This distinction is important.
The database and tables in the Data Catalog do not necessarily contain copies of the actual data. Instead, they describe where the data is located and what its structure looks like.
That metadata can then be used by services such as Athena and AWS Glue.
Using AWS Glue Crawler to Discover the Data
After setting up the data lake, I used an AWS Glue Crawler to inspect the movie dataset.
A Glue crawler connects to a data store and uses classifiers to determine the structure and schema of the data. It then creates metadata tables in the AWS Glue Data Catalog.
The crawler was configured with:
- Name:
movies-table - Data source: Amazon S3
- IAM role:
AdminGlueServiceRole - Target database:
movies-db
After running the crawler, it analyzed the dataset and created a table in the Data Catalog.
One thing worth understanding here is that the Glue Catalog table does not represent a second copy of the movie dataset.
It is essentially a metadata definition describing the underlying data, including its schema and location.
The crawler also generated logs that could be reviewed through Amazon CloudWatch.
This was one of the more practical parts of the lab because it showed how a data engineer can move from raw files toward a structured and discoverable dataset without manually defining every column.
Controlling Access with Lake Formation
Before querying the table, appropriate permissions had to be granted.
The lab used Lake Formation to grant the required
SELECT permissions to the relevant IAM role./
This highlighted an important aspect of data lakes: data availability and data access are separate concerns.
Having a table in the Data Catalog does not automatically mean every user or role should be able to query it.
Lake Formation provides a layer for managing permissions over data catalog resources and registered data locations.
For someone interested in cloud security, this is particularly important.
A data lake can contain large amounts of potentially sensitive information. Having centralized access controls is therefore an important part of designing a secure data platform.
Querying the Data with Amazon Athena
With the table cataloged and permissions configured, I moved to Amazon Athena.
Athena allows SQL queries to be executed against data stored in Amazon S3. In this lab, Athena used the AWS Glue Data Catalog to understand the structure of the dataset.
The first query retrieved a sample of records:
1
SELECT * FROM "data" LIMIT 10;
The query returned records from the movie dataset.
This was a useful demonstration of the relationship between the services:
S3 stores the data.
Glue Catalog describes the data.
Lake Formation controls access to the data.
Athena queries the data.
That separation of responsibilities is one of the concepts I found most useful from the lab.
Running an Analytical Query
The lab also included a small analytical challenge.
The dataset contained 1,002 movies where
genres_0 was Action. I used a SQL aggregation query to verify this:1
2
3
SELECT COUNT(*) AS action_movies
FROM "data"
WHERE genres_0='Action';
The query returned 1,002 records, confirming the expected result.
Although the query itself was simple, it demonstrated an important concept: once the data is cataloged correctly, analysts can interact with files in S3 using familiar SQL rather than manually processing individual files.
Transforming CSV Data into Parquet
The next stage of the lab focused on data transformation.
The original dataset was stored in CSV format. I used AWS Glue ETL to transform the data into Apache Parquet.
AWS Glue supports built-in transformations and uses structures called DynamicFrames for processing data. A DynamicFrame is an extension of an Apache Spark SQL DataFrame and is designed to work with data whose schemas may require additional flexibility during ETL processing.
The Glue job was configured using the visual ETL editor.
The source was:
AWS Glue Data Catalog →
movies-db → dataThe target was:
Amazon S3 → Parquet format
The job was named:
ParquetConversionand the generated script was:
ParquetConversion.pyThe job successfully completed after being submitted to AWS Glue.
Why Parquet Matters
This was probably the most important technical lesson from the lab.
CSV is a common and human-readable format, but it is not designed specifically for analytical workloads.
Parquet, on the other hand, is a columnar storage format.
Instead of storing data primarily row by row, Parquet organizes data by columns. This can make analytical queries more efficient because a query can read the columns it actually needs instead of processing unnecessary data.
After the transformation, the Glue crawler was run again and discovered additional tables, including:
movies_parquetmovies_csvdata
This allowed the lab to compare querying the original CSV-based data with querying the Parquet version.
Comparing CSV and Parquet with Athena
To make the comparison meaningful, I used essentially the same analytical operation against both data sources.
For the CSV dataset:
1
2
3
SELECT Count(*) AS action_movies
FROM "movies_csv"
WHERE genres_0='Action';
For the Parquet dataset:
1
2
3
SELECT Count(*) AS action_movies
FROM "movies_parquet"
WHERE genres_0='Action';
Both queries returned the same result: 1,002 action movies.
However, the important difference was the amount of data scanned.
The lab showed that the Parquet data source had a significantly smaller data-scanned size, even though the query produced the same result. The lab notes that the difference is related to Parquet's columnar storage format.
This is where the practical value of data transformation became much clearer.
When working with larger datasets, reducing the amount of data that has to be scanned can have a meaningful impact on analytical workloads.
If the dataset contained millions or billions of records, inefficient storage and query patterns could become much more expensive and time-consuming.
The Complete Data Pipeline
Putting everything together, the lab demonstrated a complete basic data lake workflow:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
Amazon S3
│
▼
AWS Lake Formation
│
▼
AWS Glue Data Catalog
│
▼
AWS Glue Crawler
│
▼
Data Table
│
▼
Amazon Athena
│
│
SQL Analysis
│
▼
AWS Glue
ETL Job
│
▼
CSV → Apache Parquet
│
▼
Amazon S3
│
▼
AWS Glue Crawler
│
▼
Parquet Data Table
│
▼
Amazon Athena
This architecture demonstrates how AWS services can be combined rather than used independently.
This lab gave me a practical understanding of what goes into building a basic data lake on AWS.
Before the exercise, it was easy to think of a data lake simply as a large storage location. The hands-on workflow made it clear that there is much more to it.
A useful data lake needs to make data discoverable, governable, queryable, and efficient to process.
The combination of Amazon S3, AWS Lake Formation, AWS Glue, and Amazon Athena provided a good demonstration of these concepts.
The CSV-to-Parquet transformation was particularly valuable because it connected an architectural concept with an observable result. The data remained the same, but the columnar Parquet representation significantly reduced the amount of data scanned during the demonstrated Athena query.
For me, this lab was another step toward understanding cloud infrastructure from both a data engineering and cloud security perspective. Building systems that can store and process data is important, but understanding how that data is cataloged, accessed, transformed, and governed is equally important.
Hands-on labs like this help bridge the gap between knowing what an AWS service does and understanding how multiple services work together in a real cloud environment.
Note: Photos are been flagged here. You can check the complete blog post here:
https://muhakeem.hashnode.dev/building-a-data-lake-on-aws-with-lake-formation-glue-and-athena
https://github.com/devmuhakeem/Building-a-Data-Lake-with-AWS-Lake-Formation-Glue-and-Athena
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article