Automating Data Lake Creation with AWS Lake Formation Blueprints

The Lake Formation blueprint created a workflow responsible for processing the CloudTrail data. That workflow included AWS Glue crawlers, jobs, and triggers that worked together to ingest and transform the data. Once the workflow completed, the processed data could be queried using Athena through the AWS Glue Data Catalog.

Abdulakeem Muhammed

Published Oct 3, 2026

Last modified Oct 3, 2026

Building a data lake manually is useful for understanding how the individual components work, but in a real environment, repeatedly configuring crawlers, ETL jobs, triggers, and databases can become time-consuming.

I recently completed another AWS Builder Lab focused on automating data lake creation using AWS Lake Formation Blueprints. This lab took the data lake concepts I had previously worked with and introduced a more automated approach to ingesting and processing data.

The main objective was to build a data lake for AWS CloudTrail logs and make the data available for analysis through Amazon Athena.

What made this lab particularly interesting was the use of a Lake Formation Blueprint to automatically create and orchestrate an AWS Glue workflow. Instead of manually building every component of the ingestion pipeline, the blueprint generated the workflow and connected the necessary crawlers, jobs, and triggers.

The lab focused on three main objectives: creating an AWS Glue workflow using a Lake Formation blueprint, automating the data lake setup process, and creating a custom AWS Glue workflow.

Understanding the Architecture

The architecture in this lab centered around CloudTrail logs stored in Amazon S3.

The basic flow was:

AWS CloudTrail → Amazon S3 → Lake Formation Blueprint → AWS Glue Workflow → Data Catalog → Amazon Athena

Once the workflow completed, the processed data could be queried using Athena through the AWS Glue Data Catalog.

This architecture demonstrates an important idea in cloud data engineering: automation can turn a collection of individual data-processing tasks into a repeatable workflow.

Exploring the Existing Environment

The first task was to understand the resources that had already been provisioned for the lab.

An Amazon S3 bucket with a name beginning with cloudtraildatabucket was provided as the main storage location. The bucket contained a data/ folder for the dataset and a results/ folder for Athena query results.

I also inspected the LabCloudTrail trail.

The purpose of this was to understand where the CloudTrail logs were being stored and how that S3 location would later be used by the Lake Formation blueprint.

This provided the starting point for the data pipeline:

CloudTrail generates logs → S3 stores the logs → Lake Formation processes the data.

Setting Up AWS Lake Formation

Before creating the automated workflow, the S3 storage location had to be registered with AWS Lake Formation.

Lake Formation provides a centralized way to manage data lake storage locations and permissions.

I first configured the appropriate data lake administrator and then registered the S3 location using the LakeFormationServiceRole.

![](https://cdn.hashnode.com/uploads/covers/631edab3f73f4734bc059f39/665493ba-8e21-4e55-a114-b1a71bba2391.png align="center")

One important configuration was the permission mode:

Lake Formation

This is significant because it allows Lake Formation to manage permissions for the registered data location.

The workflow role, LakeFormationWorkflowRole, was also granted access to the registered S3 location.

![](https://cdn.hashnode.com/uploads/covers/631edab3f73f4734bc059f39/93b19cf5-a65c-432c-9808-3391ebe59cc8.png align="center")

![](https://cdn.hashnode.com/uploads/covers/631edab3f73f4734bc059f39/686b41d6-2201-45dd-8fc8-6d2871745316.png align="center")

This setup created the foundation required for the automated workflow to interact with the data lake.

![](https://cdn.hashnode.com/uploads/covers/631edab3f73f4734bc059f39/cdcd2a6a-c84e-4eef-8c09-42aa78430180.png align="center")

![](https://cdn.hashnode.com/uploads/covers/631edab3f73f4734bc059f39/acec33b5-b587-406e-959b-14539074926a.png align="center")

Creating the CloudTrail Data Catalog Database

The next step was creating a database in the AWS Glue Data Catalog.

The database was named:

cloudtraillogs-db

![](https://cdn.hashnode.com/uploads/covers/631edab3f73f4734bc059f39/ef4d9de0-a441-4ff9-bfb9-402121f79316.png align="center")

This database would eventually contain the metadata describing the CloudTrail data processed by the workflow.

At this point, the basic data lake structure consisted of:

Amazon S3 for storage
Lake Formation for data lake management and permissions
AWS Glue Data Catalog for metadata
A database dedicated to CloudTrail logs

The next challenge was automating the ingestion process.

Introducing Lake Formation Blueprints

This was the main focus of the lab.

A Lake Formation Blueprint provides a predefined workflow that can be applied to a data lake to automate data ingestion and processing.

Instead of manually creating every crawler, ETL job, and trigger, the blueprint can generate an AWS Glue workflow containing these components.

The lab describes a workflow as a way of encapsulating a complex multi-job extract, transform, and load (ETL) process.

The workflow can contain:

Jobs
Crawlers
Triggers

These components are connected together and can be viewed as a directed acyclic graph (DAG) in AWS Glue.

Lake Formation also tracks the workflow as a single entity, making it easier to monitor the overall process.

Creating the CloudTrail Blueprint

I created the blueprint using the AWS CloudTrail blueprint type.

![](https://cdn.hashnode.com/uploads/covers/631edab3f73f4734bc059f39/cf497e2c-ac8d-42b5-8159-b3f3e57d53d0.png align="center")

![](https://cdn.hashnode.com/uploads/covers/631edab3f73f4734bc059f39/2834a9cc-1335-4e0b-9684-6f2f9603f26e.png align="center")

The configuration included:

CloudTrail: LabCloudTrail
Target database: cloudtraillogs-db
Data format: Parquet
Frequency: Run on demand
Workflow name: lf-cloudtrail-workflow
IAM role: LakeFormationWorkflowRole
Table prefix: lab

The use of Parquet was particularly interesting because it meant that the workflow would not simply copy the raw CloudTrail logs into another location. The data would go through a processing pipeline before becoming available for analysis.

After creating the blueprint, the lf-cloudtrail-workflow workflow appeared in Lake Formation.

At this point, the blueprint had effectively converted a set of data-processing requirements into an orchestrated workflow.

Running the Automated Workflow

The next step was to run the workflow.

I started:

lf-cloudtrail-workflow

![](https://cdn.hashnode.com/uploads/covers/631edab3f73f4734bc059f39/943b945c-0d0e-4b31-8579-d75175578b68.png align="center")

The workflow run then appeared in the workflow history.

I moved into the AWS Glue console and opened Workflows (orchestration) to monitor the workflow.

![](https://cdn.hashnode.com/uploads/covers/631edab3f73f4734bc059f39/d1683f0a-04c8-4cfc-85a1-66b639e8cddc.png align="center")

One of the most useful features here was the graphical representation of the workflow.

Instead of seeing a collection of unrelated AWS resources, the workflow displayed the processing pipeline as connected nodes.

The workflow took approximately 15–20 minutes to complete, and the DAG made it possible to observe the progress of the individual components.

![](https://cdn.hashnode.com/uploads/covers/631edab3f73f4734bc059f39/87e8ac4f-c882-4184-965d-91c134f332a0.png align="center")

This is one of the advantages of workflow orchestration: it provides visibility into not only whether a process is running, but also which individual stages have completed or are still processing.

Understanding the Workflow DAG

The workflow generated by the Lake Formation blueprint contained multiple stages.

Conceptually, the workflow followed a pattern similar to:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
Start

  │

  ▼

Pre-Crawler Job

  │

  ▼

Pre-Crawler Trigger

  │

  ▼

Crawler

  │

  ▼

Post-Crawler Trigger

  │

  ▼

Post-Crawler Job

  │

  ▼

ETL Trigger

  │

  ▼

ETL Job

  │

  ▼

Post-ETL Trigger

  │

  ▼

Post-ETL Job

Each component had a specific responsibility.

The crawler discovered the structure of the source data.

The ETL job processed the data.

The triggers controlled when subsequent components should run.

The post-processing jobs updated the workflow state.

This demonstrated how a complex data pipeline can be broken down into smaller, dependent stages while still being managed as a single workflow.

Creating a Custom AWS Glue Workflow

The lab did not stop at using the predefined blueprint.

I also created a separate workflow called:

sample-workflow

![](https://cdn.hashnode.com/uploads/covers/631edab3f73f4734bc059f39/67435f7a-1e75-4af8-b6a4-be309581b19e.png align="center")

The purpose was to understand how the workflow generated by the Lake Formation blueprint was constructed.

![](https://cdn.hashnode.com/uploads/covers/631edab3f73f4734bc059f39/dd89db82-3fce-46b9-b303-f95091ea3f0d.png align="center")

AWS Glue workflows support several node types, including:

Start
Trigger
Job
Crawler

I recreated the main structure of the blueprint workflow using these components.

The first part consisted of a start node and a job.

I then added a pre_crawl_trigger, followed by a crawler.

After the crawler completed, a post_crawl_trigger activated another job.

The workflow was then extended with ETL and post-ETL stages.

The final structure demonstrated how the different components depend on each other.

This was useful because it changed the way I looked at AWS Glue workflows. Instead of seeing the blueprint as something that simply "creates resources," I could see that it was essentially creating an orchestrated dependency graph.

Understanding Triggers in AWS Glue

One of the concepts that stood out during the custom workflow exercise was the role of triggers.

A trigger determines when a particular part of a workflow should execute.

For example, a trigger can wait for a previous job or crawler to complete before activating the next stage.

In the workflow I created, the logic followed a sequence similar to:

Job completes → Trigger activates → Crawler runs → Trigger waits → Next job starts

This is important in ETL pipelines because the order of operations matters.

You wouldn't want a transformation job to process data before the crawler has discovered the required schema.

Likewise, you wouldn't want the final stage of a pipeline to run before the ETL process has successfully completed.

Workflow triggers provide the dependency logic needed to coordinate these operations.

Monitoring the Workflow

After exploring the custom workflow, I returned to the original:

lf-cloudtrail-workflow

I opened its history and inspected the workflow run through the DAG.

The objective was to verify that the individual components had completed successfully.

A Completed workflow status indicated that the CloudTrail data had successfully passed through the workflow and that the new CloudTrail logs database had been created.

This highlighted another important aspect of cloud data engineering:

A workflow is only useful if you can monitor it.

When pipelines become more complex, monitoring and troubleshooting become just as important as creating the pipeline itself.

The DAG provides a visual way to identify where a workflow currently is and which components have completed.

Validating the Data Lake with Athena

Once the workflow completed, I moved to AWS Lake Formation to inspect the resulting tables.

Two tables were available:

_lab_cloudtrail
lab_cloudtrail

The _lab_cloudtrail table represented the CloudTrail data before transformation, while lab_cloudtrail represented the transformed data.

The lab_cloudtrail table used the Parquet format and contained 20 columns.

This showed that the blueprint had done more than simply move files around. It had created a processing pipeline that transformed and cataloged the CloudTrail data.

Querying CloudTrail Data with Athena

The final validation was performed using Amazon Athena.

I queried the processed table using:

1
2
3
SELECT *

FROM "cloudtraillogs-db"."lab_cloudtrail"

LIMIT 10;

The query returned records from the processed CloudTrail dataset.

This completed the full pipeline:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
CloudTrail

    ↓

Amazon S3

    ↓

Lake Formation

    ↓

Blueprint

    ↓

AWS Glue Workflow

    ↓

Crawler + ETL Jobs + Triggers

    ↓

Parquet Data

    ↓

Glue Data Catalog

    ↓

Amazon Athena

The important part is that Athena did not need to know how the data was originally collected or processed. It simply queried the cataloged table.

Finding Errors in CloudTrail Logs

The lab also included a practical query challenge: identifying CloudTrail records containing an error code.

The query was:

1
2
3
SELECT *

FROM "cloudtraillogs-db"."lab_cloudtrail"

WHERE NOT errorcode='';

This demonstrates a simple but useful operational analytics use case.

CloudTrail contains information about activity within an AWS environment. Being able to query those logs makes it possible to investigate events, identify failed operations, and potentially support security monitoring and incident investigation.

For someone learning cloud security, this part of the lab was particularly relevant.

CloudTrail is not just another dataset. It can provide valuable visibility into activity occurring across an AWS environment.

This lab was a natural progression from manually building a data lake to understanding how the process can be automated.

In the previous data lake exercise, I worked with Lake Formation, AWS Glue, and Athena to catalog, transform, and query data.

This time, the focus shifted toward orchestration and automation.

The Lake Formation Blueprint created an AWS Glue workflow that brought together crawlers, ETL jobs, and triggers to process CloudTrail data automatically.

What I found most valuable was understanding what happens behind the scenes.

A blueprint is not simply a shortcut. It creates a structured workflow where individual AWS services perform specific tasks and triggers control how those tasks depend on one another.

The final result was a data lake containing processed CloudTrail data in Parquet format, cataloged through AWS Glue and available for SQL analysis through Athena.

For my continued learning in cloud security, cloud infrastructure, and data engineering, understanding this type of architecture is important. Security data is only useful when it can be collected reliably, processed efficiently, governed properly, and queried when needed.

This lab provided a practical example of how AWS can bring all of those pieces together through Lake Formation, Glue, S3, CloudTrail, and Athena.

Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.

Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article

Builder Center

Automating Data Lake Creation with AWS Lake Formation Blueprints

Abdulakeem Muhammed

Understanding the Architecture

Exploring the Existing Environment

Setting Up AWS Lake Formation

Creating the CloudTrail Data Catalog Database

Introducing Lake Formation Blueprints

Creating the CloudTrail Blueprint

Running the Automated Workflow

Understanding the Workflow DAG

Creating a Custom AWS Glue Workflow

Understanding Triggers in AWS Glue

Monitoring the Workflow

Validating the Data Lake with Athena

Querying CloudTrail Data with Athena

Finding Errors in CloudTrail Logs