Beyond the Standard Pipeline: Building a Governance-First Lakehouse with PySpark, S3, and AWS Glue
Beyond the Standard Pipeline: Building a Governance-First Lakehouse with PySpark, S3, and AWS Glue
Most AWS data engineering tutorials follow the exact same script: upload a raw CSV to Amazon S3, run an AWS Glue Crawler, execute a basic PySpark script, and query the results in Amazon Athena. On paper, it looks flawless. In production, you quickly end up with a Data Swamp filled with thousands of 2 KB files, outdated metadata, zero lineage, and exposed sensitive data.
To stand out on the AWS Builder Center, the goal shouldn't just be demonstrating how to move bytes from point A to point B. It should be illustrating how to build resilient architectures where performance, cost-efficiency, and data governance are treated as first-class citizens from the very first line of code.
The Hidden Bottlenecks: Performance vs. Governance
When designing distributed pipelines with PySpark against Amazon S3 storage, two common pitfalls frequently emerge:
When designing distributed pipelines with PySpark against Amazon S3 storage, two common pitfalls frequently emerge:
The Small File Problem: Unpartitioned writes or overly granular partition keys generate thousands of tiny files. This degrades Glue/Spark reading performance due to excessive S3 GET/LIST API calls and inflates operational costs.
Late-Stage Governance: Attempting to apply masking, fine-grained access control, and auditing after datasets are already landed in S3 leads to downstream breaking changes and expensive refactoring cycles.
The solution is decoupling compute and storage efficiently: using PySpark with deliberate compaction strategies while binding AWS Lake Formation directly into the metadata lifecycle.
Architecture Blueprint
This approach adopts a Medallion Architecture pattern on S3, coupled with proactive governance controls at the catalog layer:
This approach adopts a Medallion Architecture pattern on S3, coupled with proactive governance controls at the catalog layer:
[Data Sources / Events]
│
▼
[S3 Raw Bucket] ──(AWS Glue Job / PySpark)──► [S3 Silver / Gold]
│
▼
[AWS Lake Formation]
│
┌───────────┴───────────┐
▼ ▼
[Amazon Athena] [BI & ML Consumers]
(Column-level access) (PII masking rules)
│
▼
[S3 Raw Bucket] ──(AWS Glue Job / PySpark)──► [S3 Silver / Gold]
│
▼
[AWS Lake Formation]
│
┌───────────┴───────────┐
▼ ▼
[Amazon Athena] [BI & ML Consumers]
(Column-level access) (PII masking rules)
- Optimizing PySpark Writes to Amazon S3
Avoid unmanaged df.write.mode("append").parquet(...) calls that neglect partition layout and shuffles. Below is a production-grade Glue ETL pattern that refines raw data into the Silver layer, curbs file fragmentation, and masks sensitive fields during transformation:
Python
import sys
from pyspark.context import SparkContext
from awsglue.context import GlueContext
from awsglue.job import Job
from awsglue.utils import getResolvedOptions
from pyspark.sql import functions as F
import sys
from pyspark.context import SparkContext
from awsglue.context import GlueContext
from awsglue.job import Job
from awsglue.utils import getResolvedOptions
from pyspark.sql import functions as F
Initialize AWS Glue Job
args = getResolvedOptions(sys.argv, ['JOB_NAME', 'RAW_PATH', 'SILVER_PATH'])
sc = SparkContext()
glueContext = GlueContext(sc)
spark = glueContext.spark_session
job = Job(glueContext)
job.init(args['JOB_NAME'], args)
sc = SparkContext()
glueContext = GlueContext(sc)
spark = glueContext.spark_session
job = Job(glueContext)
job.init(args['JOB_NAME'], args)
1. Resilient read from Raw stage
raw_df = spark.read.format("parquet").load(args['RAW_PATH'])
2. In-flight PII Sanitization
Prevent raw identifiers from persisting where broad analytical roles have access
curated_df = (
raw_df
.withColumn("document_hash", F.sha2(F.col("raw_document"), 256))
.drop("raw_document")
.withColumn("ingestion_date", F.current_date())
)
raw_df
.withColumn("document_hash", F.sha2(F.col("raw_document"), 256))
.drop("raw_document")
.withColumn("ingestion_date", F.current_date())
)
3. Mitigate the Small File Problem
Aim for partition sizes between 128 MB and 512 MB
target_partitions = max(1, int(curated_df.count() / 1_000_000))
(
curated_df
.repartition(target_partitions, "ingestion_date")
.write
.mode("overwrite")
.partitionBy("ingestion_date")
.format("parquet")
.option("compression", "snappy")
.save(args['SILVER_PATH'])
)
curated_df
.repartition(target_partitions, "ingestion_date")
.write
.mode("overwrite")
.partitionBy("ingestion_date")
.format("parquet")
.option("compression", "snappy")
.save(args['SILVER_PATH'])
)
job.commit()
2. In-Job Catalog Registration: Eliminating Redundant Crawlers
Running scheduled AWS Glue Crawlers after every batch run adds unnecessary latency and runtime costs.
2. In-Job Catalog Registration: Eliminating Redundant Crawlers
Running scheduled AWS Glue Crawlers after every batch run adds unnecessary latency and runtime costs.
A cleaner strategy registers new partition metadata directly from the Spark job into the AWS Glue Data Catalog:
Python
Direct catalog registration bypassing post-run crawlers
curated_df.write
.format("parquet")
.mode("append")
.partitionBy("ingestion_date")
.saveAsTable("lakehouse_db.customer_silver")
3. Granular Data Governance with AWS Lake Formation
Once the data is cataloged, access control should not be managed by piling dozens of raw S3 IAM policies onto individual IAM roles. Instead, offload enforcement to AWS Lake Formation:
.format("parquet")
.mode("append")
.partitionBy("ingestion_date")
.saveAsTable("lakehouse_db.customer_silver")
3. Granular Data Governance with AWS Lake Formation
Once the data is cataloged, access control should not be managed by piling dozens of raw S3 IAM policies onto individual IAM roles. Instead, offload enforcement to AWS Lake Formation:
Tag-Based Access Control (LF-TBAC): Define scalable tags such as Confidentiality=Restricted and BusinessDomain=Analytics rather than managing resource ARNs individually.
Column-Level Security: Allow analytical squads to query aggregated transactional metrics in Amazon Athena while explicitly restricting access to identifying columns or cryptographic hashes.
Data Filtering & Row-Level Security: Restrict regional consumers to rows where region = 'us-east-1', eliminating the need to duplicate datasets into segregated regional buckets.
Production Readiness Checklist
Before moving any PySpark pipeline on S3 into production, verify these fundamentals:
Before moving any PySpark pipeline on S3 into production, verify these fundamentals:
[ ] S3 Lifecycle Rules: Automatic transitions for aging partitions to S3 Standard-IA or S3 Glacier to manage storage costs.
[ ] Write Compaction: Enforcing target partition counts prior to writing to avoid microscopic Parquet outputs.
[ ] Encryption Defaults: SSE-KMS enabled at rest, paired with bucket policies that enforce aws:SecureTransport (TLS) in transit.
[ ] Catalog Decoupling: Consumer access routed strictly via Lake Formation permissions rather than direct S3 bucket ARN exposure.
Building cloud data platforms isn't just about scaling compute; it's about crafting sustainable, secure architectures that remain manageable as datasets expand.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article