Why We Need Native Schema Drift Alerting in AWS Glue
AWS Glue Crawlers are excellent at discovering schemas and keeping the Data Catalog updated, but schema changes can happen silently. This article explores why native schema drift detection and alerting should be built directly into AWS Glue, and how it could reduce custom Lambda, EventBridge, and schema-diffing infrastructure for production data pipelines.
If you have managed production ETL pipelines on AWS for any length of time, you have likely run into this exact failure mode: an upstream data source quietly renames a column, drops a field, or changes an unexpected data type.
The next thing you know, your Amazon Athena queries break, or your downstream analytics pipelines fail with an obscure stack trace.
Right now, AWS Glue Crawlers are excellent at discovering data schemas and keeping the Data Catalog updated. However, they lack a crucial capability for enterprise production workloads: proactive native alerting when schemas drift.
The Current State: Silence is Not Golden
When an AWS Glue Crawler runs, it scans your data lake, such as Amazon S3, detects changes, and updates the table definitions in the AWS Glue Data Catalog.
The issue is not that Glue fails to notice the change. The issue is that it can update the catalog without providing a native mechanism to clearly alert downstream owners that a potentially breaking schema modification occurred.
Downstream consumers, whether they are Apache Spark jobs, Amazon Athena queries, or Amazon QuickSight dashboards, may remain unaware of the change until a query or pipeline fails.
The Tedious Workaround We All Build
To catch these issues before they impact downstream consumers, engineering teams often end up building custom architectures:
1. AWS Lambda + EventBridge
Triggering a Lambda function when a Glue Data Catalog or crawler-related event is captured.
2. Custom Schema Diffing
Writing custom Python or Spark code to retrieve previous and current schema definitions and compare them programmatically.
3. Manual Notification Pipelines
Connecting the detection logic to Amazon SNS or another notification mechanism to alert the on-call engineer when a potentially destructive change is detected.
These approaches can work, but they introduce additional code, services, monitoring requirements, permissions, and maintenance overhead for a problem that exists directly within the AWS Glue ecosystem.
The Solution: A Native Schema Drift Toggle
AWS Glue could provide a native schema drift detection capability directly within the Crawler configuration.
Imagine a configuration option that allows data engineers to define how schema changes should be handled.
Threshold Policies
Teams could define policies based on the type and severity of a schema change.
For example:
DESTRUCTIVE changes could include dropped columns or incompatible data-type changes.
ADDITIVE changes could include newly introduced nullable columns.
The policy could allow teams to choose whether a particular change should be ignored, generate a warning, or cause the crawler to fail.
Native Event Routing
AWS Glue could publish a structured schema drift event through Amazon EventBridge.
The event could contain information such as:
Table name
Database name
Crawler name
Previous schema
Current schema
Changed columns
Change type
Change severity
Timestamp
For example, an event could clearly indicate:
Old: INT
New: STRING
New: STRING
Instead of requiring every organization to independently implement schema comparison logic, the platform could expose this information as a first-class event.
Crawler Status Updates
Another useful capability would be allowing a crawler to reflect the severity of an unexpected schema change in its execution status.
For example:
INFO: Non-breaking additive change detected.
WARNING: Potentially breaking schema change detected.
FAILED: Schema change violates the configured policy.
This would make crawler execution results much more meaningful to both humans and automated pipeline orchestration systems.
Why This Matters
Schema is effectively an API contract between data producers and data consumers.
When that contract changes unexpectedly, downstream systems can break just as they would when a software API changes without backward compatibility.
A native schema drift capability would give data engineering teams an earlier signal that this contract has changed.
Instead of discovering the problem when an Athena query fails or a Spark job crashes, teams could detect and respond to the change immediately.
Let's Build a Better Data Developer Experience
Data pipelines should be resilient, and observability should not require every organization to build its own schema-diffing framework.
AWS Glue already understands the schemas of the data it crawls. Extending that capability with native schema drift detection, configurable policies, structured EventBridge events, and meaningful crawler statuses would provide a much better developer experience for production data workloads.
I have submitted this feature request directly to the AWS product teams.
If you have encountered unexpected schema changes in your data pipelines, or think native schema drift alerting would be useful, please consider supporting the feature request on the AWS Wishlist.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article