AWS Builder Center
Building a Self-Healing Data Pipeline: Using LangGraph Agents to Auto-Diagnose Schema Drift in AWS Glue

Building a Self-Healing Data Pipeline: Using LangGraph Agents to Auto-Diagnose Schema Drift in AWS Glue

How I built a phase-gated LangGraph agent that automatically diagnoses schema drift failures in AWS Glue jobs — and why gating tool access by phase, not by prompt, is what makes it safe to trust.

Building a Self-Healing Data Pipeline: Using LangGraph Agents to Auto-Diagnose Schema Drift in AWS Glue
If you've run production ETL on AWS Glue for long enough, you've hit this failure mode: an upstream source quietly renames a column, drops a field, or changes a data type — and your Glue job fails with a stack trace that tells you what broke, but not why, and definitely not how to fix it. Someone gets paged, opens CloudWatch Logs, cross-references the script, manually diffs the schema, and pushes a patch. Repeat every time a source system changes.
I wanted to see how far I could get automating that entire loop — not just detecting the failure, but diagnosing it and proposing a fix — using an AI agent with tightly scoped, phase-gated tool access. Here's what I built and what I learned about keeping an LLM-driven pipeline safe enough to trust.

The Problem: Schema Drift Is the Silent Killer of ETL

Schema drift — unannounced changes to the shape of incoming data — is one of the most common causes of pipeline failure, and one of the most tedious to resolve. It's rarely a logic bug; it's almost always a mismatch between what the script expects and what the data now looks like. That makes it a good candidate for automation, because the diagnosis step follows a fairly repeatable pattern: find the failure, read the logs, inspect the script, compare it against the actual current schema, and propose a targeted fix.
The catch is that "propose a fix" is exactly the kind of task where an LLM can hallucinate — inventing column names, misreading a log, or "fixing" something that wasn't actually broken. So the design challenge wasn't just "can an agent do this," but "can an agent do this in a way that's safe to run against a real pipeline."

Architecture

The system is event-driven, not polling-based:
EventBridge → Lambda → LangGraph agent
When a Glue job fails, an EventBridge rule triggers a Lambda function, which invokes a LangGraph agent scoped to that specific failure. The agent doesn't have free rein over the AWS account — it operates through a fixed, ordered set of phases, each with its own restricted toolset:
  1. find_failure — pull the failed job run details
  1. get_logs — retrieve the relevant CloudWatch logs for that run
  1. read_script — read the Glue script associated with the job
  1. diagnose — compare the script's expectations against the actual schema
  1. fix — generate a targeted patch to the script
  1. verify — sanity-check the proposed fix before it goes anywhere
  1. resolve — open a pull request with the fix
That last step is deliberate: the agent opens a PR, not an auto-merge. No matter how confident the diagnosis, a human reviews and merges the change. The agent's job is to eliminate the investigation time, not to remove human judgment from what ships to production.

Keeping the Agent Honest: Hallucination Reduction

Giving an LLM boto3 access and telling it to "fix the pipeline" is a recipe for confidently wrong changes. A few design choices helped keep it grounded:
  • Phase-gated tool restriction — the agent literally cannot call a "fix" tool during the "diagnose" phase. Each phase only exposes the tools relevant to that step, which narrows the blast radius of any single bad decision.
  • Read-only RDS access during diagnosis — the agent can query  information_schema  to check whether a column the script references actually still exists, but it can't write to the database. This turns "the agent thinks the column is named X" into "the agent checked whether column X exists" — a verifiable claim instead of a guess.
  • S3-jailed script patching — proposed script edits are scoped to a specific S3 location, not an open filesystem, which limits what the agent can touch even if it decides to.
  • Pydantic output validation — every structured output the agent produces (diagnosis, proposed fix) is validated against a schema before it's allowed to proceed to the next phase. Malformed or nonsensical output gets caught immediately rather than silently propagating.
  • Confidence-gated escalation — when the agent's confidence in a diagnosis is low, instead of guessing, the workflow escalates — including the option to route only the highest-risk phases to a stronger (and more expensive) model, rather than paying that cost on every run.

What I'd Explore Next

The current version handles the common case well: a script references a column or type that no longer matches reality. Where it gets more interesting is cascading schema changes — where a drift in one upstream table changes the meaning (not just the shape) of a derived field several jobs downstream. That's a harder diagnosis problem, and probably needs the agent to reason across a dependency graph of jobs rather than a single failure in isolation.
I also think there's a lot of room to formalize the confidence-escalation logic — right now it's a reasonable heuristic, but a more rigorous approach (tracked against real outcomes over time) would make the cost/reliability tradeoff much easier to tune.

Takeaways

If you're building anything similar, the pattern I'd generalize is this: let the agent investigate freely, but gate what it's allowed to change, phase by phase. Treating "diagnose" and "fix" as separate phases with separate permissions — rather than one big "solve this" prompt — made the system both easier to debug and easier to trust. The agent doesn't need unrestricted access to be useful; it needs the right access at each step, and a human in the loop before anything touches production.


Built with LangGraph, AWS Glue, Lambda, EventBridge, and RDS PostgreSQL.
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article