Data lineage you can defend, part 1: AI proposes, people decide
Why enterprise AI needs data lineage that's built by AI and confirmed by people. Most enterprises can't say where the numbers in their regulatory reports come from without weeks of work. AI agents make that gap more expensive. This part covers why lineage is still a manual job, why no single tool solves it, and the design principle that makes AI-generated lineage trustworthy enough for an auditor.
Key terms in this article Lineage edge: a directed link saying "column X in dataset B is derived from column Y in dataset A, by transformation T". Column-level lineage: lineage traced per field, not just per table. It's what auditors and migration teams actually need. HITL: human-in-the-loop. A person approves, edits, or rejects machine-proposed edges. LLM: large language model. ETL: extract, transform, load. The jobs and scripts that move and reshape data between systems. NL2SQL: natural language to SQL. An agent that turns a question into a database query. BCBS 239: Basel Committee principles for risk data aggregation and reporting. Being able to trace a reported figure back to its source is central to them.
The question nobody can answer quickly
Ask a bank's risk function where a figure in its regulatory return came from. The honest answer is often "give us a few weeks."
The knowledge does exist. It's just spread across a dozen places: ETL tools, a partly populated catalog, thousands of lines of stored procedures, a mainframe batch schedule, and the memory of the two people who built the original pipeline. One of them has left.
QuantumBlack's September 2026 article, Solving the data lineage problem that's holding enterprise AI back , sets out the problem clearly. Data estates rarely become complex by design. They grow through acquisitions, platform changes, and local workarounds. Each change leaves behind duplicated data, undocumented transformation logic, and conflicting definitions of what a "customer" or an "exposure" is.
I agree with most of the diagnosis and with the shape of their solution. In this first part I want to explain why that solution shape is right. Part 2 shows how to build it on AWS, code included.
Three reasons lineage is still a manual job
1. Lineage tools see inside one system, but data moves between many
Most lineage tooling is a point solution. Your Spark platform knows what its jobs read and wrote. Your warehouse knows its own query history. Your ETL vendor knows its own mappings. None of them know that the table your warehouse loads every night is a file a COBOL batch job produced, built from a Db2 table that a 2009 stored procedure keeps up to date.
End-to-end lineage only exists where these per-system views overlap, and in most estates the gaps between them are exactly where the risk is.
2. Documentation goes out of date as soon as it's written
Manual lineage is a snapshot. The day after the spreadsheet is signed off, someone adds a column, changes a join, or retires a source. Unless lineage is regenerated from the systems themselves, it slowly drifts away from the truth, and it does so without anyone noticing, which is the worst kind of failure in a control framework.
3. The people who know are a bottleneck
When lineage depends on human memory, every audit question, migration estimate, and data-quality investigation queues behind the same few experts. Teams copy data rather than reuse it because finding out where the trusted version lives takes too long. The QuantumBlack article puts the direct cost of manually tracing a single critical data element end-to-end in the millions. Whatever the exact figure in your organisation, the time cost is real and growing.
Why AI agents raise the stakes
For years, poor lineage was a compliance problem and a migration tax. Agents change the cost.
An analyst who isn't sure which
customer table is authoritative will ask someone. An agent won't. Give an NL2SQL agent access to a warehouse with five similar customer tables and no lineage, and it will pick one, join it confidently, and return a convincing number. The output looks just as authoritative whether it's right or wrong.Validated lineage gives an agent what an experienced analyst carries in their head: which source is authoritative, how a metric is derived, and which copies to avoid. Without it, every agent you deploy on enterprise data is a governance risk that's simply harder to spot than a spreadsheet.
Why one source of lineage is never enough
The useful insight in the QuantumBlack approach is to run several independent detectors in parallel and combine them, rather than looking for one perfect tool. Figure 1 shows the idea: four signals feed one graph, and a review step decides what is accepted.
Figure 1: Four independent signals, one validated graph.
Each signal has a blind spot, and together they cover each other:
Signal | What it sees well | What it misses | Trust level |
|---|---|---|---|
Runtime capture (what actually ran) | Real executions, at column level | Code that hasn't run recently; systems that aren't instrumented (mainframe, vendor ETL, desktop SQL) | High |
Static code analysis (parsers + LLMs reading the code) | Intended logic, including rarely used paths; plain-language summaries of each transform | Dynamic SQL, runtime parameters, logic hidden in configuration | Medium |
Catalog metadata (names, descriptions, owners) | Business meaning and ownership | Anything undocumented, which in legacy estates is most of it | Low–Medium |
Data similarity (profiles, distributions) | Copies and renames between systems that share no code | Causality: two similar columns aren't necessarily related | Suggestion only |
Large language models matter here because they made the second row practical at scale. Extracting column-level lineage from thousands of COBOL programs or hundreds of vendor-dialect stored procedures used to take a team of specialists months. Models can now read the code, propose the mappings, and summarise each transformation in plain English. In the QuantumBlack article, a Latin American bank cut lineage creation across more than 2,000 COBOL programs from ten weeks to a few hours.
But "propose" is the key word.
The design principle: the LLM proposes, evidence and humans decide
An LLM-derived lineage edge is a hypothesis. Even if a model were right 95% of the time, in a regulated report you couldn't defend the other 5% by saying the model was confident.
So the architecture has to treat trust as a policy, not a model score:
Evidence behind an edge | What happens |
|---|---|
Observed at runtime, or extracted by a deterministic parser | Accepted automatically |
Proposed by an LLM and corroborated by another detector | Accepted automatically |
Proposed by an LLM alone, high confidence, low-risk domain | Accepted, with a random sample reviewed |
Proposed by an LLM alone, low confidence, or feeds a regulated report | Human review |
Suggested by data similarity | Human review, always |
Detectors disagree | Human review, shown as a conflict |
Two properties make the output defensible to an auditor rather than just plausible:
- Every edge carries evidence. It points to the lines of code, the runtime event, or the profile comparison that justifies it. Reviewers judge the evidence, so a confident but wrong model doesn't get through.
- Every decision is recorded. Which detector proposed the edge, which model and prompt version, who approved it, and when. That trail is what lets you call the lineage "audit-grade".
There's a third benefit that's easy to miss. Every rejected edge is labelled data. Track how often reviewers overturn each detector, per code dialect, and you get something most lineage programmes never have: a measured accuracy figure for your own estate, not a vendor's.
One graph, three audiences
The final argument for doing this properly is reuse. The same validated lineage graph serves three groups that today often keep separate, partial versions of the truth:
- Compliance and audit trace a reported number back to its source in minutes, with evidence attached.
- Migration teams get source-to-target mappings and impact analysis before each cut-over, instead of reverse-engineering legacy code by hand.
- AI agents check provenance before they use data, and can be prevented by policy from querying datasets whose lineage hasn't been validated.
That last point is where lineage stops being a cost and starts enabling AI. Validated lineage becomes part of the guardrails for agentic data access: the agent proposes which data to use, and a deterministic policy decides whether it's allowed to.
Start with one report, not the whole estate
The most practical advice in the QuantumBlack article is to start small, and I'd go further: start with something you can measure.
Phase | Do this | You're done when |
|---|---|---|
1 | Pick one regulatory report or one migration wave. Turn on whatever lineage your platforms already capture. | You know your baseline coverage %. |
2 | Point AI-assisted code analysis at the SQL, procedures, and batch code behind that report. | Every source file is either parsed or flagged. |
3 | Have your experts review the low-confidence edges against a manually produced sample. | You have precision and recall for your code. |
4 | Publish the validated graph and connect one consumer: an auditor's trace, a migration plan, or an agent. | One report traceable end-to-end in minutes. |
Six weeks is realistic. At the end you have a repeatable recipe, an accuracy figure you can defend, and a clear case for scaling.
Conclusion
Lineage stays manual because the knowledge is split across systems, documentation goes stale, and a few experts become a bottleneck. AI agents make that more costly, because they act on whatever data they find. Running several detectors in parallel, accepting deterministic and corroborated edges automatically, and sending the rest to a person with evidence gives you lineage that's fast to build and that you can defend in an audit. Start with one report, measure accuracy on your own code, and scale the recipe from there.
How does your organisation trace critical data today: by hand, with a tool, or not at all? Tell me in the comments. I'd especially like to hear from anyone who has tried LLM-based lineage on legacy code.
What's next
Part 2: AI-assisted lineage on AWS shows how to build it: native lineage capture with Amazon SageMaker Catalog, mainframe analysis with AWS Transform, a code-parsing agent on AWS Step Functions and Amazon Bedrock (with working Python), the human review workflow, and how to expose validated lineage to agents through Amazon Bedrock AgentCore Gateway and AgentCore Policy.
References
- QuantumBlack, AI by McKinsey: Solving the data lineage problem that's holding enterprise AI back
- OpenLineage : the open standard for lineage events
Opinions are my own.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article