The Table Was Shared. Why Did Glue Still Fail?
How do large enterprises share sensitive data across AWS accounts without duplicating it everywhere? This article explores AWS Lake Formation through a real-world Glue, Iceberg, and cross-account scenario, focusing on the architecture and troubleshooting lessons that matter in enterprise data platforms.
The table was shared.
The AWS account was correct.
The resource link existed.
The IAM role looked fine.
And the Glue job still failed.
That was the interesting part.
When you work with AWS as a student, it is easy to think of permissions as a simple relationship:
User → permission → resource
Enterprise AWS doesn't stay that simple for long.
Once multiple AWS accounts, Regions, business teams, security boundaries, and data platforms enter the picture, the real relationship looks more like:
Identity → IAM → Catalog → Lake Formation → shared resource → storage → workload
I came across this kind of problem while working with enterprise data pipelines, and it changed how I think about AWS data access.
The biggest lesson wasn't a missing policy.
It was this:
Don't ask whether a user or role has “access to the table.” Follow the path the request actually takes.
This is where AWS Lake Formation becomes much more interesting than a simple data-security service.
Why an enterprise needs Lake Formation in the first place
Imagine an organization with multiple business units:
1
2
3
4
5
6
7
8
9
10
11
12
13
Enterprise
│
┌──────────────┼──────────────┐
│ │ │
Finance Risk Customer Data
│ │ │
└──────────────┼──────────────┘
│
Central Data Platform
│
Amazon S3 Lake
│
Glue Data Catalog
Now imagine the company has separate AWS accounts for:
- Data platform
- Finance
- Analytics
- Risk
- Machine learning
- Reporting
Finance owns financial data.
Risk needs some of it.
Analytics needs another subset.
ML might need customer attributes but not sensitive fields.
Copying everything into every account would solve the immediate access problem, but it creates a bigger operational problem:
Now the company has multiple versions of the same data.
Instead, an enterprise can use Lake Formation to govern access to Data Catalog resources and underlying data while allowing different accounts and teams to consume approved datasets. AWS supports cross-account sharing of Data Catalog resources and uses AWS Resource Access Manager for the sharing mechanism.
The architectural idea becomes:
1
2
3
4
5
6
7
8
9
10
11
12
13
Central Data Account
│
Amazon S3
│
Glue Data Catalog
│
AWS Lake Formation
│
┌─────────────┼─────────────┐
│ │ │
▼ ▼ ▼
Analytics Risk ML
Account Account Account
The data can remain centrally governed without every consumer automatically needing its own physical copy.
That is the enterprise value.
Lake Formation is not just “security for S3”
This was one of the first mental-model changes I found useful.
Amazon S3 stores the files.
The Glue Data Catalog stores metadata about databases and tables.
Lake Formation adds a data-governance and permission model around those resources.
A simplified architecture looks like this:
1
2
3
4
5
6
7
8
9
10
11
AWS Glue Data Catalog
│
│ metadata
▼
Lake Formation
│
authorization
│
▼
Amazon S3
actual data
Lake Formation uses a DBMS-style permission model with permissions such as
SELECT, DESCRIBE, and others depending on the resource. AWS also documents that operations generally need the appropriate IAM permissions as well as Lake Formation permissions.That means this statement:
“My IAM role can access S3.”
does not necessarily answer:
“Can my Glue job read this Lake Formation-governed table?”
Those are different questions.
The enterprise problem gets harder with multiple AWS accounts
Now take a simple example.
The data platform team owns:
1
2
Account A
us-east-1
The analytics workload runs in:
1
2
Account B
us-west-2
The table is:
1
finance.transactions
The architecture might look like:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
ACCOUNT A — DATA PLATFORM
us-east-1
S3
│
▼
Glue Data Catalog
│
▼
Lake Formation
│
│ shared resource
▼
AWS RAM
│
▼
ACCOUNT B — ANALYTICS
us-west-2
Resource Link
│
▼
Glue ETL
│
▼
Spark
Lake Formation supports this cross-account sharing model, and AWS documents cross-Region access through resource links as well.
At this point, the phrase “the table was shared” becomes dangerously vague.
Shared with whom?
In which account?
In which Region?
Through which catalog?
And what exactly was granted?
The resource link is where things get interesting
A resource link sounds simple.
You create something in the consumer account that points to the shared database or table.
Conceptually:
1
2
3
4
5
6
7
Consumer Account Producer Account
Resource Link ─────────────────► Actual Table
"transactions" "transactions"
│
▼
S3
But there is an important AWS behavior here:
Permission on the resource link is not the same as permission on the target.
AWS explicitly documents that granting permissions on a resource link does not grant permissions on the linked target. The target resource must be granted separately.
That means this can happen:
1
2
3
4
5
6
7
Resource link
│
└── DESCRIBE ✓
Target table
│
└── SELECT ✗
And the engineer looking at the consumer account may think:
“But I can see the resource link. Why can't I read the table?”
Because the link and the target are two different authorization objects.
This is one of those details that sounds obvious after you learn it—and wastes a lot of debugging time before you do.
Then Region becomes part of the problem
There is another enterprise lesson here:
Region is not just a deployment choice.
Lake Formation permissions are Region-specific. AWS explicitly states that Lake Formation permissions apply only in the Region where they were granted.
So when debugging:
1
Role has SELECT permission
I would never stop there.
I would ask:
1
2
3
4
5
Which role?
Which account?
Which table?
Which Region?
Which catalog?
AWS supports querying Data Catalog tables across Regions using resource links, including from AWS Glue ETL. The documented setup involves configuring the source Region and creating links in the consuming Region.
This is where an architecture that looked like:
1
Glue → Table
actually becomes:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
Glue
│
▼
Consumer Catalog
│
▼
Resource Link
│
▼
Producer Catalog
│
▼
Lake Formation
│
▼
S3
Why adding more IAM permissions is often the wrong first move
When a data pipeline fails, one of the easiest reactions is:
“Let's add another IAM policy.”
Sometimes that is correct.
But when Lake Formation is involved, it can completely miss the real problem.
AWS documents two separate authorization layers:
1
2
3
4
5
6
7
8
9
Request
│
┌────────┴────────┐
▼ ▼
IAM Lake Formation
│ │
└────────┬────────┘
▼
Data
IAM permissions govern access to AWS APIs and resources.
Lake Formation permissions govern access to Data Catalog resources, S3 locations, and underlying data in the Lake Formation permission model.
So instead of:
“What IAM policy should I add?”
the better enterprise question is:
“Which authorization layer rejected the request?”
That change in thinking is far more valuable than memorizing another policy statement.
A debugging method I now prefer
When a Glue workload fails, I like to follow the request path.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
1. Glue job starts
↓
2. Execution role is assumed
↓
3. Glue / Data Catalog APIs are called
↓
4. Shared resource is resolved
↓
5. Resource link is followed, if used
↓
6. Lake Formation evaluates access
↓
7. Table metadata is resolved
↓
8. Underlying data is accessed
Now the error becomes a classification exercise.
If the execution role cannot call the API
Investigate IAM.
If the catalog object cannot be resolved
Investigate the Data Catalog or resource-link configuration.
If the resource exists but access is denied
Investigate Lake Formation permissions.
If metadata resolution works but data access fails
Investigate the underlying data-access configuration.
This approach is much more useful in a large company because multiple teams may own different parts of the stack.
Why this matters in an MNC
This isn't just a technical detail.
Imagine a company with 200 AWS accounts.
The central data team should not have to create an S3 policy manually for every consumer.
The analytics team should not need ownership of the finance bucket.
The finance team should not have to duplicate every table for every consumer.
And the security team should not need to understand every ETL implementation.
Instead, you can establish an operating model:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
Data Owner
│
│ decides who should access data
▼
Data Platform
│
│ implements governance
▼
Lake Formation
│
│ grants controlled access
▼
Consumer Account
│
▼
Glue / Athena / Analytics workload
This is the difference between technical access and enterprise data governance.
The technology supports a separation of responsibilities.
A simple example of fine-grained access
Suppose an enterprise customer table contains:
1
2
3
4
5
6
customer_id
name
email
phone
income
account_balance
A reporting team may need:
1
2
3
customer_id
name
account_balance
A marketing workload may need:
1
2
3
customer_id
name
email
An ML team may need:
1
2
customer_id
income
Lake Formation supports fine-grained controls for supported workloads, including table- and column-level access, with additional row/cell-level controls available in supported configurations.
This is powerful because the enterprise does not have to answer:
“Who gets the whole bucket?”
It can ask the much better question:
“What data does this workload actually need?”
That is a much more useful security principle.
Where AWS Glue and Lake Formation fit together
For a data engineer, this is probably the most practical part.
Consider:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
S3 Data Lake
│
▼
Glue Data Catalog
│
▼
Lake Formation
│
▼
AWS Glue ETL
│
▼
Transform / Join
│
▼
Curated datasets
The Data Catalog provides the metadata layer.
Lake Formation provides governed access.
Glue performs the ETL.
That division is important because it keeps governance from being rewritten inside every Glue script.
Instead of every application team implementing its own data-access model, the platform can establish a common enterprise policy.
Then introduce Apache Iceberg
This is where the architecture gets even more interesting.
An Iceberg table is not just a single object in S3.
You have table metadata and data files:
1
2
3
4
5
6
7
8
9
10
11
12
13
Glue Data Catalog
│
▼
Iceberg Table
/ \
/ \
Metadata Data
│ │
▼ ▼
metadata/*.json *.parquet
\ /
\ /
S3
When a Glue Spark workload consumes an Iceberg table, there is another layer to think about: how the Spark/Iceberg runtime interacts with the catalog and S3.
AWS documents Glue's Iceberg integration and additional configuration for cross-Region Iceberg access, including an S3 Multi-Region Access Point-based pattern for relevant scenarios.
So at this point, troubleshooting is no longer just:
“Does Lake Formation allow the table?”
You also need to understand the workload's actual execution path.
This is why enterprise architecture becomes less about knowing individual services and more about knowing how those services interact.
One useful surprise: Athena working doesn't prove everything is correct
This is an important distinction when testing.
Suppose an engineer says:
“I queried the shared table through Athena and it worked.”
That is useful evidence.
But it does not automatically prove that every Glue Spark configuration is correct.
The execution paths are different:
1
2
3
4
5
6
7
8
9
10
11
Shared Data
│
┌───────┴───────┐
▼ ▼
Athena Glue
│
▼
Spark
│
▼
Iceberg
For certain shared-resource patterns, AWS requires resource links for integrated services such as Athena, while Glue ETL can access shared resources under supported cross-account configurations.
So a test that works in Athena tells me something about the environment.
It doesn't mean I can stop investigating the Glue path.
What should an enterprise choose: share, replicate, or move compute?
Lake Formation does not eliminate architectural trade-offs.
There are at least three approaches.
| Approach | Advantage | Trade-off |
|---|---|---|
| Governed cross-account sharing | Avoids unnecessary data copies | More authorization and platform complexity |
| Run compute near the data | Simpler data locality | May conflict with account/Region boundaries |
| Replicate data | Consumer gets local data | Replication, consistency, and storage overhead |
The important point is that there is no universal AWS answer.
If the data is highly centralized and many teams need controlled access, sharing can make sense.
If a workload is tightly coupled to a particular Region, local compute may be simpler.
If isolation, availability, or local processing requirements dominate, replication may be justified.
Good cloud engineering is not:
“Use the AWS service with the most features.”
It is:
“Choose the architecture whose operational complexity matches the business requirement.”
What a student should take from this
This is probably the biggest reason I would recommend learning Lake Formation even before working in a large MNC.
Students often learn cloud as:
1
2
3
4
S3 = storage
Glue = ETL
IAM = security
Athena = SQL
That is a good starting point.
Enterprise engineering requires the next level:
1
2
3
4
5
6
7
8
9
10
11
12
13
Business requirement
↓
Data ownership
↓
Security boundary
↓
AWS account strategy
↓
Region strategy
↓
Data-access model
↓
AWS services
That difference matters in interviews too.
Compare these two statements:
“I know AWS Lake Formation.”
versus:
“I understand how to build a governed multi-account data platform where a central team owns the data, consumer accounts receive controlled access, and workloads can consume shared datasets without automatically duplicating the underlying data.”
The second statement demonstrates architecture thinking.
The enterprise mental model I would keep
When I see an AWS data-access problem now, I mentally draw this:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
IDENTITY
│
▼
IAM
│
▼
DATA CATALOG
│
▼
RESOURCE LINK
│
▼
LAKE FORMATION
│
▼
TABLE
│
▼
ICEBERG
│
▼
S3
│
▼
WORKLOAD
Not every architecture contains every box.
But the act of drawing the path forces you to ask the right questions.
And that is what made this problem useful to me.
Five lessons I would carry into any enterprise project
1. “Access” is not one thing
A workload can have IAM permissions and still fail a Lake Formation authorization check.
2. Metadata is not data
A resource link can point to a shared table, but permissions on the link do not automatically grant permissions on the target.
3. Region matters
Lake Formation permissions are Region-specific, and cross-Region access has its own architecture and limitations.
4. Central governance does not require central consumption
A platform team can govern data while different accounts and teams consume approved datasets.
5. Debug the request path
When the architecture is complicated, stop asking:
“Which AWS service is broken?”
Ask:
“Which boundary did the request fail at?”
That question is usually much more productive.
Final thought
The first time I encountered a problem like this, the temptation was to think about Lake Formation as another permissions screen.
That is too small a way to think about it.
In an enterprise, Lake Formation is really part of a much larger question:
How does a company allow hundreds of people and workloads to use valuable data without giving everyone ownership of everything?
That is a much harder problem than creating an S3 bucket.
And it is exactly why enterprise cloud architecture is interesting.
The table being shared was never the real problem.
Understanding who could follow the entire path to that table was.
AWS references
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article