AWS Builder Center

Builder Center

Unifying Data Discovery and Collaboration with Amazon SageMaker Studio - Bringing your Glue data into the project (part 1)

In this post you'll learn how different teams can share data products across organizationalsilos using SageMaker Studio, Glue Data Catalog, and Amazon Redshift. The first part of the series shows how to ingest Glue Data Catalog assets

Katerina

This blog belongs into a three-part series, but you may also read it independently.

Introduction

In today’s data-driven organizations, it’s common to have multiple teams working in silos—each
producing valuable datasets using different technologies and pipelines. But when it comes time to
discover, understand, and use these datasets, especially across team boundaries, the process is often
anything but smooth.
Amazon SageMaker Studio (Unified Experience) aims to bridge these gaps by offering a collaborative
environment where data producers and consumers can easily connect, publish, discover, and subscribe
to data assets—all within a governed and scalable setup.
This blog series walks you through a real-world scenario that many data teams face today. Over three
parts, you'll learn how different teams can share data products across organizational silos using
SageMaker Studio, Glue Data Catalog, and Amazon Redshift. You'll also see how a consumer can
effortlessly discover and subscribe to those assets without needing deep knowledge of the underlying
technologies.

One Business, Two Producers, and a Struggling Consumer

In a typical retail or e-commerce organization, multiple teams are responsible for different slices of
operational data. Let’s look at a common scenario involving three such teams:
  • Producer A is the Orders Team. They manage data related to customer orders—what was purchased, when, and by whom. Their data is ingested and managed within Amazon Redshift, often used to power logistics dashboards and internal KPIs.
  • Producer B is the Shipments Last Mile Team. They track when items are packed, shipped, and delivered. Their data is stored in Amazon S3 and cataloged using AWS Glue for downstream analytics and reporting.
  • Consumer C is a BI team that wants to analyze customer fulfillment patterns—how quickly orders are shipped, which regions face delays, and how order attributes affect delivery timelines.
But there's a problem.
These teams operate independently, and there’s no central business data catalog to tie their work
together. The Orders Team and Shipments Team use different AWS services, metadata definitions,
and access patterns. There’s no shared documentation or data product registry. So when Consumer C
wants to join order and shipment data, they’re stuck:
  • They don’t know what datasets exist or which team owns them.
  • They don’t know whether the data is trusted or up to date.
  • They don’t have a simple way to explore and request access.
Instead of spending their time modeling customer behavior, the Data Science team spends hours in
meetings and Slack threads trying to locate and interpret the data they need.
This lack of visibility and collaboration results in:
  • Siloed knowledge, where each team only understands their part of the data.
  • Wasted time, as consumers reinvent pipelines that already exist.
  • Slower innovation, because access friction kills momentum.

Solution Walkthrough

In this three-part series, we’ll show how Amazon SageMaker Studio (Unified Experience) can break
these silos by enabling producers to publish trusted data assets, and consumers to easily discover and
subscribe to them—all with built-in governance and access control.
In this first part, we’ll guide you through how the Last-Mile team can seamlessly integrate their
existing Glue data into a new SageMaker Studio project.

Assumptions

  • Access to one AWS account
  • A Glue database and table within the account. Database name: last-mile , Table name: last-
  • mile-data
  • The Glue database and tables are not managed by AWS Lake Formation.
  • Your AWS role is configured as a Lake Formation Data Lake Administrator in the account.
  • Have enabled IAM Identity Center
  • All deployment will take place in the AWS us-east-1 region.
  • A sagemaker unified domain is already in place

Create Last Mile Project Profile

1. Open the Amazon SageMaker Unified Studio URL in a new browser or private window and select
the option to Sign in with SSO:
2. Sign in to the application with your SSO credentials
3. In the SageMaker Unified Studio landing page, select the option to Create project.
4. Provide a project name of Last-Mile-Project, When complete, click Continue.
5. On the next screen, leave all values at the default and select Continue.
6. Review the consumer project configuration and select Create project when complete.

Getting Data into SageMaker Studio Unified

In this section, we will integrate existing data assets into your SageMaker Studio project. To access
AWS Glue Data Catalog resources—such as databases and tables—the project execution role must be
granted the appropriate permissions using AWS Lake Formation.

Provide authorization to project role on existing AWS Glue Data Catalog Resources

A. Go to AWS Lake Formation Console . At the left panel under Catalog, click Databases. Find your
database last-mile and go go to Actions and select Edit.
B. Uncheck the "Use only IAM access control for new tables in this database" option and click the Save
button. This step is important because when this option is enabled, all new tables in the database rely
solely on IAM policies for access control—bypassing Lake Formation's fine-grained permission
model. By unchecking it, you ensure that new tables will be governed by Lake Formation permissions,
allowing for more granular and centralized access management.
C. The next step is to revoke the IAMAllowedPrincipals setting for the database. Leaving this
enabled allows any IAM user or role with basic permissions to access the data, bypassing Lake
Formation's fine-grained access controls. This can introduce a security risk by weakening governance
over sensitive datasets. To revoke this permission for the dg_salesdb database:
1) Navigate to Lake Formation > Data Permissions. Apply the following filters:
Database: last-mile
Principal: IAMAllowedPrincipals
2) Locate the relevant permission entry.
3) Select it and click Revoke.

Grant Database Permissions to Your Project Role

To allow your SageMaker Studio project to access the necessary data, you need to grant Lake
Formation permissions to the project execution role on the relevant database.
1. Open the AWS Lake Formation Console.
2. Navigate to Databases, and select the database named last-mile.
3. Click on Actions, then select Grant.
4. In the dialog that appears, choose your project role as the principal
5. and assign the appropriate permissions (e.g., Describe ).

Grant Table Permissions to Your Project Role

1. In the Lake Formation Console, navigate to the last-mile database.2. Click Actions, then select Grant to open the permission dialog.
3. In the LF-Tags or Catalog Resources section:
Expand the Tables (optional) section.
Select All Tables to grant access to every table within the last-mile database.
4. Choose the required permissions (e.g., Select , Describe ) and complete the grant process.
This ensures your project role has the necessary permissions to query all tables in the database.

Register the S3 Location as a Data Lake Location

While we've granted access to the AWS Glue tables through Lake Formation, we also need to register
the underlying Amazon S3 location where the actual data resides. This step is essential because:
Lake Formation table permissions control access to the metadata (e.g., schemas, table
definitions).Data lake location permissions control access to the physical data stored in S3.
Without registering the S3 path as a data lake location, users may have metadata access but won’t be
able to read the data itself.
1. In the Lake Formation Console, use the left-hand navigation menu to go to Data Lake Locations.
2. Click Register Location
3. Choose the s3 path
4. In the Permission Mode, click Hybrid access mode and click Register location
Note: AWS Lake Formation’s hybrid access mode allows two parallel permission models to control
access to the same AWS Glue Data Catalog resources—such as databases, tables, and views:
1. Lake Formation permissions: You can explicitly grant access to selected principals by opting
them in.
2. IAM-based access: All other principals can continue accessing the data using their existing IAM
policies for Amazon S3 and AWS Glue.
By enabling hybrid access mode for the Glue table we want to ingest, we can retain existing IAM-
based access while also enabling fine-grained Lake Formation permissions—which are required for
secure access through SageMaker Studio (Unified Experience) project roles.

Let’s See the Data in Action

Now that we've completed the integration, let's test it out by checking if we can see the Glue data
from within our SageMaker Studio project.
1. Open SageMaker Studio. In the top navigation bar, click the arrow icon to expand the dropdown
menu. Select "Browse all projects" to view the list of available projects.
2. Select the Last-Mile-Project
3. Click Data
4. You should now see a new database named last-mile listed under the Lakehouse section in
SageMaker Studio.
Now that the database is visible, let’s run a SELECT query to confirm that we can access and read
the data from our project. Click the three dots next to the table name, then select “Query with
Athena” from the dropdown menu.
6. The query editor will open in the right panel, displaying the SELECT query. Below it, you should see
the query results, confirming that your data is accessible.

Conclusion

In this post, we walked through how to integrate an existing Glue Data Catalog table into Amazon SageMaker Unified Studio and query it using the built-in tools within a SageMaker Studio project. In the next part of this series, we will explore how the Orders team can register an existing Redshift catalog into their project and leverage that data within SageMaker Unified Studio.
Continue to part 2 here : https://builder.aws.com/content/2zDmCCXc6fCzR91u8Eoi6sZ3EPm/unifying-data-discovery-and-collaboration-with-amazon-sagemaker-studio-bringing-your-redshift-data-into-the-project-part-2
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article