Rift Rewind Developer Challenge #2 - Day 2

📊 Why Process Data?

• Cleaning data (removing those pesky missing values like dodged games)
• Transforming formats (converting timestamps to actual dates, like knowing exactly when Baron was slain)
• Aggregating statistics (calculating your average CS per minute or KDA across multiple matches)
• Filtering (focusing only on your main role, like a true OTP Teemo)
• Joining datasets (combining champion stats with match history, like pairing Yuumi with her perfect ADC)

🎯 What You Will Accomplish Today

🔮 What You Will Need

• An AWS account 💻
• An email address 📧
• Data in your S3 bucket from yesterday’s challenge 🪣
• 10 mins for setup + 30 mins for setup to load + 20 mins for challenge ⏰

🌿 What’s Amazon SageMaker?

🧪 What's SageMaker Unified Studio?

• Create workflows using an intuitive drag-and-drop interface (easier than Yuumi's kit)
• Connect notebooks, queries, and data processing jobs graphically (chain them like a perfect combo)
• Automatically convert your visual designs to Python DAG definitions for Airflow (automatic CS improvement!)

Setting Up SageMaker Unified Studio (30-40 mins)

🌐 Setup Step 1: Create a SageMaker Unified Studio domain (~5 min)

1. Open up your AWS Console and search Amazon SageMaker.
2. On the SageMaker homepage, click Create a Unified Studio domain.
3. Keep the Quick setup selected. For the VPC, choose Create VPC.
4. This will take you to AWS CloudFormation. CloudFormation is like using a blueprint for your entire gaming setup—instead of manually installing League of Legends, configuring your graphics settings, and setting up Discord, you create one template that automatically builds everything exactly how you want it. Scroll to the bottom of the page and click Create stack.
5. Go back to your SageMaker domain setup. Scroll down and expand the Quick setup settings. Scroll down within the Quick setup settings, and you’ll see a section where you can add the SageMakerUnifiedStudioVPC that CloudFormation just created for you! Make sure to also add the 3 private subnets.

Note: You may need to click the blue refresh button if the VPC isn’t appearing yet.

1. After a few minutes, you should receive an email to the address you provided, granting you access to the domain. Click on the link provided in the email and follow the steps to register as an SSO user. Make sure to note down the email and password you choose!

🗂️ Setup Step 2: Create a SageMaker Unified Studio project (~5 min)

• Next up, we’ll need to make a SageMaker Unified Studio project. SageMaker Studio projects are like your League of Legends premade team compositions, but for data science! Just as you'd gather your favorite champions with complementary abilities to dominate the Rift, Studio projects let you bundle all your machine learning resources—notebooks, datasets, and models—into one organized team that works together perfectly!

1. First, you’ll need to login to your SageMaker Unified Studio domain using the email and password you chose in the prior step.
2. After you’ve logged in, you should see the SageMaker Unified Studio domain homepage. Select Create project.
3. Give your project a name. Enter something like rift-rewind-project-[your-name]. In Description, enter “Riot API data processing”. For Project Profile, select All capabilities.
4. Select Continue, then scroll to the bottom of the second page and press Continue again. Finally, on the third page, review all your configurations and click Create Project.

Note: Project creation can take several minutes to complete.

⚙️ Setup Step 3: Setup a Workflow Environment (<1 min setup + 20-30 mins wait time)

1. In the menu on the left-hand side, click Compute. This takes you to the Compute page.
2. On the Compute page, select the Workflow environments tab on the right side. Then choose Create.
3. In the Create workflow environment window, choose Create workflow environment.

Note: Workflow environment creation can take several minutes (20-30) to complete.

Processing Your Data Using Visual ETL (25 mins)

🪪 Step 1: Assign IAM Permissions to Access Your S3 Bucket (~5 mins)

Creating the Policy

1. Go to IAM - Search “IAM” in the AWS console
2. Click Policies > Create Policy
3. In the JSON editor, paste the following policy:

1. Click Create Policy

Attach Policy to Role

1. Click Roles in the IAM menu on the left side
2. Search for datazone_usr_role_64d46teve978dl_dkx17oztq4zq21
3. Select the role and click Add permissions > Attach policies
4. Find and attach the policy you just created!

🪄 Step 2: Create a Data Processing Job (~15 mins)

Setting up Data Source

1. Navigate to Build > Visual ETL flow in the top menu
   2. Select the project you just created (rift-rewind-project-[your-name])
   3. Click Create flow
   4. Select Data managed using full-table access (compatibility) and click Continue
2. Under Add nodes, ensure Data sources is selected. Choose Amazon S3 as the data source. You should now see the visual editor.
3. Getting data from Amazon S3: Next up, we need to get our data from the Amazon S3 bucket we created yesterday. In a new tab (keep this one open because we’ll be coming back to it), navigate to Amazon S3 in the console and open your rift-rewind-match-data-[your-name] bucket.
   1. You should have a folder inside called match-history which was automatically created by the Lambda function we ran.
   2. Inside match-history, you’ll find a folder with user data for a specific user (for example, Hide on bush#KR1). Click on that folder to expand it.
   3. Inside the user data folder, there should be two subfolders: full and stats. full contains the full match data with all the different variables, while stats just includes player stats. Click on stats.
   4. Select one of the the JSON objects in the bucket and click Copy S3 URI. This is the resource identifier, or the unique ID of that individual match! We’ll give this URI to SageMaker Unified Studio so it has access to the player data.

Note: If Copy S3 URI is greyed out, make sure you’ve navigated all the way to the bottom of the folders. Ensure the path on top looks something like: Amazon S3 > Buckets > rift-rewind-match-data-[YOUR NAME] > match-history/ > [PLAYER ID] > stats/ > file name ending in .json

1. Go back to SageMaker Unified Studio.
   1. Click on the Amazon S3 node in the visual editor and paste the S3 URI. It should auto-detect that the object is in JSON format.
   2. Alternatively, you can choose Browse S3 and select your file.

Note: If you are getting the error “NoSuchKey”, it can be due to spaces in the file path name. Try rerunning the Lambda function for a user without spaces in their name (such as riotID: thoxx#na1, with region: na1.)

1. 1. You should now be able to see a preview of the data with info from that match under Data preview!

Configuring Transformations

1. On the left-hand menu, click the large green +. Select the Transforms tab. Here, you’ll see a bunch of options you can use to edit the data.
   1. Choose Drop columns.
   2. Click the plus symbol on the right of the S3 node and drag + drop it to connect it to the Drop columns node.
2. Configuring Drop columns: Click the Drop columns node to open the editor like we did with S3 earlier.
   1. Here, you’ll see a list of the different fields in this data. Fields are pieces of information about that match — like the number of assists, the name of the champion, or number of deaths. It’ll also indicate the format of those fields.
   2. Let’s remove the championId and the gameCreation fields. Select the tick box next to those fields.

Note: Feel free to play around with the different transformation options, there are plenty more in addition to just the one we covered today!

Setting Data Target

1. One last time, click the big green + in the menu, and this time select the Data targets tab.
2. Choose Amazon S3 as your target.
3. Drag and drop Drop columns to connect to the S3 target node.
4. For the S3 URI, choose the S3 path under Project overview when you open Projects. Make sure to open this in a new tab so you don’t lose the current project you’re working on!
   2. This is the S3 bucket associated with your SageMaker Unified Studio project.
5. For format, select JSON and leave the rest of the selections blank.

1. Click Save.
2. Name your Job something descriptive, like rift-rewing-data-processing.
3. Scroll to the bottom and click Save again to create the job.

🎉 Step 3: Running your Data Processing Job (~5 mins)

• It may take a few minutes for your job to run. Check back after a bit and you should see the completed status of your job!
• On the Project overview tab, you should now be able to see the new file that was created!
• To see the output of your data processing job, navigate back to Amazon S3 in the console. You should see a folder that starts with amazon-sagemaker-.... This folder stores the output of your job!
  • Open the folder and open all its subfolders until you reach shared. This folder contains the JSON object output by our job, which will be in the format part-00000-...
  • If you open this JSON file, you’ll see the columns we chose to drop are no longer present!

Conclusion

• Each transformation is like adding another ability to your champion's kit
• Experiment with joins, aggregations, and feature engineering like you'd test different rune combinations
• Don't be afraid to dive the backline of your complex datasets - that's where the high-value insights are hiding