AWS Builder Center

Builder Center

A Beginner’s Guide to Orchestrating AWS Glue Jobs with Amazon Managed Workflows for Apache Airflow (MWAA)

A Beginner’s Guide to Orchestrating AWS Glue Jobs with Amazon Managed Workflows for Apache Airflow (MWAA)

If you’re building data pipelines on AWS, you’ve probably used AWS Glue to run ETL (Extract, Transform, Load) jobs. Glue automates data movement and transformation so you can focus on insights rather than infrastructure. But what if you need to schedule those jobs, run them in sequence, or trigger them based on the completion of other tasks? That’s where Amazon Managed Workflows for Apache Airflow (MWAA) comes in. This blog will provide everything you need to set up your first MWAA Workflow.

David McAmis

If you’re building data pipelines on AWS, you’ve probably used  AWS Glue  to run ETL (Extract, Transform, Load) jobs. Glue automates data movement and transformation so you can focus on insights rather than infrastructure. But what if you need to schedule those jobs, run them in sequence, or trigger them based on the completion of other tasks? That’s where  Amazon Managed Workflows for Apache Airflow (MWAA)  comes in.
This guide walks you through how to orchestrate AWS Glue jobs using MWAA, even if you’ve never used Airflow before. By the end, you’ll understand how to create, schedule, and monitor Glue jobs from Airflow, complete with a sample DAG (Directed Acyclic Graph) that will runs a Glue job every night at 10 PM.

What is MWAA?

Amazon Managed Workflows for Apache Airflow (MWAA) is a managed service that runs  Apache Airflow , which is an open-source tool used to programmatically author, schedule, and monitor workflows. Instead of managing servers or worrying about scaling your Airflow environment, MWAA does all of that for you.
In Airflow, workflows are written as DAGs ( Directed Acyclic Graphs ) in Python. Each DAG defines a set of tasks (like running an AWS Glue job, invoking a Lambda function, or moving data in S3) and the order in which they run.
In short:
  • AWS Glue handles your ETL processing.
  • MWAA (Airflow) schedules, sequences, and monitors those Glue jobs automatically.

Why Orchestrate AWS Glue Jobs with MWAA?

You could run AWS Glue jobs on their own, but MWAA offers major benefits, including:
  • Centralized Scheduling – Manage multiple Glue jobs, triggers, and dependencies in one place.
  • Flexible Workflows – Combine Glue with other services like S3, Redshift, or Lambda in a single workflow.
  • Built-in Monitoring – Airflow gives you a web-based UI to visualize and track job runs.
  • Automatic Retries – Automatically retry failed tasks or alert your team when something breaks

Setting Up MWAA

Before writing your first DAG, you’ll need an MWAA environment. Here’s a high-level overview of what you need to do:
1. Create an S3 bucket
Before you can create an MWAA environment, you will need to create an S3 bucket. This bucket will be used to store your Airflow DAGS, Plugins and Requirements files, using the structure shown below.. To create this bucket, login to the AWS Management Console and navigate to the S3 console. In the General Purpose bucket area, click on Create bucket, give the bucket a unique name and then click Create bucket. You will be using this bucket later when you setup your MWAA environment.
Airflow bucket setup

3. Create an MWAA environment
To create a Airflow environment, navigate to the Managed Apache Airflow console in the AWS console. From the home page, click on the Create environment button. You will need to give your environment a name, and use the Browse S3 bucket to select the bucket you created earlier.
You will also need to specify where you want to store your DAG files. Using the bucket name you just created, add the /dags prefix on the end and then click Next.
In the next step, you will need to select or setup a VPC. It is very easy to create a VPC specifically for MWAA, using the Create MWAA VPC button, shown below.
MWAA VPC setup option

This option will take you to CloudFormation to execute a CloudFormation template that will create the MWAA VPC for you. This is often the easiest way to get started with MWAA, but you can select an existing VPC if you need to.
Once you have selected the VPC, you can use the default options to continue to setup your MWAA environment. In the last setup step, you will need to click the Create environment button to kick off the MWAA deployment. This may take a few minutes, so in the meantime, let's look at creating your first DAG.
Note: For your MWAA setup, you may choose to use an existing role for the execution role, or create a new one. If your environment requires access to any other AWS services (including AWS Glue) after you create your environment, you must  add the required permissions to your execution role .

Creating your first DAG

So, Airflow DAGs may look complex, but once you start using them, they get easier to understand. An Airflow DAG is written code written in Python and is simply a smart way to automate and organize a series of tasks, ensuring they run efficiently and reliably.
For your first DAG, we are going to create one together-- for this example, we will create a DAG that triggers a Glue job every night at 10 PM. The assumption is that you have already created a Glue job, and now we are ready to schedule it.
Below is a complete example you can copy, customize and upload to your MWAA S3 DAG folder. And since this is python code, make sure you have the correct indentation before you save the code.
Python
from airflow import DAG

from airflow.providers.amazon.aws.operators.glue import GlueJobOperator

from datetime import datetime, timedelta



# Define default arguments for the DAG

default_args = {

    'owner': 'airflow',

    'depends_on_past': False,

    'email_on_failure': True,

    'email': ['datateam@somecompany.com'],

    'retries': 1,

    'retry_delay': timedelta(minutes=5),

}



# DAG Definition

with DAG(

    dag_id="aws_glue_job_nightly_10pm",

    default_args=default_args,

    description='Run AWS Glue job every night at 10 PM',

    start_date = datetime.now() - timedelta(days=1),

    schedule='0 22 * * *',  # CRON for 10 PM UTC daily

    catchup=False,

    tags=['awsglue', 'nightly', 'etl'],

) as dag:

    

    # Define the Glue job task

    run_glue_job = GlueJobOperator(

        task_id='run_glue_etl_job',

        job_name='my-glue-job-name',  # Replace with your AWS Glue job name

        script_args={

            '--job-bookmark-option': 'job-bookmark-enable',

            '--TempDir': 's3://my-temp-bucket/temp/',  # Replace with your temp S3 path

        },

        aws_conn_id='aws_default',  # Airflow connection ID for AWS

        region_name='us-east-1',  # Change as needed

    )



    # Task pipeline (in this case, just the single task)

    run_glue_job

Understanding the DAG

Let’s break down the most important part, which is the DAG definition block. Since we want to DAG to run at 10pm each Wednesday, we are using a CRON expression to indicate when it should run. It looks like this:
Python
schedule_interval='0 22 * * *'
This CRON expression means the DAG runs daily at 10 PM UTC. Adjust for your local timezone if needed (e.g., 0 5 * * * for 10 PM PST). If you are not familiar with CRON, is a time-based job scheduler used in Unix-like operating systems to automatically run tasks (scripts, commands, programs) at specified intervals or times. And Airflow uses CRON expressions for scheduling. If you aren't familiar with CRON, there is a  good primer available from here .

Uploading your DAG

Once your MWAA environment is up and running, you should be able to navigate to the Airflow UI from the details page of the environment, in the MWAA console. At this point, you won't have any DAGs listed, as shown below:

To use the DAG you have created, you will need to upload your DAG (Python script) to the S3 bucket folder that you specified earlier with the /dags prefix.
Once you have uploaded your DAG to S3, MWAA will automatically detect your DAG and make it available in the Airflow UI. With the DAG visible in the MWAA Airflow UI, you can trigger it manually or enable the DAG to let it run automatically every night at 10 PM.

Monitoring and Troubleshooting

When your DAG runs, Airflow provides a visual representation of the workflow in the UI. Each task (like the Glue job) has its own log output. You can:
  • Check logs directly in the Airflow web console.
  • View CloudWatch logs for detailed Glue job output.
  • Re-run or clear failed tasks manually from the Airflow UI.
  • View the Glue job execution in the Glue console.
If something fails, Airflow’s retry mechanism will automatically attempt to rerun the job based on your configuration.

Summary

Orchestrating AWS Glue jobs with MWAA gives you flexibility, scalability, and visibility over your data workflows. Instead of manually triggering ETL jobs, you can automate them, define dependencies, and monitor everything from one interface.
With just a few steps, setting up MWAA, creating your Glue job, and uploading your DAG, you’ve got a fully managed nightly data pipeline running in AWS.
By mastering this integration, you unlock the ability to build complex, automated workflows that can easily scale as your data needs grow.
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article