AWS Builder Center

Builder Center

Getting Started with Spark on AWS Glue, from SQL

Getting Started with Spark on AWS Glue, from SQL

A practical guide for SQL developers moving to Spark: the execution model, what Glue adds on top, and a full PySpark tutorial.

I spent a long time living inside SQL Server. Stored procedures, SSIS packages, SSRS reports, tables with indexes you take care of like a child. And all of that works. It works very well, in fact, until the day someone tells you "we need to cross three years of bookings from five different providers" and your server looks at you with that face that says "brother, no".
That is where I ran into Spark. And more specifically, into Spark inside AWS Glue, which is the most comfortable door I have found for someone coming from the relational world who has no desire, or budget, to manage a cluster.
This article is basically the document I wish I had found a few months ago. It is not an exhaustive manual. It is the mental map you need so your first weeks are not spent fighting things you do not understand. It is written for students and for people who, like me, are making the jump from databases into data engineering.
Let's go step by step.

1. Spark in five minutes, explained from SQL

Before touching Glue you have to understand what Spark is, because Glue is just the wrapper. If you do not understand Spark, Glue will look like unpredictable magic, and unpredictable magic in production is called an incident.

The mental shift you have to make

When you write a SELECT in SQL Server, there is one machine reading pages from disk, using its indexes, resolving your query and returning rows. All the work happens in one place. Your intuition about performance revolves around indexes, statistics and execution plans.
In Spark there are many machines. And one of them does not work: it coordinates.
  • The driver is the brain. It interprets your code, builds the plan and hands out tasks. It does not process the data.
  • The executors are the workers. Each one receives a piece of the data and works on it in parallel.
This sounds obvious written out like that, but it has a brutal consequence that explains 80% of beginner errors: when you bring data to the driver, you killed the parallelism. You are back to a single machine, but now with the network in the middle. We will come back to this.

Partitions: the unit that replaces your intuition about indexes

In SQL Server you think in 8 KB pages. In Spark you think in partitions.
A partition is a block of data that one executor processes independently. If your dataset has 200 partitions and you have 10 executors, each executor processes about 20 partitions, one after another. The number of partitions is, literally, your level of parallelism.
Two uncomfortable truths come out of this:
  • Too few partitions: you have executors sitting idle while one of them suffers with a giant block.
  • Way too many tiny partitions: you spend more time coordinating than processing. Every task has a fixed startup cost, and a thousand tasks of 50 milliseconds each are a waste.

Lazy evaluation: the part that was hardest to accept

This is what threw me off the most at the beginning. In Spark, this line does nothing:
1
df_filtered = df.filter(df.status == "CONFIRMED")
Zero. It did not read data, it did not filter anything. It only wrote a note in a notebook: "when they ask me for results, I will have to filter by status".
Spark splits operations into two families:
  • Transformations (filter, select, join, withColumn, groupBy): they build the plan, they are lazy, they execute nothing.
  • Actions (show, count, write, collect): they trigger the real execution of the whole accumulated plan.
Why is it designed this way? Because it lets Spark see the complete chain before moving a single byte. Its optimizer (Catalyst) can reorder filters, push them down toward the read, drop columns you never used and fuse operations together.
If you come from SQL, you already know this: it is an execution plan. The difference is that in SQL Server the engine receives the complete query at once and plans; in Spark you build the query line by line in Python, and the engine plans at the end, when you call an action. Same concept, with the order reversed.

Shuffle: the villain of the story

There are operations Spark resolves inside each partition without talking to anyone: a filter, a withColumn. Those are cheap.
And there are operations where one executor needs data that lives in another executor. A groupBy by country needs to gather all the rows from Colombia, and those rows are spread across every partition. A join needs matching keys to land on the same machine.
That is a shuffle: redistributing data over the network. It is by far the most expensive operation in Spark. When your job takes fifteen minutes and you do not understand why, there is almost always a shuffle in the middle.
In SQL Server terms: it is like a hash join, except you also have to serialize the data, send it over the network, write it to temporary disk and read it back. It is not that joins in Spark are wrong, it is that you need to know they cost something, and that there are ways to avoid them. The broadcast join is coming up.

2. What Glue adds on top of Spark

Now Glue. Short version: Glue is serverless Spark with a metadata catalog attached to the side.
Without Glue, running Spark means spinning up machines, installing the runtime, configuring memory, maintaining versions and shutting everything down when you finish (and if you do not shut it down, you pay). With Glue you say "run this script with 2 workers", AWS brings up the infrastructure, runs it, destroys it and charges you for the seconds it used.
These are the pieces that matter:

The Glue Data Catalog

It is the sys.tables of the data lake. A central registry that says: "at the path s3://my-bucket/bookings/ there is a table called bookings with these columns, these types and these partitions".
This is more important than it looks. Files in S3 have no schema, they are files. The catalog is what puts structure on top of them, and once that structure is registered, every service consumes it: Glue, Athena, Redshift Spectrum, EMR, QuickSight. You register once, you query from wherever you want.

Crawlers

A crawler is a process you point at an S3 folder. It scans the folder, infers the schema and registers the table in the catalog. It is convenient and it is the fastest way to get started.

DynamicFrame vs DataFrame: the decision that confuses everyone

Glue introduces its own abstraction, the DynamicFrame, on top of Spark's native DataFrame. And the documentation is not always clear about when to use which.
A DataFrame is pure Spark. Fixed, known schema. The entire Spark ecosystem works on it, all the documentation in the world talks about it, and everything you learn here carries over to EMR, to Databricks or to local Spark.
A DynamicFrame is the Glue version, and its selling point is that it tolerates inconsistent schemas. If a column has numbers in some rows and text in others, a DataFrame forces you to decide the type up front; a DynamicFrame stores both possibilities as a "choice type" and lets you resolve it later with ResolveChoice. It is also what the native Glue connectors use and what enables job bookmarks.
My practical recommendation, after tripping over this:
Come in and go out with DynamicFrame, work inside with DataFrame.
That is: read with create_dynamic_frame (to take advantage of the catalog, the connectors and the bookmarks), convert immediately with .toDF(), do all your logic in pure Spark, which is where the transferable knowledge lives, and convert back only if you need the Glue sink to write.

Job bookmarks

You are going to like this one if you come from SSIS. A bookmark is the Glue mechanism for incremental processing: it remembers which files or which rows it already processed, and on the next run it only picks up what is new.
If you have built incremental ETLs by hand, you know the pain: a control table, a last-processed-date column, logic for reprocessing, handling failures mid-load. Bookmarks solve the common case of that for you. They are not magic and they have limits, but they save a good amount of plumbing code.

Versions, and why to check this before you start

Glue moves in versions, and each one packages a different version of Spark:
Glue versionApache SparkPython
6.04.1.13.13
5.1 (default)3.5.63.11
5.03.5.43.11
4.03.3.03.10
Glue 6.0 came out in August 2026 and brought Spark 4.1, full Apache Iceberg v3 support and a price reduction of around 30% compared to earlier versions. If you are starting from scratch today, start on the most recent version your case allows: you pay less and you do not inherit technical debt from day one.
Watch out for this when searching the internet: a lot of the tutorials you find are written for Glue 2.0 or 3.0, with syntax and behavior that changed. Versions 1.0, 2.0 and 0.9 reached end of life in April 2026.

3. How all the pieces connect

This is usually where it clicks. The pieces on their own do not say much; the complete flow does.
Read in words, the flow goes like this:
  1. The sources dump into the RAW zone. Untransformed, exactly as they arrived. This discipline looks unnecessary until the day you find a bug in your transformation logic and you need to reprocess six months of history. If you have the raw data, you reprocess. If you do not have it, you are going to ask a provider to resend you files from March and you are going to remember this paragraph.
  2. The crawler scans RAW and registers the schemas in the catalog. Now there is a queryable table where before there were loose files.
  3. The Glue Job reads, transforms and writes to CURATED. Your logic lives here: typing, cleaning, deduplication, joins, business rules. The output is no longer CSV: it is partitioned Parquet, a columnar and compressed format designed for analytics.
  4. Athena and the BI tools consume CURATED. And this is the nice detail: Athena does not need a server. It reads straight from S3, using the catalog to know what is there. You run SQL over files and you pay for the data scanned.
The catalog carries metadata, not data. The data always goes from S3 to Spark and from Spark to S3. The catalog only says "this is what exists and this is how it is organized". Separating those two things in your head helps enormously when debugging: when something fails, did the data read fail, or did the schema understanding fail? Those are different problems with different solutions.

4. Use cases where this really pays off

I am going to use examples from the sector I work in, which is corporate travel, plus one from financial services. These are the cases where I have seen Spark on Glue earn its place.

Case 1: consolidating bookings from multiple providers

The problem. Travel data arrives from sources that do not talk to each other: different GDS systems, airlines directly, hotel aggregators, expense management platforms. Each one sends its own format, its own airport naming, its own way of expressing dates and its own idea of what "cancelled booking" means. And the corporate client wants a single consolidated report of their travel spend.
Why Spark fits. There are many files, with heterogeneous structures, with different normalization rules per provider, and the volume grows every month. On a SQL server this turns into a monster of stored procedures with one branch per provider. In Spark you write one normalization module per source, they all produce the same canonical schema, and you combine them with unionByName. On top of that, processing files from 40 providers in parallel is exactly what Spark exists for.
The detail that matters. Deduplication. When a booking is modified three times, you receive three records with the same booking_id. You need the latest one. That is solved with a window function, and it is identical to what you would do with ROW_NUMBER() OVER (PARTITION BY ...) in T-SQL. Your SQL knowledge transfers completely here.

Case 2: normalizing a feed of financial transactions

The problem. A daily feed of transactions that has to be categorized, enriched with the exchange rates for the matching day, validated against control rules and left ready for regulatory reporting and reconciliation.
Why Spark fits. Enriching with exchange rates is a join against a small dimension table. Here comes one of the most useful tricks I learned: the broadcast join. If one of the two tables is small (a few hundred megabytes), Spark can send a complete copy to every executor and do the join locally, with no shuffle. The performance difference is an order of magnitude. An exchange rate table is the textbook case.
What to watch out for. Numeric precision. In financial data never use float or double for amounts, use decimal with explicit precision and scale. Floating point accumulates rounding errors, and a few cents off in a regulatory report is a conversation you do not want to have. This applies equally in SQL Server, but in Spark it is easier to get wrong because schema inference usually hands you a double by default.

Case 3: taking load off an SSIS ETL that cannot cope anymore

The problem. An SSIS package that moves and transforms a volume every night that no longer fits. It started taking twenty minutes, now it takes three hours, and every month it takes a little longer. And the worst part: it runs on the same server that serves user queries, so when the ETL suffers, everyone suffers.
Why Spark fits, and how to do it without dying. You do not migrate everything at once. The pattern that works is switching from ETL to ELT: instead of transforming inside the database server, you extract raw to S3, transform in Glue (separate compute, which scales on its own and bothers nobody) and load the already processed result. The database server goes back to doing what it does well: serving queries.
What is valuable here is not just the performance: it is separating compute from storage. Your database server stops being the bottleneck for everything. That is, at bottom, the architectural change this whole modern data engineering movement is about.

And now the honest part: when NOT to use Spark

Tutorials almost never say this, so I will.
If your data fits comfortably in your relational database, you do not need Spark. Spark has a startup cost, a coordination cost and a complexity cost. For a dataset of three million rows that your SQL Server processes in forty seconds with a well placed index, bringing in Spark is bringing a crane to move a chair. And it will be slower, not faster, because the cluster startup time eats any gain.
Spark starts to win when: the data does not fit in the memory of one machine, the sources are many and heterogeneous, or the volume grows in a way where you can no longer keep buying bigger servers.
Knowing when not to use a tool is part of the craft, and in a technical interview that answer is worth more than reciting the feature list.

5. Tutorial: your first Glue job, end to end

We are going to build something complete: take a CSV of travel bookings in S3, clean it, enrich it, write it as partitioned Parquet and query it with SQL. It is a small pipeline but a real one, with the same structure as a production one.

Step 0: what you need before you start

  • An AWS account (the free tier is enough to practice if you are careful with costs, and there is a section about that later).
  • An IAM role for the Glue job. It needs the AWSGlueServiceRole managed policy plus read and write permissions on your bucket. If you get an AccessDeniedException, 95% of the time the problem is here and not in your code.
  • An S3 bucket.

Step 1: organize the bucket

Before uploading anything, define the structure. This is not cosmetic: the folder organization determines how easy it is to query and reprocess later.
1
2
3
4
5
6
7
s3://my-travel-bucket/
raw/
bookings/
2026/09/16/bookings_20260916.csv
curated/
bookings/
scripts/
The raw zone partitioned by arrival date lets you reprocess a specific day without touching the rest. The curated zone is the clean output. And scripts holds your code, because Glue reads the script from S3.
The example CSV looks like this:
1
2
3
4
booking_id,booking_date,traveler_id,origin,destination,airline,fare_amount,currency,status,updated_at
BK00001,2026-09-01,TRV-104,bog,MIA,AV,1250.50,USD,CONFIRMED,2026-09-01T10:22:00
BK00002,2026-09-01,TRV-233, MDE ,PTY,CM,890.00,USD,CONFIRMED,2026-09-01T11:05:00
BK00001,2026-09-02,TRV-104,BOG,MIA,AV,1310.75,USD,MODIFIED,2026-09-02T09:15:00
Notice I left the garbage in on purpose: airport codes in lowercase, extra spaces, and a duplicated booking_id because the booking was modified. That is how real data arrives.

Step 2: the crawler

In the Glue console, Crawlers section, you create a new one:
  • Source: the path s3://my-travel-bucket/raw/bookings/
  • IAM role: the one you created
  • Target database: travel_raw (create it from the same wizard)
  • Run it
When it finishes you will have a table in the catalog. Open it and check the inferred types before moving on. It is almost certain that fare_amount came out as double and booking_date as string. That is normal and we are going to fix it in the script, but seeing it teaches you not to trust the crawler blindly.

Step 3: develop in an interactive session, not by launching jobs

This tip will save you hours. Do not develop your script by launching the full job every time: every run means bringing up infrastructure from scratch and waiting.
Use an interactive session (a Glue notebook). It brings up the environment once, keeps it warm and you run cell by cell, seeing results instantly. It is the difference between debugging in SSMS and debugging by compiling a whole project.
And the first thing you write in the notebook, before anything else, is this:
1
2
3
4
%idle_timeout 15
%number_of_workers 2
%glue_version 6.0
%worker_type G.1X
That %idle_timeout 15 is not optional. The default for an interactive session is 2880 minutes, forty eight hours. If you close the laptop and go to sleep, the session stays alive and keeps charging. With that magic, it shuts itself down after 15 minutes of inactivity.

Step 4: the Glue boilerplate

Every Glue job starts the same way. This looks like ceremonial noise, but every line does something:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
import sys
from awsglue.utils import getResolvedOptions
from awsglue.context import GlueContext
from awsglue.job import Job
from awsglue.dynamicframe import DynamicFrame
from pyspark.context import SparkContext
from pyspark.sql import functions as F
from pyspark.sql.window import Window

args = getResolvedOptions(sys.argv, ["JOB_NAME", "target_path"])

sc = SparkContext.getOrCreate()
glueContext = GlueContext(sc)
spark = glueContext.spark_session

job = Job(glueContext)
job.init(args["JOB_NAME"], args)
Broken down:
  • getResolvedOptions reads the parameters you pass to the job from outside. Never leave paths hardcoded: parameterize them and the same script serves you for development and production.
  • GlueContext wraps the SparkContext and adds the Glue specifics: reading from the catalog, sinks, bookmarks.
  • spark is the regular SparkSession you already know. It is the one you will use for 90% of the work.
  • job.init() and its partner job.commit() at the end are what manage the bookmark state. If you forget the commit, the bookmark never advances and on the next run you reprocess everything. It is a silent error: the job finishes "successfully" and you do not find out until you see duplicated data.

Step 5: read the data

1
2
3
4
5
6
7
8
9
10
dyf_raw = glueContext.create_dynamic_frame.from_catalog(
database="travel_raw",
table_name="bookings",
transformation_ctx="dyf_raw",
)

df = dyf_raw.toDF()

df.printSchema()
print(f"Rows read: {df.count()}")
We read from the catalog with a DynamicFrame and convert immediately to a DataFrame, following the recommendation from earlier. The transformation_ctx is the identifier bookmarks use to know which read they are keeping track of; if you are going to use incremental processing, it is mandatory and it has to be unique and stable.

Step 6: typing and cleaning

1
2
3
4
5
6
7
8
9
df_typed = (
df
.withColumn("booking_date", F.to_date(F.col("booking_date"), "yyyy-MM-dd"))
.withColumn("updated_at", F.to_timestamp(F.col("updated_at")))
.withColumn("fare_amount", F.col("fare_amount").cast("decimal(12,2)"))
.withColumn("origin", F.upper(F.trim(F.col("origin"))))
.withColumn("destination", F.upper(F.trim(F.col("destination"))))
.withColumn("status", F.upper(F.trim(F.col("status"))))
)
Here we fix what the crawler inferred wrong and normalize the dirty text. Note the decimal(12,2) for the amount, for the reason we discussed about financial precision.
Now the quality rules:
1
2
3
4
5
6
7
8
df_valid = df_typed.filter(
F.col("booking_id").isNotNull()
& F.col("booking_date").isNotNull()
& (F.col("fare_amount") > 0)
)

discarded = df_typed.count() - df_valid.count()
print(f"Rows discarded for quality: {discarded}")
Notice the syntax detail, because everyone makes this mistake starting out: in PySpark you use & and |, not and and or, and every condition has to be in parentheses. Python operators have different precedence and they will give you incorrect results or a cryptic error.
And that counter of discarded rows: do not skip it. An ETL that silently drops data is an ETL nobody should trust. Even if it is just a print going to the logs, leave a trace of how much you discarded.

Step 7: deduplication with window functions

We have duplicated booking_id values because bookings get modified. We want the most recent version of each one:
1
2
3
4
5
6
7
8
window = Window.partitionBy("booking_id").orderBy(F.col("updated_at").desc())

df_unique = (
df_valid
.withColumn("rn", F.row_number().over(window))
.filter(F.col("rn") == 1)
.drop("rn")
)
If you know T-SQL, you just read code you already knew:
1
2
3
4
SELECT * FROM (
SELECT *, ROW_NUMBER() OVER (PARTITION BY booking_id ORDER BY updated_at DESC) AS rn
FROM bookings
) t WHERE rn = 1;
It is exactly the same idea with different syntax. This kind of equivalence is why coming from SQL is an advantage and not a burden: the vocabulary changes, the concepts do not.
What you do need to know is that a window function causes a shuffle. Spark has to gather all the rows with the same booking_id into the same partition. It is unavoidable in this case and that is fine, but now you know where the execution time is going to come from.

Step 8: enrich with a broadcast join

We are going to convert amounts to dollars using an exchange rate table:
1
2
3
4
5
6
7
8
9
10
11
12
13
df_rates = spark.read.parquet("s3://my-travel-bucket/dim/exchange_rates/")

df_enriched = (
df_unique
.withColumn("year", F.year(F.col("booking_date")))
.withColumn("month", F.month(F.col("booking_date")))
.withColumn("route", F.concat_ws("-", F.col("origin"), F.col("destination")))
.join(F.broadcast(df_rates), on=["currency", "year", "month"], how="left")
.withColumn(
"fare_usd",
F.col("fare_amount") * F.coalesce(F.col("rate_to_usd"), F.lit(1.0)),
)
)
The F.broadcast() is the detail worth gold. It tells Spark: this table is small, send a complete copy to every executor and do the join in memory, without redistributing the big table. Without it, Spark shuffles both sides of the join. With it, the big table does not move.
The practical rule: if one of the tables in the join is small, a dimension, a catalog, a code mapping, mark it with broadcast. In real pipelines this is one of the optimizations with the best effort to result ratio.
The coalesce in the calculation is defensive: if a currency has no rate registered for that month, the left join leaves null, and null times anything is null. Without that coalesce, a missing rate silently turns amounts into nulls.

Step 9: write partitioned Parquet

Two ways to do it, and the difference matters.
Option A: Spark's native writer.
1
2
3
4
5
6
7
8
(
df_enriched
.repartition("year", "month")
.write
.mode("overwrite")
.partitionBy("year", "month")
.parquet(args["target_path"])
)
Option B: the Glue sink, which also updates the catalog.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
dyf_out = DynamicFrame.fromDF(df_enriched, glueContext, "dyf_out")

sink = glueContext.getSink(
path=args["target_path"],
connection_type="s3",
updateBehavior="UPDATE_IN_DATABASE",
partitionKeys=["year", "month"],
enableUpdateCatalog=True,
transformation_ctx="sink",
)
sink.setCatalogInfo(
catalogDatabase="travel_curated",
catalogTableName="bookings",
)
sink.setFormat("glueparquet", compression="snappy")
sink.writeFrame(dyf_out)
Option A writes files and nothing else. Option B writes files and registers or updates the table in the catalog in the same operation, including the new partitions. With option A you would have to run a crawler again afterwards so Athena sees the new data; with B it is already visible.
Three things from this section you should understand well:
Why Parquet and not CSV. Parquet is columnar: if you query two columns out of a table of fifty, it only reads those two. It is compressed, it stores the schema and the types inside, and it carries per block statistics that let it skip entire files. In analytical queries the difference against CSV is orders of magnitude, both in time and in cost, and in Athena you pay per byte scanned, so the format translates directly into money.
Why partition by year and month. When you partition by a column, Spark creates one folder per value (year=2026/month=9/). When you query with a filter on that column, the engine only reads those folders. It is called partition pruning and it is the conceptual equivalent of an index for data lakes.
The key is choosing the column well: it has to be something you filter by often and that has reasonable cardinality. Year and month: perfect, dozens of values. booking_id: catastrophe, a million folders with one tiny file each.
The repartition before writing. Without it, every Spark partition writes its own file into every output folder, and you end up with hundreds of small files per month. With repartition("year", "month") you group each month's data into the same Spark partition and one file per month comes out. This connects with the small files problem, which is in the mistakes section.

Step 10: close the job

1
job.commit()
One line, easy to forget, indispensable. It commits the job and advances the bookmarks.

Step 11: query the result

In the Athena console:
1
2
3
4
5
6
7
8
9
10
11
SELECT
route,
airline,
COUNT(*) AS bookings,
SUM(fare_usd) AS spend_usd,
AVG(fare_usd) AS avg_ticket
FROM travel_curated.bookings
WHERE year = 2026 AND month = 9
GROUP BY route, airline
ORDER BY spend_usd DESC
LIMIT 20;
And there is the nice part: you are running SQL over files in S3, with no database server in the middle. That WHERE year = 2026 AND month = 9 is doing partition pruning, Athena only reads the September folder. Remove the filter and watch how the amount of scanned data reported by the console changes. It is the partitioning lesson live.

Step 12: turn it into a scheduled job

What you ran in the notebook, now as a real job:
  1. Upload the script to s3://my-travel-bucket/scripts/
  2. In Glue, create a job pointing at that script
  3. Configure it: Glue version 6.0, G.1X worker, 2 workers to start
  4. Add the parameter --target_path with the value s3://my-travel-bucket/curated/bookings/
  5. Enable bookmarks if you are going to process incrementally
  6. Schedule a trigger if you want it to run on its own
And you have a complete serverless pipeline. When you want to orchestrate several jobs with dependencies between them, that is where Airflow comes in (managed on AWS it is called MWAA) or Glue Workflows. But that is a topic for another article.

6. The beginner mistakes that really hurt

This is the section I wrote with the most enthusiasm, because every point cost me time, patience or money.

1. Bringing everything to the driver

1
2
3
# Bad. Very bad.
data = df.collect()
pdf = df.toPandas()
collect() and toPandas() bring every row from the cluster into the driver's memory. You are back to a single machine, and one that is not sized for that. The typical result is a driver OutOfMemoryError that takes down the whole job after twenty minutes of work.
These functions are legitimate, but only for already reduced results: the output of a groupBy with twenty rows, a sample to inspect. To look at data during development, use df.show(20) or df.limit(100).toPandas().

2. The small files problem

Your job runs fine, writes the output, and you discover it generated 4,000 Parquet files of 30 KB each. The queries afterwards crawl, because opening 4,000 files in S3 is 4,000 network requests and each one has its own latency.
It is solved by controlling the partitions before writing: repartition(n) (with shuffle, distributes evenly) or coalesce(n) (no shuffle, just merges, cheaper but can end up unbalanced). As a reference, aim for files between 128 MB and 1 GB.

3. Partitioning by the wrong column

I mentioned it above but it deserves its own entry because it is an expensive mistake to reverse. Partitioning by a high cardinality column (an ID, a full timestamp, an email) generates an explosion of folders with one tiny file each. You combine the two worst problems in a single move.
Partition by columns you filter on and that have low or medium cardinality: date at month or day level, country, line of business.

4. Not caching what you reuse

Remember lazy evaluation: every action re-executes the complete plan from the source.
1
2
3
4
# This reads and transforms the source THREE times.
print(df_transformed.count())
df_transformed.filter(F.col("status") == "CONFIRMED").show()
df_transformed.write.parquet(path)
If you are going to use a DataFrame several times, cache it:
1
2
3
4
5
df_transformed.cache()
print(df_transformed.count()) # materializes here
df_transformed.filter(...).show() # uses the cache
df_transformed.write.parquet(path)
df_transformed.unpersist() # release when you finish
But do not cache by reflex: the cache takes memory the executors need to work. Cache what you genuinely reuse, and release it with unpersist().

5. Writing Python UDFs when the native function already exists

It is tempting to solve everything with a Python function and a udf(). It is also one of the most efficient ways to make your job five times slower.
The native functions in pyspark.sql.functions execute inside the JVM and the optimizer understands them. A Python UDF forces serializing every row, sending it to a Python process, processing it and sending it back, and the optimizer treats it as a black box it cannot optimize.
Before writing a UDF, search the pyspark.sql.functions documentation. It almost always already exists. If you genuinely need custom logic, vectorized UDFs (Pandas UDFs) are the way: they process in batches instead of row by row.

6. The overwrite that takes the whole table with it

This one is genuinely scary:
1
df.write.mode("overwrite").partitionBy("year", "month").parquet(path)
With the default static behavior, this deletes the entire target directory and writes only what your DataFrame carried. If you were reprocessing only September, you just deleted January through August.
The solution is dynamic partition overwrite mode, which only replaces the partitions present in your DataFrame:
1
spark.conf.set("spark.sql.sources.partitionOverwriteMode", "dynamic")
Set it before writing, and test on a development path first. S3 has no recycle bin.

7. Oversizing just in case

The temptation to set 20 workers so it runs fast is real, and it is a waste in two directions: you pay twenty times more, and if your dataset has few partitions, half of those workers do absolutely nothing.
Start with 2 G.1X workers. Measure. If the job takes too long, raise it and measure again. Scale is earned with evidence, not with optimism.

8. Debugging with print() instead of reading the Spark UI

print() works, and it goes to CloudWatch Logs. But when the problem is performance, print tells you nothing.
The Spark UI shows you the real execution plan, how long each stage took, how much data moved in each shuffle and whether there are unbalanced partitions. Glue lets you enable it per job and review it after the run.
Learning to read the Spark UI is what separates someone who uses Spark from someone who understands what is happening. It is the equivalent of reading an execution plan in SQL Server: at first it looks like hieroglyphics, and afterwards you do not know how you worked without it.

9. Trusting schema inference in production

The crawler is excellent for exploring. But in a pipeline that runs on its own every night, inference is a time bomb: the day a file arrives with an extra column, or with a different type, the schema changes under your feet and your logic breaks, or worse, it does not break and produces incorrect results.
In production you declare the schema explicitly and you fail loudly when the data does not comply. A job that fails with a clear message is infinitely better than one that works and delivers garbage.

7. Cost tips, or how not to get a shock with the bill

If you are learning on your own account, this section matters more than any other.
Understand the model. Glue charges per DPU-hour. A DPU (Data Processing Unit) is a compute unit: a G.1X worker is 1 DPU with 4 vCPU and 16 GB of memory; a G.2X is 2 DPU with 8 vCPU and 32 GB. Billing is per second, with a one minute minimum per run. The exact rate varies by region, so check it for yours, and remember Glue 6.0 ended up around 30% cheaper than earlier versions, which is a solid argument for using the new version.
The interactive session timeout is your biggest risk. I repeat it because it is the most expensive mistake a student can make: the default idle timeout is 48 hours, and a session starts with 5 DPU. A session forgotten over a weekend is a number you do not want to see. Put %idle_timeout 15 in the first cell, always, and use %stop_session when you finish.
Two workers are enough for learning. You are practicing syntax and concepts, not processing terabytes. With 2 G.1X workers you can do this entire tutorial.
Use Flex for anything that is not urgent. Flex execution uses spare AWS capacity at a noticeably lower rate. The price is that the job may take longer or start later. For development, testing and overnight loads with no SLA, it is free money.
Take advantage of the Data Catalog free tier. The first million objects stored and the first million accesses per month are not charged. For practice, you are not going to leave that range.
Develop locally before uploading. You can run PySpark on your machine, or use the official Glue Docker image, and debug the transformation logic without spending a cent. Upload it to Glue when it already works. You also develop faster, because there is no startup latency.
Set a budget alert right now. AWS Budgets emails you when spend passes the threshold you define. Configure it at five dollars before creating your first job. It is the cheapest safety net there is.

8. Where to go next

If you completed the tutorial, you already have the base: you understand Spark's execution model, you know what Glue adds on top and you built an end to end pipeline. That is already much more than it sounds.
What I have on my own list, in the order I am working through it:
Read the Spark UI seriously. Before any new tool. Understanding stages, tasks, shuffles and partition skew is what lets you optimize with judgment instead of trial and error.
Open table formats: Apache Iceberg. This is the next conceptual jump. Iceberg brings the data lake things you take for granted in your database: ACID transactions, schema evolution without rewriting everything, time travel to query yesterday's state. Glue 6.0 ships full Iceberg v3 support. If you already know SQL, this will feel familiar and liberating at the same time.
Orchestration with Apache Airflow. One isolated job is an exercise; a real pipeline is twenty jobs with dependencies, retries, alerts and execution windows. Airflow is the industry standard and on AWS it is managed as MWAA.
Dimensional modeling applied to the lake. This is where database experience becomes a competitive advantage. Facts, dimensions, type 2 SCDs, the concepts did not change because the technology changed. A lot of people arrive at Spark knowing how to distribute compute but not how to model, and they produce data lakes nobody can query. If you come from the relational world, you already have that muscle.

Closing

What surprised me most about this process was how much of my SQL knowledge turned out to be transferable instead of obsolete. Window functions are the same. Joins are the same. Thinking about partitioning is thinking about indexes with another name. Reading an execution plan is reading an execution plan.
What changes is the scale and the execution model. And that is learnable: it is not esoteric knowledge, it is a new mental model you build by writing code, breaking things and reading logs.
If you are in the same transition, my advice is the most boring one possible: do the full tutorial, with your own data, even if it is a CSV of twenty rows you made up. Reading about Spark and writing Spark are two different activities, and only the second one teaches. The moment you truly understand lazy evaluation is the moment a count() takes you three minutes and you have to figure out why.
Let it break. That is how you learn.
And I would like to read where you are coming from. SSIS, Informatica, Talend, Pentaho, or a folder full of stored procedures? Which habit was the hardest one for you to unlearn? If you are just starting and something is not clicking yet, say so in the comments. You are not the only one.
This guide is also available in Spanish: https://builder.aws.com/content/3JQoVykhOSV0CIzeSIwKUhGB9G4/como-empezar-con-spark-en-aws-glue-lo-que-aprendi-viniendo-de-sql
# aws-glue# apache-spark# data-engineering# etl# getting-started
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article