Coming from SQL Server to AWS Glue, three habits I had to unlearn

I work with SQL Server and SSIS every day. I build packages that move travel transaction data, and at night I study data science in Bogotá while I learn the AWS side of things. A few weeks ago I opened AWS Glue for the first time expecting to recognize something. I did not.

That feeling is the reason I am writing this. Not as a tutorial, there are already very good ones here, but as a short note from someone who is still in the middle of the change. If you are coming from a traditional ETL tool, maybe some of this sounds familiar.

In SSIS everything lives in one package. The control flow decides what runs and in what order, the data flow moves the rows, and both sit in the same file. In AWS that is split on purpose. The orchestration goes to Step Functions, the transformation goes to a Glue job running Spark. My first mistake was hunting for the one screen that did both. There is no such screen, and the day I accepted that, everything else got easier.

This one took me longer than I want to admit. In SSIS I think in rows. A Script Component receives one row. An error output redirects that row. In Spark the unit is the partition, not the row, and there is no "send this bad record over there". What replaced it for me was splitting the data on purpose, valid on one side, rejected on the other, and writing the rejected part to its own prefix in S3. It felt wrong at first. Now I prefer it, because my rejects became a table I can query instead of a log nobody opens.

The thing I missed most was not a feature, it was the feedback. Right click an arrow, attach a data viewer, press F5, watch the rows go by. Spark is lazy, so there is nothing to look at until something forces it to run. I had to learn explain, the Spark UI, and writing intermediate results to a scratch folder so I could query them. I will be honest, some days I still miss the green arrows. What I got in return is that the whole pipeline is text now, so it lives in git, a teammate reviews it, and a bad change can be undone.

If you are starting this same move, these are the ones that saved me the most time. First, stop translating package by package and translate concepts instead, a Lookup with full cache is a broadcast join, a Conditional Split is a filter, an Aggregate is a groupBy. Second, learn window functions early, because deduplication, latest record per key, and slowly changing dimensions all collapse into them once they click. Third, make every job safe to run twice. An SSIS package usually runs once and a person restarts it if it fails, but a Glue job can be retried automatically, so appending blindly will quietly duplicate your data and nobody notices until a report looks wrong.

None of that is advanced, and that is kind of the point. The hard part was never the syntax, it was letting go of a mental model that had worked for me for years.

Incremental loads. In SSIS I keep a watermark in a control table, I read it, I use it, I update it at the end. Glue has job bookmarks that do something similar with much less code, but I am not sure how they behave when a file arrives late carrying an older business date. I have read the documentation twice and I still want to hear it from someone who has run it in production.

I am sharing this because I think many of us are in the middle of a migration like this one, and we do not talk enough about the confusing part. We publish the tutorial after we figured it out, never the three weeks when nothing made sense.

So I would really like to read your story. Where are you coming from, SSIS, Informatica, Talend, Pentaho, or a folder full of stored procedures. Which habit was the hardest one for you to unlearn, and what finally made it click.

And if you are at the beginning and something is not clicking yet, say it in the comments. I promise you are not the only one, and I am happy to think it through together.