
Building a Reliable Daily Reporting Pipeline with AWS Glue, S3, Python, and Supabase
A practical look at how I used AWS Glue, Amazon S3, Python, Supabase PostgreSQL, and workflow automation to turn data from multiple business systems into dependable daily reporting.
A data pipeline can run successfully and still produce bad data.
That is one of the first things real production work teaches you.
A job can finish with a green status while a source is missing records, dates fall into the wrong reporting period, or a rerun quietly duplicates yesterday's numbers.
I worked on a daily reporting workflow where the goal was not simply to move data from one system to another. The goal was to make data from several business platforms consistent enough that the reporting layer could use it without someone manually fixing the same issues every day.
At a high level, the workflow looked like this:
Business Sources → AWS Glue / Python → Supabase PostgreSQL → Reporting
Amazon S3 supported the Glue jobs by holding the ETL scripts, while n8n and Slack helped automate parts of the operational workflow.
The architecture was not unusually complicated. Making it dependable was the interesting part.
Starting With Multiple Business Sources
The reporting process brought together data from systems such as WooCommerce, Shopify, Amazon, Meta Ads, Google Ads, GA4, and supporting cost sheets.
Each source had a different job.
Commerce platforms provided orders, products, refunds, and sales activity.
Advertising platforms provided campaign performance such as spend, clicks, conversions, and revenue-related metrics.
GA4 added website activity.
Supporting sheets filled in business data such as product costs and other values used in reporting.
The problem is that these systems do not naturally agree with one another.
A date may be represented differently across sources. Product identifiers may not match. One platform may treat refunds differently from another. Some records may arrive late, while others may need additional mapping before they are useful.
The reporting layer should not have to solve those problems every time someone opens a report.
That work belongs upstream.
AWS Glue and Python Handled the ETL
AWS Glue handled the main processing work, with Python used for the transformation logic.
The jobs took incoming data and prepared it for reporting by handling things such as:
- Date normalization
- Missing or inconsistent values
- Product and category mappings
- Duplicate handling
- Business-rule transformations
- Reporting fields and calculated values
- Validation before loading
None of these steps sounds especially complex by itself.
The difficulty comes from doing them consistently across several sources and making sure tomorrow's run behaves the same way as today's.
That is where ETL stops being a script and starts becoming a production workflow.
Where Amazon S3 Fit In
Amazon S3 was used to store the Python scripts used by the Glue jobs.
The relationship was straightforward:
Python Script → Amazon S3 → AWS Glue
Keeping the ETL code in S3 meant the Glue job could reference the required script without relying on a local machine.
When the transformation logic changed, the script could be updated and the job could run with the new version.
It was a simple part of the setup, but it kept the processing code separate from someone's workstation and made the workflow easier to manage.
Supabase Became the Reporting Data Layer
After the data was cleaned and transformed, it was loaded into Supabase, using PostgreSQL underneath.
This gave the reporting process a central place to work from.
Instead of every report reaching directly into WooCommerce, Shopify, Amazon, advertising platforms, and spreadsheets, the data could first be standardized and stored in a structure designed for reporting.
That separation made a noticeable difference.
The source systems were responsible for generating operational data.
The ETL process was responsible for cleaning and organizing it.
Supabase held the reporting-ready version.
The reporting layer consumed it.
Each part had a clear responsibility & it also made troubleshooting easier.
If a number looked wrong, I could trace it backward:
Report → Supabase → Transformation → Source
That is usually much faster than staring at the final dashboard and guessing where something went wrong.
A Successful Rerun Shouldn't Change the Numbers
One of the easiest ways to damage a reporting pipeline is to make it unsafe to rerun.
Suppose yesterday's orders load correctly.
Later, the same job runs again.
If the pipeline blindly inserts every record a second time, the job may still report success while the business numbers are now wrong.
Revenue and unit counts can double, and conversion metrics can become misleading.
Nothing technically "failed," but the data did.
For datasets that could be refreshed, the workflow needed a predictable way to rebuild or replace the relevant period instead of repeatedly adding the same information.
For other cases, records needed to be matched carefully before updating or inserting them.
The exact technique depends on the dataset.
The principle does not:
A production pipeline should be safe to rerun.
That becomes especially important when a job fails halfway through and needs to be restarted.
Dates Can Cause Quiet Problems
Dates are another area where a pipeline can look healthy while producing the wrong result.
Daily reporting often depends on windows such as:
- Yesterday
- Last seven days
- Last thirty days
- Current month
- Previous month
Now add source timestamps and different time zones.
A small mistake in date handling can move transactions into the wrong day or month without causing the job itself to fail.
That is why I treat date logic as part of data quality rather than simple formatting.
A successful pipeline run is not enough.
The records also have to land in the right reporting period.
One Issue Wasn't an ETL Problem at All
One of the more useful troubleshooting lessons came from data that was expected in a report but simply was not there.
The natural assumption was that something in the transformation logic had filtered it out.
So I checked the reporting output -> then the database -> then the ETL logic.
Eventually, I traced the issue back to the upstream dataset.
The required records were never present in the source feeding the pipeline.
The ETL job had not lost them & it had never received them.
That sounds simple in hindsight, but it is an important distinction.
When data is missing, changing code immediately can make the situation worse.
I now prefer to trace the data backward first:
Where does the value disappear?
Once you know that, you know which part of the system actually needs attention.
Automation Should Reduce Manual Checking
The workflow also used n8n and Slack notifications around the reporting process.
That may sound like a small detail, but it matters operationally.
A pipeline is only partly automated if someone still has to open AWS every morning and check whether it completed.
A status message gives the team a quick signal that the workflow ran successfully or that something needs attention.
The goal is not to create more notifications but it is to remove unnecessary checking.
If the process works, people should be able to move on with their day.
If it fails, they should know where to start looking.
What This Workflow Reinforced for Me
The individual tools were useful, but the bigger lesson was how the pieces worked together.
AWS Glue ran the processing.
Python handled the transformation logic.
Amazon S3 stored the ETL scripts.
Supabase PostgreSQL held the reporting-ready data.
The reporting layer consumed that cleaned data.
n8n and Slack helped close the operational loop.
What mattered most was not the number of services involved.
It was whether the workflow could answer a few basic questions reliably:
Did the pipeline run?
Did it process the right data?
Can it be rerun safely?
If a number is wrong, can I trace it back to the source?
Those questions are much closer to day-to-day data engineering than building the most impressive architecture diagram.
What I Would Improve Next
The next area I would strengthen is automated data-quality validation.
A job showing SUCCESS does not necessarily mean the dataset is healthy.
Useful checks could include:
- Missing reporting dates
- Duplicate records
- Unexpected row-count changes
- Null values in critical fields
- Missing product mappings
- Unusual changes in sales or revenue
I would also continue improving monitoring so failures are easier to diagnose without digging through multiple systems.
The goal would be simple: catch bad data earlier and make troubleshooting faster.
Final Thoughts
Real data pipelines are rarely difficult because one AWS service is hard to understand.
They are difficult because real data is messy.
Sources change. Records arrive late.
Mappings are incomplete. Jobs need to be rerun.
A source can fail while the rest of the pipeline continues normally.
Building a useful pipeline means accounting for those situations instead of assuming everything will arrive perfectly every day.
For this workflow, AWS Glue, S3, Python, Supabase, and automation tools provided the technical foundation.
The real value came from turning those pieces into a process that could produce reporting people could actually rely on.
That is the part of data engineering that matters most to me.
Not simply moving data from one place to another.
Making sure it arrives in a form that people can trust.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article