AWS Builder Center

Builder Center

Pre-Flight Your Data Pipelines on Aurora PostgreSQL

Pre-Flight Your Data Pipelines on Aurora PostgreSQL

ZeroGuard is a visual pipeline editor with a built-in pre-flight check. It compiles your DAG into one SQL statement, runs EXPLAIN (FORMAT JSON) against Amazon Aurora PostgreSQL to forecast real per-query cost, and traces PII lineage from source to sink. Catch a $5,000 Cartesian-join bill and a GDPR-breaching email leak in the same click—then fix the leak with one-click SHA-256 hashing. Built with Vercel v0 for the H0 Hackathon. #H0Hackathon

Disclosure: I built ZeroGuard and wrote this article for the H0 Hackathon ("Hack the Zero Stack with Vercel v0 and AWS Databases"). If you share it, the hashtag is #H0Hackathon.
Try it live: https://zeroguard-data-pipeline-builder.vercel.app/ 
Three-minute demo: https://youtu.be/5wud78_RkR0 
Source code: https://github.com/TLiu2014/zeroguard-data-pipeline-builder 

The problem nobody catches in code review

Every commercial flight runs the same checklist before takeoff: engines, flaps, fuel, instruments. Data pipelines don't get that treatment. They get code-reviewed, merged, and shipped — and then the failures show up somewhere else and much later: in next month's cloud invoice, in an incident channel at 2 a.m., or in a compliance finding raised by a team that wasn't even in the review.
I kept running into the same two flavors of post-merge bug:
  • A join with a missing key on a customer-scale table. On paper it looks fine; in practice it's a Cartesian product, and the bill arrives thirty days later.
  • A column of personal data — an email, a phone number — that quietly flows through a transformation and lands in a marketing table or an analyst's spreadsheet. The math is correct, but you've just created a GDPR problem.
What bothered me most is that both bugs are visible in the pipeline definition itself. You don't need production data to find them. You need a tool that reads the pipeline before it runs and says no. That tool is ZeroGuard.

What ZeroGuard does

ZeroGuard is a visual editor for data pipelines with a pre-flight checker built into the canvas. You compose a pipeline as a directed graph of typed stages — load, filter, join, group by, sort, select, custom SQL — by dragging nodes around a React Flow canvas. Then you hit Run Pre-Flight Check, and in a single round trip it does three things.
  1. Compiles the entire graph into one SQL statement. Every stage becomes a common table expression, and the terminal node becomes the outer SELECT.
  2. Runs EXPLAIN (FORMAT JSON) against Amazon Aurora PostgreSQL and parses the planner's row estimates and total cost. The verbatim plan is shown in an expandable panel, so there's no hand-waving — you can read exactly what the planner said.
  3. Traces data lineage by walking the graph from every source column to every output sink. Any column that looks like personal data and reaches a sink without first passing through a hashing or masking stage produces a critical finding.
Out of those three steps come three categories of issue:
  • Runaway cost — the cost card shows a real dollar estimate and the offending node glows red on the canvas.
  • Compliance breaches — the security score drops and you get a one-click Auto-Fix Compliance button that inserts a SHA-256 hashing stage upstream and rewires the graph for you.
  • Silent data corruption — broken column references, joins on nonexistent columns, type mismatches that pass code review and then quietly produce wrong numbers.

Why Aurora PostgreSQL — and why the cost model is the interesting part

The database is Amazon Aurora PostgreSQL Serverless v2. There's no AWS-proprietary code on the hot path: ZeroGuard talks to Aurora through Drizzle ORM over a standard node-postgres connection pool, using the same SQL, the same EXPLAIN, and the same information_schema you'd hit in any Postgres. Aurora is the deployment target, not a lock-in, and that portability is deliberate.
The part I'm most opinionated about is the cost model. Aurora Serverless v2 on the Standard configuration bills two things: compute at $0.12 per ACU-hour, and storage I/O at $0.20 per million I/O operations. To turn a query into a dollar figure, ZeroGuard estimates a compute term from rows, throughput per ACU, and the number of ACUs in use, plus an I/O term from rows times an I/O-per-row factor:
1
2
3
cost = compute + io
compute = (rows / rows_per_sec_per_acu / acus) × ($/ACU-hour / 3600) × acus
io = (rows × io_per_row / 1,000,000) × $/M-I/Os
Here's the lesson that surprised me. For large scans on Aurora Standard, the I/O term dominates — not compute. The famous "$5,000 typo" in the demo is the I/O term doing the damage:
1
50,000,000,000 rows × 0.5 I/Os/row × ($0.20 / 1,000,000) ≈ $5,000
The compute for that same runaway query — five hours at the ACU ceiling — is under ten dollars. If you're cost-modeling Aurora workloads, the I/O term is where the dollars live.
I also kept the real AWS rates and the empirical knobs separate in the source. AWS publishes the dollar rates, but it doesn't publish "rows per ACU-second" or "I/Os per row" — those depend on your data, your indexes, and your cache hit rate. So the code clearly distinguishes "real rates, don't touch" from "empirical, calibrate to your workload," and the app surfaces a costBasis label on every estimate (live-explain, heuristic, or cartesian-pinned) so a user always knows whether a number came from a live Aurora EXPLAIN, a graph-shape heuristic, or a pinned disaster scenario. Honesty about where a number comes from matters more than a single confident-looking estimate.

Three demos, three complete stories

The workspace ships with one-click samples.
Revenue by region is a real query — join customers and orders, keep active customers, total revenue by region. It runs a genuine EXPLAIN against Aurora, scores a clean 100 / 100 because the email column is dropped before the output, and lets you expand the raw planner JSON to prove it.
Cartesian join is the five-thousand-dollar typo. A join with no key: fifty billion rows scanned, execution blocked, and the offending node pulsing red on the canvas — all before a single byte is processed in production.
PII leak to marketing is the GDPR story. Customers joined with orders, grouped by total spent, and a final SELECT that pipes raw customer_email into a marketing staging table. The lineage check fires, the security score drops to 60, and one click on Auto-Fix Compliance inserts a SHA-256 stage upstream and rewires the graph. Re-run pre-flight, and the score is back to 100. No code edited, no SQL written.

A few things I learned connecting to Aurora

Connecting a Next.js app to Aurora over TLS had one sharp edge worth sharing. Modern node-postgres treats sslmode=require in the connection string as verify-full, which means it tries to verify the server certificate chain. Aurora presents a certificate signed by the Amazon RDS CA, which isn't in Node's default trust store, so the handshake fails with unable to get local issuer certificate — and the app reports the database as offline even though it's perfectly reachable. The fix is to configure SSL explicitly in the pool: either verify against the Amazon RDS CA bundle for full verification, or encrypt without chain verification to get connected quickly. Once I handled SSL deliberately instead of leaning on the connection string, the live EXPLAIN path lit up.
The other lesson is about EXPLAIN itself. The planner gives you Plan Rows and Total Cost, but Total Cost is in planner units, not dollars — converting to currency means modeling ACU-time and I/O yourself. And Plan Rows from a table that has never been analyzed is essentially a guess, since Postgres defaults to one row when statistics are missing. My seed scripts end with ANALYZE for exactly this reason. Without it, the whole cost path silently collapses to "everything is one row."

How it fits together

A Next.js front-end and three API routes — a health check, a schema lookup, and the pre-flight simulator that compiles your pipeline to SQL and runs EXPLAIN — with Drizzle over a standard Postgres connection pool talking to Amazon Aurora PostgreSQL. Same SQL, same EXPLAIN you'd hit in production.

Built on the Zero Stack

ZeroGuard was scaffolded with Vercel v0 and deployed on Vercel, with Amazon Aurora PostgreSQL as the data foundation. That's the whole premise of the H0 Hackathon: prototype a front-end in minutes, and connect it to a database that's ready for real-world traffic from day one. The thing I shipped over a weekend runs on the same data infrastructure I'd reach for in production.
If you want to break a pipeline on purpose, open the live app, drop one of the disaster samples onto the canvas, and watch the offending node light up:
Built for the H0 Hackathon. #H0Hackathon
# amazon-aurora# postgresql# nextjs# hackathon
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article