
I Built a Workflow Engine from Scratch. Here's How It Went.
I had no idea what I was getting into.
Why I Built It
What happens when your backend needs to run tasks in order — and something breaks halfway through?
Like this:
Validate Payment → Reserve Inventory → Notify Warehouse → Email Customer Step 3 works. Server crashes. Step 4 never runs. Now what?
I didn't have a clean answer. So I built one. That's Atlas.
What It Does
Atlas is a system that runs workflows. You define tasks and their dependencies. Atlas schedules them in the right order, runs them across distributed workers, retries failures automatically, and recovers if a machine crashes — without you touching anything.
The Stack
- Python + FastAPI — backend API
- PostgreSQL — stores everything (single source of truth)
- Redis — task queue between the scheduler and workers
- Next.js — real-time operator dashboard
- Docker + AWS — production deployment
The Hard Part Nobody Talks About
Handling failures is 80% of the work. The happy path took a week. The rest of the time I was asking: what if the worker dies mid-task? What if two workers grab the same task?
My solution: every worker gets a 30-second lease. It renews every 10 seconds with a heartbeat. If it goes quiet — the system assumes it crashed and gives the task to another worker automatically. No human needed.
On AWS
The backend runs on EC2 inside Docker containers. The database is RDS (PostgreSQL) and the queue is ElastiCache (Redis) — both in a private VPC subnet, invisible to the public internet. The frontend deploys automatically on AWS Amplify every time I push to GitHub.
What I Learned
Distributed systems are about failure, not success. I spent way more time on "what could go wrong" than "how does this work normally."
Also — a single SQL
UPDATE ... WHERE status = 'READY' is enough to prevent two workers from ever claiming the same task. No external locking. PostgreSQL just handles it. That blew my mind a little.Build things you don't fully understand yet. You'll figure it out by breaking them.
Atlas is live. 74 tests passing. Workers heartbeating. Tasks completing.
🔗 Live Demo — https://main.d1xijusitagxbu.amplifyapp.com/
🔗 API Docs — https://atlas-api.abhinav.sbs/docs
🔗 GitHub — github.com/Abhinav-0709/atlas
Thanks for reading.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article