Waylo: How We Built a Self-Improving AI Guide That Gets Cheaper Every Time Someone Uses It

H0 Hackathon. #H0Hackathon

Teammate: Yash Aggarwal @yash4428

Waylo is a macOS menu bar app that guides anyone through any software task using a pulsing red dot overlay and voice instructions, step by step, with zero technical knowledge required.

You type a task. The red dot appears on the exact element you need to interact with. You click. The dot moves to the next step. You follow it until the task is done.

No manuals. No YouTube tutorials. No calling someone for help.

We didn't start with a technical problem. We started with a frustrating observation.

My grandmother calls every week - not to chat, but to ask how to attach a photo to an email. Every time, we walk her through the same steps. Every time, she forgets by next week. As computer engineering students, we face a different but equally real version of this - every new semester brings new software, new tools, new workflows that nobody formally teaches you. Whether it's learning Excel formulas at 2am before a deadline or explaining Gmail to a parent over the phone, the frustration is the same: software assumes you already know how to use it.

We asked ourselves: what if instead of explaining where to click, the app just showed you? A dot. Right on the thing you need to click. Move. Click. Next step. That's Waylo.

For our demo, we tested this with two real tasks - how to click a picture and send it on WhatsApp, and how to deploy a GitHub repo on Vercel. Two completely different users, two completely different skill levels, same red dot guiding both through from start to finish.

Try it out: https://waylo-web-virid.vercel.app/

Waylo has five layers working together - the macOS Swift app captures the screen and handles the red dot overlay, the Node.js backend on AWS EC2 orchestrates the logic, AWS Bedrock runs the AI (Amazon Nova Micro for step plans and voice Q&A, Nova 2 Lite for vision grounding), and Amazon Aurora PostgreSQL with pgvector stores everything - the semantic plan cache, step label cache, and user-created guides. The Next.js 14 frontend is deployed on Vercel for the analytics dashboard.

The key design principle: every layer tries to solve the problem for free before passing it to the next, more expensive layer.

This is where things got interesting - and expensive.

Early versions called AWS Bedrock Nova for every single step detection. The latency was ~800ms per step and the cost was unsustainable at any real scale. With real users doing hundreds of different tasks across dozens of apps, the cost would scale linearly. That's not a sustainable model.

We also noticed something subtler: the same task gets asked in slightly different ways. "Freeze the top row" and "lock the first row in Excel" are the same thing. A naive cache would miss that match entirely and make two separate Nova calls, paying twice for the same answer.

The fix was making every Nova call an investment rather than just an expense.

We built a 4-layer pipeline where each layer is cheaper than the last, and every paid call at Layer 3 automatically populates the cache so future users never pay for the same thing again.

L0 - Accessibility Tree (~5ms, free): Reads the target app's native accessibility tree via AXUIElement. Handles all text labels, menu items, and standard buttons instantly with zero API cost.

L1 - On-Device OCR (~40ms, free): Apple Vision framework scans the screen for text matches. Handles labeled UI elements that the accessibility tree misses.

L2.5 - Dual YOLO Models on EC2 (~150–300ms, near-free): For icons with no readable text - the toolbar share button, the Safari back arrow, the FaceTime video icon - OmniParser and Screen2AX run in parallel on EC2 to classify them visually. Screenshots stay in memory only, never written to disk.

L3 - Amazon Nova 2 Lite Vision on Bedrock (~800ms, paid, last resort): We send the screenshot to Nova. It returns the exact bounding box. Then we cache that result in Aurora with a pgvector embedding - so the next user with a semantically similar step gets it free via cosine similarity at threshold 0.92.

Every interaction feeds Aurora PostgreSQL with pgvector - plan cache, step label cache, and user-taught corrections all live here. The result: the majority of steps resolve in under 50ms at zero marginal cost. The system gets cheaper the more it's used.

Building Waylo for the #H0Hackathon taught us that the best architecture isn't the one with the most AI - it's the one that knows when not to use it.

The biggest surprise was how much Aurora PostgreSQL with pgvector simplified everything. We went in expecting to need a separate vector database alongside our relational data. Instead, pgvector gave us semantic search right inside Aurora - one connection, one query, no sync issues, no extra infrastructure to manage. What we thought would be a complex part of the system ended up being the most elegant.

Vercel handled the frontend side with the same kind of quiet reliability. Auto-deploys, serverless functions that cold-start fast enough to not matter - it just worked, and kept working, so we could focus on the actual product instead of infrastructure.

The deeper lesson: the right cloud primitives don't just reduce cost. They change what's possible to build. We shipped a self-improving AI pipeline with semantic caching, a four-layer fallback system, and a full web frontend - and the reason we could go that deep is because Aurora and Vercel genuinely got out of our way. We spent our time solving the hard product problems, not fighting infrastructure.