AWS Builder Center

Zero-Trust AI: Moving from Cloud APIs to a 100% Local Pipeline

A hackathon-winning AI meeting assistant sent audio straight to cloud APIs — until an interview question exposed why that's a dealbreaker for regulated industries. Here's how I rebuilt Voldio AI as a 100% on-premise pipeline using pyannote.audio, IndicWhisper, and Llama 3.1, plus the diarization, Hinglish, and GPU latency problems I'm still fighting.

If you're building AI tools for regulated industries — law enforcement, legal, healthcare — this is the mistake I made, and how I fixed it.
Every developer knows the high of a hackathon.
Back in November 2025, at Sphinx '25 in MNIT Jaipur, our team — Team Alpha (me, Aayushi Bansal, Dheeraj Bilani, and Vansh Soni) — built Voldio AI, an AI meeting assistant that listens to meeting recordings, figures out who's speaking, writes summaries and notes, and syncs deadlines to Google Calendar and Notes.
Like most hackathon teams, we just wanted it working fast. So we connected Django and FastAPI to cloud APIs — Gemini and ElevenLabs — and it worked fine. We presented it, won the hackathon, and moved on.
Then came July 2026.

The Question That Stopped Me Cold

I was in an internship interview, walking the interviewer through Voldio AI's architecture, feeling pretty good about it. They listened, nodded, and then asked something like: if a police department or a law firm used this, wouldn't they be streaming highly confidential audio straight to a third-party's cloud servers? How would you solve that?
Silence.
They were right. In our rush to build features, we'd completely overlooked data privacy. If an AI tool handles sensitive conversations, sending raw audio across the public web isn't a minor oversight — it's a dealbreaker for exactly the users who'd benefit most from the tool.
I couldn't stop thinking about that question. So, as a personal challenge, I sat down to rebuild Voldio AI's engine from scratch — this time with a strict, 100% on-premise, air-gapped pipeline.
Here's what that rebuild looks like, what's still breaking, and what I'm learning along the way.

The New Architecture: Keeping Everything Local

To make sure no audio byte ever leaves the server, I replaced every cloud API with a self-hosted, open-source model:
Flowchart showing audio flowing through pyannote.audio for speaker diarization, IndicWhisper for transcription, Llama 3.1 for summarization, then FastAPI/Django syncing results to Google Calendar and Notes


1. Speaker diarization with pyannote.audio

Before an AI can understand what was said, it needs to know who said it. Instead of relying on a cloud service, I brought in pyannote.audio locally. It analyzes acoustic features — pitch, frequency, voice prints — to slice the audio timeline into distinct speaker turns.

2. The Hinglish problem: why plain Whisper wasn't enough

Most tutorials tell you to drop OpenAI's Whisper Large-v3 into your project and call it done. But real Indian meetings rarely happen in pure English — people constantly switch between Hindi and English mid-sentence ("Hinglish").
Standard Whisper trips over this. Its aggressive text normalization strips diacritics and mangles context. To fix it, I switched to IndicWhisper by AI4Bharat. Since it's trained on thousands of hours of native Indian-language audio, it handles vernacular nuance far better.

3. Reasoning on-device with Llama 3.1

The last piece was replacing the Gemini API for natural-language extraction. I deployed Meta's Llama 3.1 on a local server. It takes the diarized transcript and structures the raw conversation into an executive summary, clear takeaways, and concrete action items.

Let's Be Honest: What's Still Broken

A lot of tech blogs pretend everything worked on the first try. It didn't. Running heavy ML models locally is genuinely hard, and this proof-of-concept is far from polished. Here's what I'm still fighting:
  • Speaker overlap and diarization bleed. pyannote.audio is solid when people take turns, but in messy cross-talk — two people speaking at once — it frequently misattributes who said what.
  • The word-by-word accuracy gap. The transcription pipeline isn't fully granular yet. In fast-paced audio or heavy background noise, words get dropped, and that error compounds into the summary.
  • Llama 3.1's Hindi ceiling. It's excellent at English reasoning, but comprehension drops noticeably on raw Hindi text — it just wasn't trained on much Hindi reasoning data.
  • GPU latency. Chaining three large models sequentially, all locally, takes real compute. Getting inference down from minutes to seconds without a server farm is an ongoing fight.

What This Taught Me

This project started as a hackathon idea with my team. Turning it into a privacy-first rebuild taught me more about system design and ML engineering than any tutorial could.
Building features is only half the job. Thinking about data privacy, model efficiency, and messy real-world edge cases is what separates a weekend prototype from actual software engineering.
I'm still tinkering with model quantization (GGUF/vLLM) and looking at native bilingual LLMs to close the remaining gaps.
Have you worked with local speech-to-text or multilingual diarization pipelines? I'd love to hear how you've tackled latency and speaker overlap — drop a comment below.
#buildatpce
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article