The most advanced AI product runs on EC2
A real-time voice AI company trains on SageMaker and Bedrock but serves live inference on EC2 because voice has no latency budget.
I host a cloud podcast, which means I spend a lot of airtime joking that nobody wants to hear about another batch of EC2 announcements. The interesting stuff is supposed to be up the stack: managed services, Bedrock, agents, the layers where AWS does the heavy lifting so you don't have to think about instances.
Then I sat down with Mike Pappas and Carter Huffman, the founders of Modulate, and most of their production stack turned out to be Linux boxes with GPUs on EC2, auto-scaling groups, load balancing, and some of the hottest code paths written in C. This is a company doing real-time voice AI: content moderation behind games like Call of Duty, deepfake and fraud detection on live calls. It is genuinely advanced work, and it runs close to the metal on purpose.
That gap between where I expected the value to be and where they actually built is the whole article.
Voice has no latency budget to spend
The reason they're down on EC2 is latency, and voice is the most unforgiving place to fight that battle.
In a real conversation, the gap between one person finishing and the next person starting is roughly zero. You don't notice it because it's never there. The moment you add a delay, the whole interaction degrades, and not in a subtle way. A pause after you stop talking makes people uncomfortable enough to fill it, so they start talking again, now over the top of the bot, and the experience falls apart. Carter's number for their voice-skin models was 20 milliseconds. For context, 100 milliseconds on a voice call is instantly noticeable to a human.
This is why the obvious architecture doesn't work. Transcribe the audio, send the text to an LLM, run text-to-speech on the reply: it sounds clean on a whiteboard and it fails in practice. To hit a voice latency budget you're forced onto the flash and lite versions of every model, which makes your bot fast and dumb. People would rather talk to a phone tree than a fast idiot. And transcription throws away everything voice actually carries beyond the words, the emotion and tone and cadence, which is exactly the signal Modulate needs to tell two kids messing around apart from real harassment.
Average latency is the easy half
Here's the part that reframed the infrastructure question for me. They don't optimise for average latency. They optimise for jitter and predictability.
On a real-time audio stream, a model that's fast 99.9% of the time and occasionally stalls is worse than a model that's slightly slower but steady, because the stalls land in the middle of someone's sentence. That's why the latency-critical models go down to C while everything else stays in Python. Memory allocation in a managed runtime can introduce a variable delay that almost never matters, until you're streaming audio in real time and the 0.1% tail cases start piling up across thousands of concurrent conversations.
So the abstractions get stripped out, even valuable ones, where the latency budget can't absorb them. On AWS that looks like:
- Plain EC2 with GPUs, in auto-scaling groups behind load balancers, rather than a managed inference layer that hides the instance.
- Pre-warming as a first-class concern: loading models into VRAM and warming queues and caches before the traffic arrives, so a cold start never lands on a live call.
- Multi-region and multi-AZ deployment so they can route a session to the physically closest box, because the speed of light is part of your latency budget once you're chasing tens of milliseconds.
None of this is exotic. It's the EC2 toolkit a lot of us learned years ago and assumed we'd graduate out of.
Why not just one big model
Worth a quick detour, because it's the same instinct in a different costume. The audience reflex is to ask why you wouldn't just throw the audio at one large foundation model. Carter's answer was that a big general model tends to hyper-focus on one aspect of the input and run with it. You get an answer to your question, not a holistic read of the whole conversation.
Modulate instead runs up to 100 models in parallel on a single conversation in real time, more than a dozen transcription models alone, some general, some tuned for medical or gaming terminology, and combines the outputs into one dense representation that they layer analysis on top of. Starting from that rich base is far more deterministic than dropping a file into a model and hoping. It's also, not coincidentally, a design you can only run if you control the hardware it runs on.
The actual lesson: access, not altitude
I went in with a tidy narrative. Cutting-edge AI company, surely they live high on the managed stack, and the EC2 stuff is plumbing they'd rather not touch. Carter corrected it cleanly. Moving up the layers of abstraction was never the point. The point is having access to all of them, plus the judgment to stay high when you can and drop low when you have to.
Their own stack proves it in both directions. Training lives up high: SageMaker, and a lot of back-and-forth with Bedrock for data throughput and agility, because when you're training a model a little latency is completely fine. Inference on a live call drops all the way to C on bare instances, because there the latency budget is zero. Same company, same AWS account, opposite ends of the abstraction ladder, and the skill is knowing which workload sits where.
That's the bit I keep coming back to. We talk about "moving up the stack" like it's a direction of travel, a place you're supposed to end up. Modulate treats the stack as a menu. The managed services are there when the workload tolerates them, and the bare metal is there when it doesn't, and the engineering judgment is picking correctly per workload instead of per fashion.
If your instinct, like mine, is that serious AI work has to mean managed services and agents, it's worth remembering that one of the more impressive real-time AI products out there is running on the boring layer, deliberately, because that's the layer the problem demanded.
This came out of a conversation with Mike Pappas and Carter Huffman of Modulate on the Talking Cloud podcast. Full episode: Inside Modulate's Voice AI Stack on AWS (video ).
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article