Five Rules That Keep a Voice Agent Inside Its Latency Budget
A follow-up to The 800ms Rule. The per-stage targets I design against, what a speech-to-speech model changes, and five rules that keep a production voice pipeline inside its turn-taking budget.
I wrote about why a voice agent's reply has to land inside roughly a second in The 800ms Rule . This piece picks up where that one stopped: the per-stage targets I design against, what changes if you collapse the pipeline into a speech-to-speech model, and the five rules that keep a build inside the budget once real retrieval and guardrails are in the loop.
The numbers below are the targets I design against, not measurements from one deployment. Treat them as a starting budget and replace each row with your own instrumentation.
The per-stage targets
The clock starts when the user stops speaking and ends when the first audio byte of the reply reaches their ear. Here is how the budget splits.
Stage | Target | Why it's on the critical path |
|---|---|---|
Endpointing (VAD confirms the user stopped) | 150–200ms | Wait too long and every turn feels slow before it even starts. Cut it too short and the agent talks over people who paused to think. |
ASR finalization (streaming transcript settles) | 150–250ms | Streaming ASR beats batch here: you want the partial hypothesis feeding the next stage before the "final" flag ever fires. |
LLM time-to-first-token | 200–450ms | This is usually where a budget quietly blows past 1s: one unbudgeted RAG lookup or guardrail check between transcript and first token adds 300–800ms nobody planned for. |
TTS time-to-first-audio-byte | 150–300ms | For a streaming neural voice. Waiting for the full sentence before synthesizing anything is the single most common budget killer in early builds. |
Network and jitter buffer | 50–100ms | More on a cellular leg. Small, but it is not free, and it is the one line most teams forget to budget at all. |
Add those five stages up and a good-case turn costs roughly 700ms end to end, comfortably inside the 1-second talk-over threshold. The bad case is the one that ships anyway: a RAG lookup on the critical path, a guardrail check that blocks on a second model call, or a TTS engine that waits for full-sentence punctuation before it starts synthesizing. Any one of those pushes the total to 1.2 to 1.5 seconds, and now the agent is the one interrupting itself.
What a speech-to-speech model changes
Everything above assumes a cascaded pipeline: separate ASR, LLM, and TTS stages you can instrument and budget one at a time. A speech-to-speech model collapses those three stages into a single audio-in, audio-out pass, and the latency math changes with it. You are no longer summing four line items; you are budgeting one model call, and vendors report saving somewhere around 200 to 500ms versus a well-tuned cascade.
The trade you make for that speed is controllability. In a cascaded pipeline, the text transcript is a natural injection point: you can run a guardrail check, pull a RAG lookup, or rewrite a response before it ever reaches TTS. Collapse the pipeline and that seam disappears, because there is no intermediate transcript to inspect or intervene on before the audio comes back. For a voice agent that has to stay on script, such as a support line or anything with a compliance requirement, that missing seam is not a minor detail. It is the reason a cascaded design is still the safer default whenever the guardrail requirement is non-negotiable, and the speech-to-speech model is better saved for cases where raw latency matters more than mid-turn control.
Five design rules that protect the budget
- Budget from the ear, not the model card. Vendor benchmarks measure the model in isolation; your users experience the whole chain, including your own retrieval and guardrail calls. Instrument time-to-first-audio-byte end to end in production, not just time-to-first-token in a notebook.
- Treat endpointing as a probability, not a switch. A fixed silence timeout forces a bad trade between cutting off people who pause to think and feeling slow for people who do not. Use a VAD that outputs a running end-of-turn probability and let context (a trailing question mark in the partial transcript, filler words, prosody) shift the threshold turn by turn.
- Stream every stage; never wait for a full turn. Stream partial ASR into the LLM prompt as it settles, stream LLM tokens into TTS as they are generated, and stream TTS audio to the client as it is synthesized. Buffering any one stage until it has a complete unit of output is where most of the 300 to 800ms overruns actually come from.
- Keep a filler ready for when you blow the budget. Even a well-tuned pipeline occasionally misses, usually on the first turn after a cold start or when a RAG lookup runs long. A short acknowledgment token or backchannel, spoken the moment the budget is at risk, buys another 500 to 800ms without the caller perceiving dead air.
- Put guardrails and retrieval inside the budget, not after it. A safety check or a RAG lookup that happens between the finalized transcript and the first LLM token is the single most common way a good architecture ends up with a bad latency number. Design them to run in parallel with ASR finalization or as part of the same call, not as a serial step bolted on afterward.
None of this is exotic. It is the same discipline as any other latency-sensitive system: measure the real critical path, budget every stage on it, and treat anything off that path, however useful, as something that has to earn its way back in. The difference with a voice agent is that the person on the other end feels every millisecond you get wrong, and they will not tell you about it. They will just stop talking.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article