The 800ms Rule: Budgeting Latency for a Real-Time Voice Agent on AWS
A practical latency budget for production voice agents on AWS. Where the milliseconds go across endpointing, ASR, LLM and TTS, what a speech-to-speech model changes, and five design rules that protect the budget.
Most articles about voice agents stop at "it works". In production, working is not the bar. The bar is whether the person on the other end of the phone keeps talking to you or hangs up.
Humans start to feel a gap at roughly 500ms of silence and start to talk over you at around 1 second. That is the whole design constraint. Everything below is how to budget against it.
The numbers below are the targets I design against, not measurements from one deployment. Treat them as a starting budget and replace each row with your own instrumentation.
The budget
The clock starts when the user stops speaking and ends when the first audio byte of the reply reaches their ear. Here is how the budget splits.
- Endpointing (VAD), 150-300ms. Deciding the user actually finished, not just paused.
- ASR final transcript, 100-200ms. Streaming ASR flushing the last partial into a final.
- LLM time to first token, 300-600ms. Prompt processing plus the first token.
- TTS time to first byte, 100-250ms. Synthesis starting to stream.
- Network and telephony, 50-150ms. Region hop, SIP leg, jitter buffer.
- Total, roughly 800ms to 1.5 seconds. What the caller experiences as "the pause".
Two things jump out of that list. First, endpointing is not free, and it is the stage most teams never touch. A 700ms silence threshold feels safe in testing and feels awful on a real call. The workable pattern is an aggressive threshold plus a cancel path, so an early trigger can be thrown away when the user keeps talking.
Second, the LLM is only one line item. Teams spend weeks shaving tokens off the prompt and then lose all of it to a synchronous tool call.
Speech-to-speech versus the stitched pipeline
There are two shapes of architecture available on AWS today. Across the six dimensions that decide the choice:
- Turn latency. Stitched: the sum of four stages, typically 800ms or more. Speech-to-speech: a single model, noticeably lower.
- Barge-in. Stitched: you implement it. Speech-to-speech: native.
- Prosody and emotion. Stitched: lost at the transcript boundary. Speech-to-speech: preserved end to end.
- Language coverage. Stitched: whatever your ASR and TTS vendors support. Speech-to-speech: limited to the languages the model ships with.
- Debuggability. Stitched: high, you see every transcript. Speech-to-speech: lower, fewer seams to inspect.
- Vendor surface. Stitched: often third party, and audio leaves AWS. Speech-to-speech: stays on Bedrock.
The stitched pipeline throws away everything that is not words. Tone, hesitation, whether the person is annoyed. The model gets a clean sentence and answers a question nobody quite asked. A speech-to-speech model keeps that signal, and it removes two serialisation boundaries from the budget.
I would use it for every deployment if I could. I cannot, because Amazon Nova Sonic does not support Vietnamese yet, and for a Vietnamese collections or support call that is a hard stop. So the stitched path it is, and the engineering effort goes into the budget above.
Five rules that protect the budget
- Stream everything, including the part you think is too small to stream. Partial transcripts into the prompt builder, tokens into the TTS as soon as the first sentence boundary appears. If any stage waits for the previous stage to finish, you have already lost.
- Speak before you think. A short acknowledgement while a tool call runs buys back a full second of real budget, and it is what a human agent does anyway. Keep the set small and pick by intent, otherwise it reads as a robot with three moods.
- Pick the region for the caller, not for the team. Every 1000km of round trip is real milliseconds, and a model that only lives in a far region can quietly cost 100ms per turn. For Southeast Asian traffic the nearest Nova Sonic region today is Tokyo, which is measurable on a call.
- Cap the tool call, not just the model. Give the agent a hard budget per turn. If a lookup does not return inside it, the agent says so and continues. A correct answer that arrives after the caller has started repeating themselves is a failed turn.
- Trim the context, aggressively. Long conversation history inflates time to first token on every single turn. Summarise past turns instead of replaying them, and keep the system prompt short enough to read out loud.
What I am still stuck on
Two open problems, and I would like to hear how other people handle them.
Vietnamese and English code-switching. Real conversations mix English product and brand names into Vietnamese sentences. Most ASR configured for one locale mangles the other, and the fallback of running two recognisers doubles the ASR line item in the budget.
Barge-in on a stitched pipeline. Detecting that the user has interrupted is easy. Cleanly cancelling in-flight TTS, truncating the assistant turn to what was actually heard, and keeping the conversation state honest about it is the part that takes longest to get right.
If you are running voice agents in a language Bedrock does not cover natively yet, I put up a wish for Vietnamese support in Nova Sonic on the AWS Wishlist. A vote helps make the case.
The takeaway
Write the budget down before you write the agent. Put every stage in a list, attach a number to each one, and instrument all of them from day one. When a call feels slow you want to know which line went red, not whether "the model is slow".
What does your turn budget look like, and which stage surprised you the most?
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article