Killing dead air: how we made a phone agent sound like it is listening
Our voice agents answer real customer calls, and the worst part of the experience was silence while the agent thought. Here is how we pre-generated filler audio, stored it in Amazon S3, and made the agent answer instantly with a voice that matches the rest of the call.
What our agent does and who it is for
I am an AI Engineer at Revve AI, a startup in Vietnam that builds voice agents for businesses. Our agents pick up the phone for banks, toll and transport services, and customer service teams. They answer questions, check information, open support tickets and transfer the caller to a human when needed. The people who use them are not developers. They are ordinary callers who dialed a hotline and want an answer in their own language, mostly Vietnamese and also English.
A voice agent is not judged like a chatbot. On a phone call, nobody reads the answer. They hear it, and they hear every pause. A two-second gap on a call feels like the line dropped. Callers start saying "Hello? Hello?", talk over the agent and lose trust in it, even when the answer that follows is correct.
How we built it
The agent runs as a pipeline. Speech-to-text turns the caller's words into text, a language model decides what to say, and text-to-speech speaks it back. A conversational flow (a graph of nodes and edges, designed in our web editor) keeps the agent on script for regulated tasks like banking.
- Voice runtime: Python 3.12 with LiveKit Agents, one worker per call.
- Web app and flow editor: Next.js, where teams design flows, choose voices and listen to test calls.
- Language model: a self-hosted model served with SGLang, plus OpenAI-compatible endpoints configured per team. We are adding Amazon Bedrock as one of those endpoints.
- Amazon S3: stores call recordings and the pre-generated filler audio described below. The web app writes with the AWS SDK for JavaScript (
@aws-sdk/client-s3), and the voice worker reads with boto3. - AWS Marketplace: the web app integrates the Marketplace metering and entitlement APIs, so customers can subscribe to Revve through their AWS account.
The delightful detail: the agent never leaves you in silence
We measured the problem first. On one banking bot, over 1,192 replies in 3 days, the model's first token took about 0.47 s with a cached prompt, plus about 0.3 s for every 1,000 uncached prompt tokens. A caller turn that moves the conversation to a new step pays that cost several times. The agent picks the next step (0.7 s), extracts details like an account type (1.0 s), then writes a reply on the new step (1.3–1.9 s). In practice, the caller heard a filler only after 1.5–2.8 s, then another 1.2–1.7 s of silence.
We did three things.
1. Several natural fillers per step. Each transition in a flow can hold up to three short phrases, such as "Dạ, em kiểm tra ngay ạ" ("Sure, let me check that for you"). The model picks one by number when it moves to the next step, so the agent does not repeat the same line on every turn.
2. Pre-generated filler audio in Amazon S3. Synthesizing a filler with live TTS adds its own delay, exactly when the caller is waiting. So the web app synthesizes each static phrase once, in the bot's own voice, and stores a WAV file in S3. The key is
filler-audio/{team}/{fingerprint}.wav, where the fingerprint is a SHA-256 of the phrase text plus the voice, speed and pronunciation dictionary. At call start the worker computes the same fingerprints and loads the clips into memory. When the agent needs a filler, it plays immediately with no TTS call.Content addressing keeps this safe. If a team changes the bot's voice or speed, the fingerprint changes, the old file is simply not found, and the agent falls back to live TTS until someone regenerates the audio. Phrases with a variable or a number are never pre-generated. If S3 is slow or a file is broken, the call still starts on time and uses live TTS. A filler should never be the reason a call fails.
3. The clip has to sound like the same person. Teams can listen to each clip in the editor and regenerate it if they don't like the take.
How we know it worked
The best evidence came from a problem. In the first version, a prod call showed that a stored clip played 3.5 dB louder than the live answer right after it, and it peaked close to full scale. The filler arrived instantly, but the join between the clip and the live voice was audible. It sounded like two different speakers.
So the worker now levels every clip once when it loads. It trims the silence at both ends to 30 ms, sets the voiced part to -17 dBFS RMS (peak at most -3 dBFS), and fades both ends over 10 ms. On our dev clips, levels went from a spread of -12.7 to -15.6 dBFS to a tight -16.3 to -17.0 dBFS, at a cost of 5–21 ms per clip, paid at call start and not during the conversation.
The lesson for us: being fast is not enough. The filler has to be fast and sound like the same voice, or callers notice the seam. We also apply the same idea to knowledge-base searches. While a long search runs, the agent can play soft hold music and keeps whatever the caller says during the search, so it can answer that too.
Proof it works
Revve AI: https://revve.ai
What I would tell another builder
- Measure the silence, not only the latency. Our dashboards showed model latency. The caller's experience was the gap between their last word and the first sound they hear.
- Pre-compute anything that does not change. Static phrases in a fixed voice are a perfect fit for object storage like Amazon S3.
- Name files by their content. A fingerprint key means you never play stale audio and never need a cache-invalidation job.
- Listen to real calls. The 3.5 dB jump did not show up in unit tests. We found it on a production call.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article