"Please hold": teaching a phone agent to wait politely while it searches
When our voice agent searched its knowledge base, callers heard seconds of silence and talked over it. Here is how we added hold music, kept every word the caller said during the search, and checked every change against real calls with per-call telemetry in Amazon S3.
Picture a phone call. You ask a hotline, "What is the fee if I pay my card late?" The voice on the other end says "One moment", and then there is nothing. One second. Three. Five. You start to wonder if the line dropped. You say "Hello?" right at the moment the answer was about to arrive, and now it starts over.
That was our voice agent, every time it had to look something up. This article is about how we taught it to wait politely.
What our agent does and who it is for
Our team at Revve AI, a startup in Vietnam, builds voice agents that answer hotline calls for banks, transport services and customer service teams, in Vietnamese and English. I work on it as an AI Engineer.
The people who talk to it are ordinary callers. They did not choose an AI, and they will not read documentation about it. They want a correct answer and the feeling that someone is still on the line. Many of their questions need a lookup: required documents, fees, opening hours. For these, the agent searches a knowledge base built from the business's own documents before it answers. That search is where the experience broke.
How we built it
- Voice runtime: Python 3.12 with LiveKit Agents. Speech-to-text, a language model and text-to-speech run as one pipeline, one worker per call. The same worker runs two engines: a single-prompt agent and a conversational-flow agent for scripted tasks.
- Knowledge base search: a tool the model calls. A fast mode runs one hybrid search (keyword plus vector). A deeper "agentic" mode lets a sub-agent search several times and rerank the results before it writes the answer.
- Web app: Next.js, where teams upload documents, design flows and turn options like hold music on or off per bot.
- Amazon S3: stores call recordings and a per-call telemetry bundle. At the end of each call, the worker collects its OpenTelemetry spans and logs, gzips them into one JSON file, uploads it to S3 with boto3, and reports the link to our API. This gives us a timeline of every call, so we can see what the caller heard and when.
The delightful detail: the agent knows how to put you on hold
A good human agent does three things when they need to look something up. They say "one moment, let me check". They don't leave you in dead silence. And if you add something while they look, they don't forget it.
Our agent did none of those well. A deep search can take several seconds; on one test call this week the knowledge base search alone took 7.9 s. During that time the caller heard nothing. A caller who said "Hello?" or repeated the question cancelled the reply that was being prepared. The agent started over, and the call got longer and more confusing.
We changed three things.
1. Hold music instead of silence. When a search starts, the agent speaks a short filler ("Dạ, anh chị chờ em một chút ạ"). Then it loops soft hold music until it is ready to speak. The music runs at 30% volume and fades out just before the answer, so it never competes with the voice.
2. The caller's words are kept, not lost. If the caller speaks while the search runs, the agent does not throw the search away. Their turn is held and written into the conversation. The answer from the search then takes it into account. If that answer cannot run, the agent replies to the held words separately. Both engines behave the same way.
3. Small things a person would do. If the caller spoke over the filler and never heard it, the agent says it again before playing music. If the model launches two searches at once, the music keeps playing until the last one returns, not the first.
How we know it worked
We judged it by what the caller actually hears, not by how long the model takes. Every call leaves a recording and a telemetry timeline in S3, so we can check each change against real calls.
The first version stopped the music the moment the search returned. That was not enough: after the search returned, the model still needed 2–3 seconds to write the answer. The music had stopped, so callers heard silence again. If they spoke into that gap, they cancelled the answer they were waiting for.
So the music now plays until the agent's first audio, not until the search ends. We found one more problem: when a caller spoke before the answer was heard, the next reply saw the search result placed before the caller's words and ignored it. Now an unheard result is moved after the caller's latest message, so the agent still uses it.
Each fix removed a moment where the caller would otherwise wonder whether anyone was still on the line.
Proof it works
Revve AI: https://revve.ai
filler word, hold music and held caller turns while a knowledge base search runs
What I would tell another builder
- On voice, silence is a bug. Track the gap between the caller's last word and the first sound they hear, not only model latency.
- "Done" is not "heard". A search finishing does not mean the caller has heard anything. End the hold when audio starts, not when the tool returns.
- Never drop what the user said. Interruptions are normal on the phone. Keep the caller's words and answer them.
- Store one telemetry file per call. One gzipped bundle per call in Amazon S3 is cheap, and it shows what the caller heard, second by second.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article