Why a slow voice agent sounds broken

How much delay will a caller tolerate?

Less than you would guess. Gaps between turns in ordinary human conversation are typically measured in a couple of hundred milliseconds, and speakers often begin before the other has finished. Around half a second still reads as normal. Somewhere near a second the caller starts to suspect a problem, and beyond about a second and a half most people either say hello or start their sentence again.

The reason is that a telephone gives the listener no other signal. In person you can see someone thinking. On a call, silence means one of three things: the line has dropped, they did not hear you, or they have walked away. All three prompt the same behaviour, which is to speak again.

This is why consistency matters as much as the average. An agent that answers in four hundred milliseconds nine times and three seconds on the tenth will be remembered for the tenth, because that is the turn where the caller talked over it and the conversation fell apart.

Perceived delay also depends on what the caller just said. A pause after a complicated question is tolerated, because the caller expects work to be involved. The same pause after yes reads as a fault, since there was obviously nothing to think about. If you can only make some turns fast, make the short ones fast: acknowledgements, confirmations and yes or no follow-ups are where silence is least forgivable.

Where does the delay actually come from?

Six places, and the first one is usually the largest and the least discussed. Before anything else can happen, the system has to decide the caller has finished speaking. That is endpointing, and the naive implementation waits for a fixed period of silence. If that timer is set to a second, you have spent a second before a single component has started work.

After endpointing come recognition finalisation, any tool or API call the agent needs, the language model producing its first token, and speech synthesis producing its first audio. Then network and telephony buffering on top of all of it. Each is individually small when done well, and they add up rather than overlapping unless the system is built to stream.

Streaming is the difference between a usable agent and a slow one. Recognition should emit partial transcripts, the model should be sent them before the caller stops, generation should be streamed sentence by sentence into synthesis, and the first sentence of audio should start playing while the rest is still being generated. A pipeline that waits for each stage to complete cannot get under a second, whatever hardware it runs on.

Why does a slow agent get interrupted so often?

Because delay and interruption are the same problem seen from two ends. The caller waits, decides the agent has not heard, and starts speaking. The agent, which had been generating a reply, now begins playing it. Both are talking. What happens next depends entirely on whether barge-in is handled properly.

Barge-in means detecting caller speech while the agent is speaking and stopping the agent's audio immediately, within a fraction of a second. That requires echo cancellation, because otherwise the agent hears its own voice and interrupts itself, and it requires the ability to cancel synthesis mid-stream rather than waiting for the current sentence to finish playing.

A system without proper barge-in produces the single most recognisable failure in this field: caller and agent talking over each other, then both stopping, then both starting again. If you hear that on a demo, no amount of prompt tuning will fix it, because it is an architectural gap.

Which tricks hide latency, and which backfire?

The one that works is speaking before you know the answer. When the agent has to check a calendar, a short acknowledgement said immediately buys the entire round trip: let me check that for you covers about a second and a half of real work, and it is what a human receptionist does for exactly the same reason. Say it before the lookup starts, not after it returns.

The one that backfires is filler everywhere. An agent that says right, let me see before every reply becomes obviously scripted within three turns and adds length to calls you are paying for by the minute. Reserve acknowledgements for turns that genuinely involve a lookup.

Also treat injected hesitation noises with suspicion. Adding um and er to sound human tends to be noticed in the wrong way on a phone line, where audio quality already strips nuance, and it makes the agent seem uncertain about facts it is actually confident about. Naturalness comes from timing and short sentences, not from imitating disfluency.

What makes latency spike in production?

Chained tool calls, first. An agent that looks up the caller, then fetches availability, then writes a booking, performs three sequential round trips inside one turn, and any one of them being slow stalls the whole reply. Parallelise what you can and accept an acknowledgement phrase for what you cannot.

Then context growth. Time to first token rises with the size of what you send, so a conversation that started with a short prompt and now carries fifteen turns of history is slower at turn sixteen than it was at turn two. Long calls get slower as they go, which is exactly the wrong direction, and summarising older turns fixes it.

Finally, provider variance and geography. Hosted model and synthesis endpoints have variable queueing, retries silently double the wait, and a media path that crosses a continent adds real milliseconds on every leg. Monitor the ninety-fifth percentile rather than the average, because the average hides precisely the turns that ruin calls.

How do you measure your own response time?

Record five calls and open them in any audio editor. Measure the gap from the end of your speech to the start of the agent's, on every turn, and write down the numbers. This takes fifteen minutes and it is more informative than any vendor dashboard, because it measures what the caller experienced rather than what the server logged.

Do it from a mobile phone on a cellular connection, not from a laptop on your office broadband. Mobile audio paths add delay and degrade the audio that recognition receives, and a demo that sounded fine on a headset can be visibly worse on a car call. Test the conditions your callers are actually in.

Then ask your vendor for per-stage timings and the ninety-fifth percentile for each. If they can only give you an average, or cannot break it down by stage, they cannot diagnose a slow agent either, and that answer tells you something about what happens when you report a problem.

One further test is worth running before you sign anything, and it takes a minute. Interrupt the agent mid-sentence and time how long its audio keeps playing. Anything much beyond a quarter of a second means barge-in is being handled after the fact rather than in the audio path, and every busy caller will experience that as the system refusing to listen. This single check separates platforms more reliably than any feature list.