Accents, noise and the calls it will mishear

What does a voice agent mishear most?

Proper nouns and strings, in that order. Surnames, business names, street and place names, postcodes, email addresses, reference numbers and spelled-out letters. These fail more than ordinary conversation because speech recognition resolves uncertain sounds using the surrounding words, and a name has no surrounding words to help. When the recogniser has never encountered a name, it substitutes something it has.

Spelled letters are the worst case, because several English letter names are near identical over a narrow-band phone line. The confusable groups are predictable: b, p, d, t, e, v and g cluster together, as do m and n, and s and f. A caller carefully spelling their surname is producing exactly the audio the channel is worst at carrying.

Digits fail differently, and more dangerously, because the error is plausible. Fifteen and fifty, thirteen and thirty, and any pair separated only by a final consonant. A misheard digit in a phone number produces a callback to a stranger, and nothing in the transcript looks wrong.

A fourth category is misheard because it was never said cleanly. Callers on the move mumble, trail off, and correct themselves mid-sentence: actually no, make it Thursday. Self-correction defeats any system that takes the first confident interpretation and acts on it, and it is common enough in real speech that handling it is not an edge case. An agent that cannot process a change of mind will book the wrong day repeatedly.

Do accents actually cause failures?

Yes, but unevenly and less than people expect from mainstream accents, and considerably more for regional accents, second-language speakers and older voices. The mechanism is that models perform best on the speech that dominated their training data, so an accent that is common in the training set is handled well and an accent that is rare is handled worse. It is a data property rather than a mystery.

The practical effect is not usually total failure. It is a higher error rate on exactly the items that were already fragile, which means an accent problem shows up as more misheard names rather than as an incomprehensible call. That is why teams often report that the agent works fine and separately report that half the callback numbers are wrong.

You can test this in an afternoon, and you should test it with the voices your business actually receives rather than the voices in your office. Ask six people with different accents to leave the same booking request, and count how many surnames and numbers come through intact.

What does the phone line itself do to accuracy?

It removes information before recognition begins. Traditional telephony carries a narrow frequency band, roughly the range needed for intelligible speech and no more, and the consonant distinctions that separate confusable letters live partly above it. A recogniser that performs well on a laptop microphone is working with materially less signal on a phone call, and no amount of prompt engineering recovers what the codec discarded.

Then the caller's environment adds noise the system must separate from speech. A car, a building site, a supermarket, a busy salon. Hands-free and speakerphone are the compounding factor, because they pick up room reverberation along with the voice and reverberation smears the very transitions that distinguish one consonant from another.

Noise also breaks turn-taking rather than only accuracy. Voice activity detection has to decide when the caller is speaking, and a television in the background, a passenger talking, or a burst of traffic can be read as caller speech. That produces the agent interrupting itself and cutting off replies, which callers experience as rudeness rather than as a technical fault.

How should the agent confirm what it heard?

Selectively, and by reading back rather than asking again. Confirm only the items where an error is costly and undetectable later: the callback number, the appointment time, the address, the spelling of a name if you will use it in writing. Do not confirm the reason for the call; if that is wrong, whoever rings back will find out cheaply.

Read back in the form a person would check. Numbers grouped rather than as a single stream, and letters using the phonetic alphabet when the caller has spelled something: p for papa is unambiguous where p alone is not. Accepting the phonetic alphabet from callers is equally useful, since anyone who has read a reference number over a bad line already knows it.

Then keep a confirmation budget. Every readback costs a turn, and turns cost seconds and money. Two confirmations in a two minute call is unremarkable; five makes the call feel like an interrogation and pushes callers to hang up. If you find yourself needing five, the answer is to stop collecting some of those fields by voice.

What should never be captured by voice at all?

Email addresses, first. They are strings with no context, they usually contain a spelled component, and getting one character wrong means the confirmation never arrives and nobody knows why. The better pattern is to text the caller a link, or to confirm an address you already hold rather than take a new one.

Long reference numbers, account numbers and card details belong on the keypad rather than in speech. Keypress collection is exact, it is unaffected by accent or noise, and for card details it keeps the digits out of your transcript and recording entirely, which matters for what you are then obliged to protect. Any platform that cannot collect a keypress mid-conversation is limiting you here.

Anything the caller would have to spell twice is a signal to change channel. If the exchange has become the caller repeating letters, the agent should offer to send a text or to have a person call back. Persisting is where callers decide the whole system is useless, and it is a two-sentence fix in the conversation design.

How do you test recognition with your own callers?

Build a list of the fifty words your business cannot afford to get wrong: your trading name, your locations, your service names, the streets you work on, the surnames of your ten biggest customers. Most platforms let you supply a vocabulary or bias list, and supplying one is the single highest-return tuning step available. Then check each word by saying it into a real call.

Next, sample real calls rather than testing scripts. Pull twenty transcripts from last week alongside the audio, and mark every point where the transcript and the audio disagree. Count the errors by category: name, number, address, other. The distribution tells you what to fix, and it is almost never what the team assumed.

Finally, run the environments deliberately. One call from a car on hands-free, one from a busy room, one on a weak mobile signal, one from a landline. If the agent only works on the last of those, you have a system that will perform worst on exactly the callers who ring while they are out and about, which for most trades is all of them.

Keep the results as a short list of failing examples with the audio attached, and replay it after every configuration change. Recognition tuning is not a one-off task: vendors update models, vocabulary lists go stale, and your own name list grows as you take on new areas. A fixed set of twenty utterances you re-run each quarter is the difference between catching a regression and hearing about it from a customer.