Judge an AI Receptionist on the Calls It Refuses to Handle

Every demo of a voice agent is a demo of a call it handles. A caller asks for opening hours, the agent answers in a pleasant voice, the caller books an appointment, everyone nods. That call was never the risk. The risk is the caller who rings at eleven at night about a leak, or who asks whether one medication interacts with another, or who says the word "solicitor" halfway through a sentence. What the agent does with those calls decides whether the deployment lasts a quarter.

So the useful evaluation question is not how well it converses. It is what the agent refuses, how quickly it recognises that it should refuse, and where the call goes afterwards.

An agent that never refuses is not more capable, it is less honest

A language model is built to produce a plausible continuation. Given a question about your returns policy, a model with no returns policy in front of it will still generate something that sounds like a returns policy. That is not a defect awaiting the next model release. It is the mechanism working as designed, applied to a question it has no grounds to answer.

On a chat widget, a confidently wrong answer is text on a screen that the user can check against other pages. On a phone call, it is a spoken statement from something the caller experiences as your organisation, with no record they can re-read and no visible seams. Callers do not discount it. They act on it, then they ring back annoyed, and now you are handling two calls instead of one.

This is why refusal behaviour belongs on the scorecard next to containment. Containment, the share of calls resolved without a human, is the number every vendor leads with, and it is trivially inflated by an agent that answers everything. A high containment rate with no visible refusal behaviour is not a good result. It is an unmeasured liability.

What a competent refusal actually contains

"Sorry, I cannot help with that" is a refusal in the same sense that hanging up is a refusal. It satisfies the safety requirement and fails the caller. A refusal that works has four parts, and all four are engineering decisions rather than matters of prompt wording:

1. Detection early enough to matter. The agent recognises the category within the first exchange or two, not after four turns of trying to help.
2. A plain statement of the boundary with no diagnosis attached. The agent says this needs a person, not "it sounds like you may have a fracture".
3. A destination that exists at that moment. A named queue, a specific mobile, or a callback commitment with a time attached, selected on the basis of the hour and the category.
4. Context carried across. Whoever picks up already has the caller's number, the reason in the caller's own words, and the transcript, so the caller is not asked to start again.

Most production failures we see sit at steps three and four. The agent correctly identifies that it should not proceed and then dead-ends, because routing was configured for business hours and the call arrived at midnight, or because the escalation path pushes the caller to a voicemail nobody monitors. The classification was right and the outcome was still worse than a plain answering machine.

The categories to write down before you configure anything

Refusal boundaries are specific to an organisation and cannot be inherited from a template. The exercise is to decide, in advance and in writing, which categories the agent may never attempt. Our guide to what an AI receptionist can and cannot do covers the general envelope. This is the version you have to fill in yourself.

Call category	Why the agent must decline	Where it should go instead
Clinical, legal or financial advice	Regulated work, and a plausible answer is the dangerous outcome	Named professional, with the caller's stated reason attached
Emergency or safety risk	Any delay caused by triage is unacceptable	Immediate transfer, or the emergency number stated plainly
Complaints and threatened escalation	Tone judgement and authority to concede are human work	Complaints owner, flagged, never a general queue
Price negotiation or discount requests	The agent has no authority and will invent one	Sales, with the requested figure recorded
Identity-dependent account changes	A voice is not authentication	Verified channel, with a reference number issued
Anything the caller repeats twice	Repetition is the reliable signal of misunderstanding	A human, on the second repetition, without being asked

That last row is worth adding even if you disagree with the others. Repetition is the cheapest confusion detector available on a voice channel and almost nobody wires it up. If a caller says the same thing twice, the agent has already failed to understand it once, and a third attempt is unlikely to go better.

Refusal is also the cheaper path

There is a cost argument running alongside the trust argument. A voice agent that keeps trying to resolve a call it cannot resolve burns turns, and every turn is speech recognition, model inference and speech synthesis billed again. Long failing calls are the most expensive calls in the system, and they end in a transfer regardless, so you pay the agent cost and the human cost on the same interaction.

Fast refusal inverts that. The call costs two turns instead of twelve, and the human receives it with context rather than cold from a frustrated caller. The breakdown of what AI features cost to run explains why per-call pricing misleads here: the denominator that matters is the resolved outcome, and a call that fails after twelve turns has a cost and no outcome at all.

How to test the refusals rather than the conversation

Refusal behaviour does not show up in a happy-path test suite, because the happy path is defined as the calls the agent is supposed to handle. It needs its own suite, built deliberately:

1. Write ten to twenty calls per refusal category, in the words real callers use rather than the words in your policy document. Callers do not say "I wish to raise a formal complaint". They say "this is the third time I have rung".
2. Include the awkward hybrids. A booking request that turns into a symptom description partway through. A routine enquiry that ends with a threat of legal action. Mixed calls are where classification actually breaks.
3. Run every category at out-of-hours timestamps as well as in-hours, and confirm the destination is live in both cases.
4. Score the destination, not only the decision. Did a human answer, was the context present, and did the caller have to repeat themselves.
5. Re-run the whole suite after every prompt change, because a wording change made to improve one category routinely widens another.

The discipline here is the same one that applies to any AI feature going live, and the approach in our guide to testing an AI integration before it goes live transfers directly: fixed input sets, scored outputs, and regression runs on every change rather than one demo before launch.

What to do next

Take your last two hundred inbound calls and sort them into three piles: calls the agent should handle, calls it must refuse, and calls you are not sure about. The third pile is the specification. It is also, reliably, the largest pile, and working through it is what separates a deployment that holds from one that gets switched off after a single bad call.

Once those categories are written down, they are the right thing to bring to a conversation about AI voice assistants for inbound calls, because they turn a vague scoping discussion into a specific one about routing, hours and escalation paths.