The latency budget
Humans notice conversation lag above ~700ms. Below 500ms it feels natural. Above 1.2s it feels broken. The full round trip — VAD → ASR → LLM → TTS → playback — has to fit inside 500ms to sound alive.
That means: streaming ASR (not batch), a fast reasoning model with speculative decoding, streaming TTS with a real-time voice, and a runtime that can barge-in the moment the buyer starts talking.
Turn-taking is the whole game
The single biggest tell of a bad voice agent is bad turn-taking — it interrupts, or it leaves a two-second gap after you speak. Good agents use end-of-turn detection that models the semantics of what you're saying, not just silence duration. Ask the vendor: "how do you detect end of turn?" If the answer is "500ms of silence," walk away.
Voice cloning ethics
Cloning a real rep's voice for outbound is legally murky and reputationally risky. Two rules keep you safe:
- Only clone with explicit written consent from the person whose voice you're using.
- Disclose the AI nature of the call when asked — always. In California, disclose proactively.
Most teams should use a stock voice with a distinctive tone, not a clone. It converts nearly as well and carries zero legal risk.
Evaluation criteria that matter
- P95 latency under 600ms on a real call, not a demo.
- Interruption handling — try to talk over it three times. It should back off cleanly.
- Objection handling depth. Give it your five hardest real objections. Score the answers with your top AE.
- Live handoff. When the buyer says "put me on with a real person," it should warm-transfer in under 8 seconds.
- Post-call artifacts. Structured summary, sentiment, next-step, CRM write-back — automatic.
Where it fits in your stack
Voice is one channel. The best AI sales systems (like CoLive) route the same conversation across chat, email, and voice with shared context — so when the buyer answers a call after ghosting three emails, the agent picks up exactly where the thread left off.