Realtime Voice AI: Architect the Call, Not Just the Model

Realtime voice models collapse a stack that used to require four services: speech understanding, reasoning, tool calls, and spoken response inside one continuous interaction. Fewer moving parts is a real gain. It does not remove the product decisions that determine whether a call feels competent or feels like a phone tree with a better voice.

Voice is unforgiving in a way text is not

A chat user will wait several seconds without concern. A caller experiences the same silence as the agent having no idea what to do. There is no typing indicator on a phone line, no way to scroll back, no skimming. Everything the system does is judged in real time, in the order it happens, by someone holding a phone to their ear.

That constraint drives the architecture. Every stage needs a budget, and every stage needs a plan for what the caller hears while it runs.

Give every stage a latency budget

Measure speech detection, model response, tool execution, audio generation, and network delivery as separate numbers. Stream wherever the pipeline allows it. Keep tool responses small — a voice agent waiting on a payload built for a dashboard is an avoidable pause. And move anything that does not affect the caller’s next sentence out of the call entirely: log writes, CRM syncs, and follow-up emails can happen after the line goes quiet.

• Stop talking immediately when the caller interrupts — barge-in latency is a product metric
• Read back names, dates, amounts, and anything else expensive to get wrong
• Ground factual answers in approved business knowledge, never in model recall
• Keep turns short enough to sound like speech rather than a document being read aloud
• Transfer with the transcript, inferred intent, and completed actions attached

Design for noise, accents, and half-finished sentences

Real calls come with construction in the background, a spotty connection on I-95, a toddler, and people who start a sentence three times. Build clarification that sounds human rather than accusatory — “I want to make sure I got that, was it four-one-five?” instead of asking the caller to repeat themselves a third time.

And set an explicit rule for when uncertainty stops being recoverable. Two failed clarifications on the same field should route to a person. Not because the model cannot try again, but because the caller has already decided this is not working.

The handoff is part of the architecture

A transfer that dumps the caller into a queue with no context is worse than never picking up the AI in the first place — they have now spent four minutes explaining a problem to nobody. The receiving agent should get a screen with the transcript, what the system already verified, what it changed, and what it could not resolve, before they say hello.

Improve from outcomes, not from listening to demos

Review whether the call resolved the job, whether a human had to correct something afterward, whether the transfer went to the right place, and whether the data it wrote downstream was valid. Conversation quality is an operational metric measured across a week of traffic. One impressive recording is a sales asset, not evidence.