Voice AI Latency: What Fits in a One-Second Budget

An AI phone agent has about one second between the moment a caller stops talking and the moment it has to start talking back. Roughly half of that is gone before any AI runs, spent waiting through a silence window to confirm the caller actually finished. What is left, often 400 to 600 milliseconds, has to cover speech recognition, the model, and speech synthesis. The design question for a voice agent is therefore not which vendor is fastest. It is which of your bot's jobs fit in what remains.

Where the one second actually goes

Human conversation runs tighter than most people assume. Stivers and colleagues measured turn-taking across ten languages on five continents and found the same shape in every one: the most common gap between one speaker finishing and the next starting falls between 0 and 200 milliseconds, with an overall mean of +208 ms. That is the rhythm your caller is unconsciously expecting. You will not hit it, and you do not need to. But the clock starts the instant they stop.

Here is where the time goes in a typical stack, as of July 2026. These are observed ranges across commodity voice components, not any single vendor's benchmark.

Stage	Typical range	Who controls it
Endpointing silence window	400 to 800 ms	You, by configuration
Speech to text	100 to 300 ms	Vendor
Model, first token out	300 to 800 ms	You, by model and prompt size
Text to speech, first audio chunk	90 to 200 ms	Vendor
Network and carrier transport	50 to 200 ms	Mostly fixed

The important row is the first one, and almost nobody quotes it. Vendor dashboards and comparison posts start the clock at "end of speech detected," which is row two. The caller starts the clock at row one, when they stop talking. LiveKit's own engineering writeup on turn detection puts it bluntly: an 800 ms silence timeout adds nearly a full second to every response before the pipeline even starts. Two agents with identical published latency can feel half a second apart on a real call, and the difference is entirely a config value someone picked once and forgot.

Speech-to-speech models compress rows two through four. OpenAI's Realtime API takes audio in and produces audio out through a single model rather than chaining recognition, a text model, and synthesis, which removes two handoffs and the buffering around them. It does not touch row one. Endpointing is still a policy decision, and it is still yours.

The silence window is your biggest lever and your riskiest one

Shorten the window to 300 ms and the agent starts talking over people. Lengthen it to 900 ms and every single turn drags, which callers read as the agent being slow or confused.

The damage is not evenly distributed, which is what makes a single global setting a bad idea. Callers pause longest mid-utterance when they are reading something aloud: a policy number off a card, a street address, an order number, a date of birth. Those are exactly the turns where an interruption costs the most, because the caller has to start the whole string over and now distrusts the system. A window tuned against "yeah, that works for me" will reliably chop "my account is four four eight, uh, two zero one."

This is why turn detection has moved past pure voice activity detection. VAD tells you there is silence. It cannot tell you the caller is halfway through a number. LiveKit ships a small transformer trained specifically to predict end of utterance from the last few turns of transcript, which is a different question from "is there sound right now."

Two settings worth copying from LiveKit's telephony guidance: raise the VAD threshold to around 0.7, because phone audio is noisy and a sensitive threshold turns line noise into speech, and keep prefix padding near 300 ms so the caller's first syllable is not clipped off the transcript. Then tune the silence window per intent rather than per agent. A yes or no confirmation can close at 300 ms. A "read me your account number" turn should hold considerably longer.

The latency budget is a feature budget

This is the part the vendor comparisons skip entirely. Every capability you would add to a text chatbot without a second thought is a serial addition to a budget that is already nearly spent.

What the agent does	Adds, serially	On a live call	Do this instead
Answers from the system prompt	0 ms	Fine	Nothing
Vector search over a small local index	50 to 150 ms	Fine	Keep the index in-process
Reads one record by a known key	200 to 600 ms	Borderline	Fetch at call start
Fuzzy search across a CRM or ticket system	0.5 to 2 s	No	Acknowledge out loud, then fetch
Checks live calendar availability across several reps	0.5 to 2 s	No	Precompute open slots on a schedule
Runs a second model pass to review the answer	300 to 800 ms	No	Constrain with tool definitions instead
Verifies identity or payment with a third party	1 to 4 s	No	Move it off the call

In a chat window, a two-second lookup is a typing indicator and nobody notices. On a call, it is silence, and silence is a signal to a human being. So the working rule for a phone agent is that it should answer most turns without leaving the process, and anything it must fetch it should either fetch before it needs it or announce before it does.

That flips the usual build order. When we build a text bot that books appointments, the tools get wired first and the timing gets tuned later, because there is slack everywhere. On voice, you set the budget first and then decide which tools you can afford to buy with it. A phone agent with four tool calls in a turn is not an ambitious phone agent, it is a broken one.

Latency on a call does not degrade gracefully

A slow text response is just a slow text response. The user waits, the answer arrives, the conversation continues. Voice has no such property.

Picture a 1.5-second gap. The caller says "hello?" or repeats the question. That speech triggers barge-in, the agent cancels the reply it was about to speak, and the turn restarts with a stray "hello" now in the transcript. You did not pay 1.5 seconds of delay. You paid for the entire turn, twice, plus a caller who now thinks the line dropped. Latency on a phone call compounds instead of accumulating.

The mirror-image failure is false interruption. Backchannels like "mhm," "right," and "ok" are how humans signal they are still listening, and a barge-in rule that cancels on any detected speech will cancel on those. If your agent stops mid-sentence every time someone grunts agreement, that is the setting to find: a false-interruption timeout that resumes the reply when the interruption was too short or too few words to be a real turn.

One more thing the averages hide. Every row in that first table has a p95 well above its median, and the rows do not spike together. They spike independently and add. A stack that averages 900 ms will produce 2.5-second turns several times in a ten-minute call. Design against the p95, the same way you would when setting timeouts anywhere else in an automation.

Three ways to buy time that chat never needs

Speak first, fetch second. Start speaking a real acknowledgment and fire the tool call at the same instant. Saying "let me pull up that account" buys roughly 800 to 1,200 ms of cover, which is enough for most single-record lookups to land before the sentence ends. One rule: the filler has to be true and specific. "Let me check the schedule" is doing work. "Great question!" is a tell, and callers notice it faster than they notice a pause.

Split the turn. Read back what you heard while the lookup runs. "So that is a service call at the Newton address, one moment." The confirmation was something you wanted anyway for accuracy, and it doubles as the cover for the fetch. Two problems, one second.

Move the slow job off the call. Anything reliably over two seconds should not happen while a human waits on the line. Texting a payment link, sending three appointment options by SMS, scheduling a callback for a case that needs a person: all of these are better outcomes than 4 seconds of hold music inside a sentence. The same logic that decides when a workflow should hand off to a human applies to handing off to a different channel.

Two smaller levers are worth naming. Prompt caching cuts the time-to-first-token on a long system prompt, which is exactly the part of the budget you are fighting for. And model size matters more here than anywhere else: a smaller model whose first token lands 300 ms sooner usually beats a smarter one on a call, because the caller is grading the pause and not the phrasing. Reserve the bigger model for the few turns that genuinely need it, the same way you would build any model cascade.

Also, keep the answers short. Time to first token is what you optimize, but total speaking time is what the caller endures. At a normal conversational pace of about 150 words a minute, a 75-word answer takes 30 seconds to say out loud, and nobody listens to the end of it.

How to start

Measure your own agent the way the caller experiences it, not the way the dashboard reports it. Record one real call, open the audio in any waveform editor, and measure from the last visible syllable of the human's speech to the first sample of the agent's. Compare that number to the latency your platform reports. The difference is your endpointing window, and for most agents we have looked at, it is the single largest improvement available without changing a vendor.

Then write down every job your agent performs during a call and put a millisecond number next to each one. Anything above two seconds gets precomputed, gets covered by speech, or leaves the call entirely. That list, not a vendor comparison chart, is what determines whether your phone agent sounds like a receptionist or like a hold queue. If you want a second set of eyes on the architecture before you commit to a stack, that is the kind of thing we build in custom software platforms, and you can tell us what the call flow looks like.