Voice Agent Latency: Getting Under One Second
On a phone call, a pause longer than about a second feels broken. Callers talk over the agent or hang up. Latency is the difference between a voice agent people tolerate and one they like.
- Published
The latency budget
| Stage | Target |
|---|---|
| Detecting the caller finished speaking | 200–500 ms |
| Final transcript | 100–300 ms |
| LLM first token | 200–500 ms |
| First audio from TTS | 100–250 ms |
| Network and telephony | 100–200 ms |
Add those up and you see why every stage must stream: nothing can wait for the previous stage to finish completely.
Techniques that work
- Streaming everything: transcription, generation and speech run concurrently.
- Smarter turn detection: combine silence with semantic cues so the agent does not wait needlessly or interrupt.
- Short first sentences: the agent starts speaking a brief acknowledgement while it finishes thinking.
- Fast models for conversation, with slower tools called asynchronously.
- Prompt caching to cut time to first token.
- Co-locate services in the same region as the telephony provider.
- Filler for slow tools: "Let me check that for you" while a lookup runs.
Interruptions
Callers interrupt. The agent must stop speaking immediately, discard the rest of its planned reply, and listen. Handling barge-in well matters as much as raw speed for perceived quality.
Frequently asked questions
What latency should we target?
Under one second from the caller finishing to the agent starting, consistently — not just on average.
Does a bigger model make it slower?
Usually. Use the fastest model that passes your evals for the conversational turn.
Does Indian telephony add delay?
Routing and region matter. We test latency on real local numbers before launch.
