TL;DR
- Speed matters most. The bot should start replying within 300ms of the caller finishing.
- Don't talk over people. The bot needs to know when someone's done talking, and stop instantly if they interrupt.
- Phone systems have rules. Twilio (a common phone platform) expects audio in a specific format, which affects how you build the bot's voice output.
- Choose the right setup. An all-in-one voice AI works for simple calls like intake; a system built from separate parts (speech-to-text, AI, text-to-speech) works better when you need more control.
Phone support only feels helpful when it sounds instant, knows when to stop talking, and recovers the moment a caller interrupts.
If you've ever heard a voice bot pause too long, talk over you, or drag out a simple answer... that's the failure mode this guide is about.
We'll cover the real pipeline behind live phone agents: telephony transport, streaming speech-to-text, streaming text-to-speech, turn detection, and barge-in handling - plus where latency piles up and which design choices keep a call feeling natural instead of robotic.
What Makes a Call Feel Natural Or Frustrating
A caller notices timing before they notice model quality. If the assistant answers after a long pause, talks over a question, or keeps speaking after the caller has already started again, the call feels off even when the transcript is accurate.
The failure modes are usually plain. The bot waits too long to commit to a turn. It starts text-to-speech (TTS) only after the full response is ready. It misses a caller's interruption and keeps reading from the same script.
In a phone call, there is no visual buffer to hide that behavior.
That is why teams working on voice support end up watching partial transcripts, silence detection, and the time to first audio from TTS. Those are the points where a live call either keeps its rhythm or falls apart.
How A Low-Latency Speech Pipeline Works On Live Calls
A phone support flow usually starts with live telephony audio arriving at an app server.
Twilio Media Streams is a common example. It sends call audio over WebSockets, and in bidirectional mode, the app can send audio back into the call. Twilio requires returned audio to be audio/x-mulaw, 8000 Hz, base64-encoded.
From there, the common path looks like this:
- Telephony ingest and audio normalization
- Streaming STT or speech-to-speech
- Turn detection and interruption handling
- LLM reasoning and tool calls
- Streaming TTS back to the caller
Some systems keep each layer separate. Others collapse more of the path into a realtime speech-to-speech loop. OpenAI's Realtime docs describe low-latency voice interaction over WebRTC, WebSocket, and SIP, which is why those transport choices come up so often in phone-support architecture discussions.
A modular stack gives more control over transcription, voice selection, and retrieval. A unified realtime stack reduces glue code, but the tradeoff is less room to swap pieces independently.
How to Spend Your Latency Budget
Latency comes from several places at once:
- Transport adds delay
- STT needs time to emit partials
- The model needs time to think
- TTS needs time to start speaking
- Jitter can make all of that feel worse, even when the average looks fine
For phone support, the most useful number is the gap between the caller stopping and the assistant starting to speak. Deepgram's guidance puts low-latency voice AI under 300 ms end-to-end, from when the speaker stops talking to when the AI begins replying, which Deepgram frames as the complete round-trip delay between the last word and the AI's first syllable.
That is a better metric than overall response time because callers do not experience a call as a single API call. They experience a sequence of pauses. A system that is fast most of the time but stalls on a few turns still feels unreliable.
Time-to-first-audio matters as well. If the assistant can say a short acknowledgement quickly, the caller gets a cue that the line is active while the rest of the response is still being assembled.
Why Knowing When to Talk (And Stop) Matters
- Voice activity detection, or VAD, is what tells the system that speech has started or stopped.
- Turn detection uses that signal, plus transcript state and silence timing, to decide when the caller is done.
- Barge-in lets the assistant stop speaking when the caller starts speaking.
Those are small pieces of the stack, but they shape the whole call. If endpointing waits too long, the assistant sounds hesitant. If it triggers too early, it cuts people off. If the bot cannot interrupt itself mid-response, the caller ends up fighting for the floor.
The interruption handling docs for Vision Agents describe interruption support, turn detection, and a VAD-based commit strategy in the voice stack, including an interrupt() path that drops stale audio when barge-in happens. After an interruption, the agent stops active TTS, flushes the audio track, and ignores any already-generated audio that belonged to the previous response. That kind of plumbing matters more than most model comparisons when the call has to feel live.
The practical test is: Can the caller speak over the bot without a messy overlap or a repeated answer? If the answer is no, the rest of the stack is still too rigid.
Telephony Constraints That Shape Your Architecture
Phone support is constrained at the telephony edge before the model ever sees anything. That is easy to miss if most of the team has only worked on browser audio.
Twilio's media stream setup is a good example. It streams call audio into your app, and the return path has to match Twilio's format. Bidirectional streams are created with <Connect><Stream> and receive only the inbound track, the return format is fixed, and the stream ends when the call ends. That changes how retries, handoffs, and teardown logic need to work.
The main constraints usually are:
- Codec and sample rate on the return path
- WebSocket or SIP integration at the edge
- Inbound, outbound, or both
- Transfer and escalation handling
- How much buffering is tolerable before the call feels delayed
OpenAI's Realtime stack also matters here because it supports WebRTC, WebSocket, and SIP, which gives teams a few different ways to place the voice layer relative to the phone network.
Realtime vs. Modular: Which Setup to Use
The choice comes down to control versus simplicity. The table below breaks down when each approach fits.
| Realtime speech-to-speech | Modular STT -> LLM -> TTS | |
|---|---|---|
| Best for | Intake, triage, basic call routing | Support flows needing retrieval or company docs |
| Setup complexity | Lower - one unified path | Higher - more integration work |
| Control over transcription/voice | Limited | Full control over each stage |
| Vendor flexibility | Harder to swap pieces independently | Easy to swap STT, LLM, or TTS vendors |
| Tuning | Less room to tune individual stages | Easier to fine-tune each stage separately |
| Tradeoff | Simpler, but less control | More control, but more glue code |
OpenAI's Realtime docs describe voice-to-voice interaction and note that it can also handle realtime transcription, while the platform's SIP support makes it usable closer to the phone edge. Microsoft's call transcription docs point in the other direction, where real-time transcription is the priority, and the agent can use transcripts during the call rather than waiting for a finished recording.
What Production Support Needs Beyond The Model
The model is usually the easiest part to buy. The rest is what keeps the system working on real calls.
A production setup needs session tracking, retry logic, and a clean recovery path when the media stream drops or the transcription service stalls. It also needs metrics that reflect live calls, not just API throughput. p95 and p99 turn latency, barge-in success rate, and call drop rate are more useful than a single average response time.
Support calls also need retrieval. If the caller asks about a billing rule, a return policy, or a product exception, the agent needs an answer tied to company data. In practice, that means the phone flow needs a knowledge lookup layer, whether that is RAG over an internal doc set or a smaller scoped FAQ index.
Deployment matters too. Call handling, session state, and auth need to survive restarts. Twilio media streams, SIP handoff, and transcript events all need to land somewhere persistent enough to diagnose when a call fails in the middle of a turn.
Checklist For Shipping Natural Voice Support
Before launch, check the pieces that fail on a real phone line:
- Place a live inbound call through the exact Twilio number, webhook, and media-stream path you will use in production, then measure the gap from the caller's last word to the first audible reply.
- Test barge-in on a real handset with three cases: interruption mid-answer, overlap during a long explanation, and a caller saying stop or wait after the bot has already begun speaking.
- Confirm the return-audio format end to end, including resampling to Twilio's 8 kHz μ-law requirement and whether your TTS output needs a buffer before it can be sent back into the call.
- Watch partial transcripts during an actual support call and verify that endpointing fires on caller silence, not on a fixed timer that can cut off people who pause between clauses.
- Log p95 and p99 separately for STT partials, model first-token time, TTS first-audio time, and packet loss from the same region where your telephony ingress lands.
- Decide what the caller hears when transcription fails, the knowledge lookup times out, or the system has to hand the call to a human agent. A short spoken fallback or transfer is better than silence.
For phone support, the useful question is still the simplest one: Does the call feel normal when the caller interrupts, the network stutters, or the answer has to come from a knowledge base in the middle of the conversation?
The open-source Vision Agents framework from Stream covers this stack, with a Twilio phone support example and a quickstart for building voice agents end to end.
Frequently Asked Questions
What latency target should a phone support voice AI aim for?
A practical target is under 300 ms from the caller finishing a turn to the assistant starting to speak. That is not the only number that matters, but it is a useful benchmark when testing turn-taking.
Why does Twilio use 8 kHz μ-law for media streams?
Twilio Media Streams returns call audio in a telephony format that matches the phone network. The documented return path is 8 kHz μ-law, base64-encoded, so any TTS output has to be resampled before playback.
What is barge-in in a voice agent?
Barge-in is the ability for the caller to interrupt the assistant while it is speaking. A good implementation stops playback quickly and avoids replaying stale audio after the interruption.
When should a team use a realtime speech-to-speech stack instead of separate STT and TTS services?
Use a realtime stack when the goal is to keep the call moving with minimal orchestration, such as intake or routing. Use separate STT and TTS services when you need tighter control over transcription, voice choice, or retrieval.
Why are partial transcripts useful in live phone support?
Partial transcripts let the system react before the caller has finished the whole sentence. That helps with turn detection, live routing, and faster acknowledgments during the call.