How to Build an AI Voice Agent with Python and WebRTC: Step-by-Step Guide

Voice should be the default interface for AI. It is the sci-fi dream: talk into your watch and have an assistant answer your questions, book your movie tickets, and remember your dog's birthday.

That future isn't quite here yet, but AI voice agents are already embedded in everyday systems. Think customer support lines, language tutors, drive-thru ordering, and in-app copilots. Each lets you talk instead of type, using the same thing under the hood: an AI voice agent.

A few years ago, such an agent would have taken an entire team to build and encompassed a full codebase. Now, if you're a Python developer, building one is a single-file problem.

So that's what we'll do here. We'll build a working voice agent you can talk to in your browser during a Stream call. But we also want to help you grok the concepts, so we'll cover the architecture voice agents share, explain why the transport layer, not the AI, is the hard part, and walk through the design decisions that separate an agent that feels like a conversation from one that feels like a voicemail system.

What Is an AI Voice Agent?

An AI voice agent is a program that takes part in a live audio conversation as a full participant.

It listens to a continuous stream of a person's speech, uses a language model to decide how to respond, and answers in synthesized speech, in real time, taking turns the way a person would.

That definition rules out most voice features you've already seen. A voicemail transcriber runs speech-to-text in batch after the recording ends; an agent transcribes a live stream while you're still talking. A "read this aloud" button is one-directional TTS. And the classic voice assistant interaction (press button, say command, wait, hear result) is really a request/response API with audio attached; one utterance in, one utterance out, session over.

A voice agent, by contrast, holds a single continuous two-way session for the length of the call. Being a real conversation partner inside that session comes down to four specific behaviors:

1. Turn detection. Deciding when you've finished your turn versus paused mid-thought, sometimes called endpointing. A half-second of silence after "so what I'm thinking is" means keep listening; the same silence after "what do you think?" means respond.
2. Barge-in. Listening even while it's speaking, over a full-duplex audio channel, so when you cut in, it can stop mid-sentence, discard the unspoken rest of its reply, and yield the turn.
3. Latency. Keeping the gap between your last word and its first under about a second, ideally much less, or the exchange stops feeling live. (The next section puts hard numbers on this.)
4. State. Carrying the conversation history forward, so "no, the second one" resolves against options it listed a minute ago.

Get any of these wrong and users notice immediately, not as a bug, but as a vague sense that they're talking to a machine that isn't listening.

The Architecture: STT -> LLM -> TTS, in a Loop

Almost every voice agent is built from the same three components:

1. Speech-to-text (STT) transcribes the user's audio into text as they speak.
2. A large language model (LLM) takes the transcript (plus conversation history and instructions) and generates a response.
3. Text-to-speech (TTS) converts that response into audio that the user hears.

These run as a continuous loop for the duration of the call, all riding on a transport layer that moves audio between the user's microphone and your Python process:

Each stage of this pipeline is a solved problem.

STT providers like Deepgram or ElevenLabs transcribe streaming audio with word-level timestamps in real time. Any modern LLM can hold up its end of a conversation. TTS voices from providers like ElevenLabs are close enough to human that the difference rarely matters.

So, if the pieces are all commodity, why aren't voice agents trivial to build?

Why Real-Time Transport Matters

In human conversation, the gap between one person finishing and the other starting is about 200 - 300 milliseconds. Anything beyond roughly a second reads as hesitation, and anything beyond two, people start saying "Hello? Are you there?"

Now add up the voice agent loop. The user's audio has to travel from their microphone to your server, get transcribed, wait for the LLM's first tokens, get synthesized back into audio, and travel back to the user's speaker. Every one of those stages takes time, and the user experiences the sum.

That budget math has two consequences:

• Everything must stream. You can't wait for the full transcript before calling the LLM, or the full LLM response before starting TTS. Each stage has to start producing output while consuming input. What matters is time-to-first-audio, not time-to-complete-response.
• The network hops can't be an afterthought. If STT, LLM, and TTS each cost 200 - 400ms and you can't do much about it, the transport is where your remaining budget lives. Sending audio over a request/response protocol like HTTP, or even WebSockets, which run over TCP, means head-of-line blocking. One delayed packet holds up everything behind it while the connection waits for retransmission. For live audio, it's disastrous because a 200ms-old audio packet is worthless. Better to drop it and stay current.

This is exactly the problem WebRTC was built for. It runs audio over UDP with codecs and jitter buffers designed for one goal: keep latency minimal and quality acceptable, in that order of priority. It's the same technology underneath Google Meet-style video calls, and it's the reason a voice agent can respond in under a second while a WebSocket-based one struggles.

But WebRTC alone only solves the protocol. There's still geography. If your user is in Singapore and your server is in Virginia, physics adds 200ms+ of round trip before anything else happens. This is where Stream's global edge network comes in. The user connects to the nearest of Stream's edge servers, and audio travels the long haul over Stream's optimized backbone rather than the public internet. Your agent joins the call the same way, from wherever it's running.

The practical upshot is that you shouldn't build any of it. Not the transport, the edge routing, the codecs, or jitter buffers. You should get it as infrastructure and spend your time on what your agent actually does. That's the design philosophy behind Vision Agents, Stream's open-source Python framework for real-time voice and video AI, which is what we'll use to build ours.

Building the Agent

Time to write some code. The finished agent is a single file, and we'll walk through all of it. You'll need Python 3.10-3.13, uv, and API keys from four providers (all have free tiers):

• Stream for the WebRTC transport
• Deepgram for speech-to-text
• Google AI Studio for the Gemini LLM
• ElevenLabs for text-to-speech

Each of these is swappable. Prefer OpenAI or Anthropic for the LLM, or Cartesia for TTS? Or xAI's Grok-4 for the LLM paired with Fish Audio for TTS? It's a one-line change.

Project Setup

Create a project and install Vision Agents with the plugins we need:

1
2
3
mkdir ai-voice-agent && cd ai-voice-agent
uv init
uv add "vision-agents[getstream,deepgram,gemini,elevenlabs]" python-dotenv

Then create a .env file with your keys:

1
2
3
4
5
STREAM_API_KEY=your_stream_api_key
STREAM_API_SECRET=your_stream_api_secret
DEEPGRAM_API_KEY=your_deepgram_api_key
GOOGLE_API_KEY=your_google_api_key
ELEVENLABS_API_KEY=your_elevenlabs_api_key

The Agent

Here's the complete agent:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
# agent.py

"""A real-time AI voice agent: STT -> LLM -> TTS over WebRTC."""

from dotenv import load_dotenv

from vision_agents.core import Agent, AgentLauncher, Runner, User
from vision_agents.plugins import deepgram, elevenlabs, gemini, getstream

load_dotenv()

INSTRUCTIONS = (
    "You are a friendly voice assistant. "
    "You are on a live audio call, so keep responses short and conversational — "
    "one or two sentences unless the user asks for detail. "
    "Never use markdown, lists, or emoji: everything you say is spoken aloud."
)

async def create_agent(**kwargs) -> Agent:
    return Agent(
        # The transport layer: joins the call over WebRTC via Stream's
        # edge network, so audio hops to the nearest edge server instead
        # of crossing the world to a single origin.
        edge=getstream.Edge(),
        # How the agent appears to other participants on the call.
        agent_user=User(name="Voice Assistant", id="agent"),
        instructions=INSTRUCTIONS,
        # The pipeline: swap any of these three for another provider
        # without touching the rest of the file.
        stt=deepgram.STT(eager_turn_detection=True),
        llm=gemini.LLM(),
        tts=elevenlabs.TTS(),
    )

async def join_call(agent: Agent, call_type: str, call_id: str, **kwargs) -> None:
    call = await agent.create_call(call_type, call_id)
    async with agent.join(call):
        # Speak first so the user knows the agent is live.
        await agent.simple_response("Greet the user briefly and ask how you can help.")
        # Keep handling turns until the call ends.
        await agent.finish()

if __name__ == "__main__":
    Runner(AgentLauncher(create_agent=create_agent, join_call=join_call)).cli()

That's an entire AI voice agent. Let's break down what each part does.

The Instructions

This is a standard LLM system prompt, but with voice-specific constraints:

1
2
3
4
5
6
INSTRUCTIONS = (
   "You are a friendly voice assistant. "
   "You are on a live audio call, so keep responses short and conversational — "
   "one or two sentences unless the user asks for detail. "
   "Never use markdown, lists, or emoji: everything you say is spoken aloud."
)

Text-chat habits are wrong for audio. An LLM that answers with a five-bullet markdown list produces TTS output that reads the asterisks aloud. Telling the model it's speaking and to use speech patterns (e.g., short sentences) matters more than any fancy prompt engineering you'll do here. When you adapt this agent, this string defines your agent's actual job.

The Agent Definition

1
2
3
4
5
6
7
8
9
async def create_agent(**kwargs) -> Agent:
   return Agent(
       edge=getstream.Edge(),
       agent_user=User(name="Voice Assistant", id="agent"),
       instructions=INSTRUCTIONS,
       stt=deepgram.STT(eager_turn_detection=True),
       llm=gemini.LLM(),
       tts=elevenlabs.TTS(),
   )

This is the architecture diagram from earlier, expressed as six keyword arguments:

• edge=getstream.Edge() is the transport layer. It handles WebRTC session negotiation, connects to Stream's edge network, publishes the agent's audio track to the call, and subscribes to the user's audio track. Basically, everything from the "Why transport matters" section above is condensed into this single line.
• agent_user is the agent's identity on the call. It shows up as a named participant, just like a human would.
• stt, llm, and tts are the three pipeline stages. Each is a plugin; Vision Agents wires them together, so audio flows from the call into Deepgram, transcripts flow into Gemini, and Gemini's streaming output flows into ElevenLabs and back onto the call.

The eager_turn_detection=True flag on the STT is a latency optimization, which we'll talk about in the design decisions below.

Joining a Call

agent.join(call) is an async context manager:

1
2
3
4
5
async def join_call(agent: Agent, call_type: str, call_id: str, **kwargs) -> None:
    call = await agent.create_call(call_type, call_id)
    async with agent.join(call):
        await agent.simple_response("Greet the user briefly and ask how you can help.")
        await agent.finish()

Inside the with block the agent is live on the call, and when the block exits it cleans up the connection.

The first thing we do is simple_response(...). This sends a prompt to the LLM and speaks the result, so the agent greets the user instead of sitting in silence waiting to be spoken to. Then agent.finish() hands control to the framework's conversation loop: listen, transcribe, respond, speak, repeat, until the call ends.

Run It

1
uv run agent.py run

The first launch takes a few extra seconds. Vision Agents warms up the pipeline components and downloads a small local voice activity detection model (Silero VAD) it uses to tell speech from silence. Then it creates the call, prints the join link, and opens Stream's demo UI in your browser for you:

Open the link if the browser didn't launch on its own, allow microphone access, and say hello. You join under a randomly assigned demo name (we drew "Electric Empress"), and the agent sits alongside you as Voice Assistant, the identity we set in agent_user. It speaks first because join_call fires that simple_response greeting before the conversation loop starts:

Note the latency readout in the call's top bar, 21 ms on our run. That's the transport slice of the latency budget below, measured live rather than estimated.

Two things are worth trying before you move on:

1. Watch the terminal while you ask a question, and you'll see the loop from the architecture section run in order: transcript finalized, response tokens streaming in, synthesized audio heading back out.
2. Interrupt the agent mid-answer. It stops talking and hands you the turn, and the next section explains where both behaviors come from.

Key Design Decisions

The script works, but understanding why it works and where the knobs are is what lets you adapt it. It helps to see a full turn end-to-end first. Here's everything that happens between your voice and the agent's:

Everything in this section is about the timing inside that diagram, from when the turn gets declared over, to what happens when you talk over the reply, to where the milliseconds go.

Four decisions matter most.

1. Turn Detection: Knowing When to Speak

The hardest question in voice AI isn't "what should the agent say?" It's "when should it start saying it?"

Humans signal the end of a turn with intonation, grammar, and rhythm. A naive agent that responds after every pause will interrupt you mid-thought ("So I was thinking... [AGENT STARTS TALKING]"). One that waits too long feels laggy. This is a genuine ML problem, and voice stacks solve it with dedicated end-of-turn models rather than simple silence timers.

In our agent, turn detection comes built into Deepgram's STT, and the eager_turn_detection=True flag enables a clever latency trick. When the model suspects your turn is over, it starts the LLM speculatively. If the turn really is over, the response is already in flight, and you've saved a few hundred milliseconds. If you keep talking, the draft is discarded, and nothing is lost:

If you pick an STT provider without built-in turn detection (say, a local Whisper model), Vision Agents lets you plug in a standalone detector instead:

1
2
3
4
5
6
7
from vision_agents.plugins import smart_turn

agent = Agent(
    ...,
    stt=fast_whisper.STT(),
    turn_detection=smart_turn.TurnDetection(),
)

2. Interruption Handling: Letting the User Barge In

In real conversation, people talk over each other constantly. They correct, redirect, or say "no, silly!" An agent that plows through its full response while you're trying to interrupt it is instantly infuriating.

Vision Agents handles this automatically. When the user starts speaking while the agent is mid-response, the framework stops TTS playback, flushes any already-synthesized audio from the outgoing track (so stale speech doesn't bleed into the next turn), and goes back to listening. You get barge-in behavior without writing any code.

You can also control interruptibility per utterance when you script specific moments:

1
await agent.say("One moment while I check that.", interrupt=True)

3. Noise Suppression: Garbage In, Garbage Out

Turn detection and interruption handling are only as good as the audio feeding them. A TV in the background, a second person talking nearby, or keyboard noise can all trigger false barge-ins (the agent stops talking because it "heard" the user) or corrupt transcripts.

The right place to fix this is as early in the chain as possible, before the audio ever reaches your pipeline. Stream's client SDKs apply noise suppression and echo cancellation on the capture side, so what arrives at the STT is the user's voice, not their environment. If your users will call from cafés, cars, or open-plan offices, this is the difference between an agent that works in the demo and one that works in production.

4. The Latency Budget: Where the Milliseconds Go

Every stage of the loop spends part of your ~1-second budget. Roughly, for a well-tuned pipeline:

Two things to notice. First, the transport rows are small because of WebRTC and edge routing. Doing this over plain HTTP from another continent would mean those rows would dominate the table. Second, everything streams, so these stages overlap. The user starts hearing the response while the LLM is still generating its end.

There's one more architectural option on the latency frontier: speech-to-speech models like Gemini Live or OpenAI Realtime, which skip the STT and TTS stages entirely by processing and producing audio natively. In Vision Agents, that's a one-line change: llm=gemini.Realtime(), and drop the stt/tts arguments. You gain latency and lose control. No transcript to inspect, no ability to filter or modify the LLM's text before it's spoken, less choice over the voice. For most applications, the pipeline's control is worth its milliseconds, but it's good to know the trade exists.

Building Your Own AI Voice Agent

Everything specific to your agent lives in two places: the INSTRUCTIONS string (what the agent's job is) and the three pipeline components (which providers do the work). The transport, turn-taking, and interruption machinery doesn't change when you change the agent's purpose.

From here, some natural next steps:

• Give it tools. Vision Agents supports function calling and MCP servers, so your agent can look up orders, book appointments, or query your API mid-conversation.
• Swap providers. Try openai.LLM() or anthropic.LLM(), or a different ElevenLabs voice, and see how the agent's character changes. Or, swap in a fully local option if you need to run offline.
• Deploy it. uv run agent.py serve starts an HTTP server that spawns an agent for each incoming call. The first step from local demo toward production, with Docker and Kubernetes guides in the Vision Agents docs.

With the transport handled, a real-time voice agent is an afternoon project, and the interesting work is deciding what yours should do.