What Is Voice Activity Detection?

TL;DR

• VAD decides, frame by frame, whether audio contains speech. It's the timing signal every voice AI agent runs on.
• Signal-based VAD (like WebRTC's) is cheap but easily fooled by noise; neural VAD (like Silero) costs almost nothing more and holds up in noisy, real-world audio.
• VAD gates transcription, sets the pause length before a response starts, and lets an agent detect when it's being interrupted.
• VAD only detects sound, not intent. Pair it with semantic turn detection for agents that shouldn't cut people off mid-thought.

Voice activity detection (VAD) is the process of deciding, frame by frame, whether an audio stream contains human speech.

A VAD model listens to short slices of audio, usually 10 to 32 milliseconds each, and outputs a probability that the slice contains speech. That probability gets thresholded into a running yes or no signal. Someone is talking, or they aren't.

For voice AI agents, this signal is the foundation of conversational timing. VAD tells the agent when to start processing audio, when the user has finished speaking and a response is expected, and when to stop mid-sentence because the user started talking again. Speech-to-text, response generation, and interruption handling all key off VAD events.

The technique long predates voice agents. Telephony systems have used VAD for decades to avoid transmitting silence. What changed recently is accuracy: neural VAD models are now reliable enough in noisy, real-world audio to run turn-taking for a live conversation.

Let's break it down.

How Does Voice Activity Detection Work?

Audio is split into short frames, and each frame is classified as speech or non-speech. The interesting part is how that classification happens, and there are two families of approach.

Signal-based detectors measure properties like energy, zero-crossing rate, and spectral shape, then compare them against thresholds or a statistical model. WebRTC's built-in VAD, which uses Gaussian mixture models, is the classic example. These detectors cost almost nothing to run, but they struggle to tell loud from spoken. A keyboard, a closing door, or background music can all read as speech.

Neural detectors replace the hand-built rules with a small network trained on large corpora of labeled speech and noise. The model outputs a per-frame speech probability, and because it has learned what speech sounds like across languages, accents, and recording conditions, it holds up in exactly the noisy environments that break energy-based detection. The networks are tiny, a few hundred thousand parameters, and still run in well under a millisecond per frame on a CPU.

Raw per-frame decisions are jittery, so every practical VAD wraps the classifier in smoothing logic:

• A probability threshold, commonly around 0.5, above which a frame counts as speech
• A minimum speech duration, so a cough or a click doesn't open a speech segment
• A minimum silence duration before a segment closes, so a breath mid-sentence doesn't end the turn
• Padding added to each segment boundary, so the first and last syllables don't get clipped

The result is a small state machine that emits two timestamped events: speech started, and speech ended.

Here's what that looks like in a small app:

Why Does VAD Matter for Voice AI Agents?

A voice agent's sense of timing comes almost entirely from VAD events. Most production agents run a cascaded pipeline: audio in, speech-to-text, a language model, text-to-speech out. VAD sits at the front of that pipeline and gates everything behind it.

VAD handles four jobs in that loop:

• Gating transcription. Streaming STT transcribes whatever it's fed, and models in the Whisper family are known to hallucinate text when given silence or noise. Running STT only on VAD-approved speech avoids those phantom transcripts and cuts inference cost, since the recognizer sits idle between turns.
• Endpointing. When VAD emits a speech_end event and the silence holds, the agent treats the turn as complete and starts generating a response. That silence window sets the pause the user feels. Gaps in human conversation run a few hundred milliseconds, so every extra half-second of waiting reads as lag.
• Barge-in. VAD keeps running while the agent's own audio plays. Speech detected mid-playback means the user is interrupting, and the agent should stop talking and listen. Echo cancellation keeps the agent's voice out of its own microphone feed so it doesn't interrupt itself.
• Cost control. STT and LLM inference are the expensive stages of the loop. VAD runs continuously for near-zero cost, which means nothing expensive spins up until someone actually speaks.

Most of the tuning tension lives in the endpointing window. A short minimum silence makes the agent snappy but prone to cutting people off mid-thought. A long one makes it patient but slow, with dead air after every turn. Production agents typically land somewhere between 300 and 800 milliseconds and adjust per use case, since a customer reading out an account number pauses very differently from someone asking a quick question.

How Is VAD Different From Turn Detection?

VAD detects sound; turn detection infers intent.

A VAD can tell you that speech stopped. It can't tell you whether the speaker finished their thought, and those are different questions. A pause after "my account number is" and a pause after "thanks, that's everything" look identical at the audio level.

Silence-based endpointing, meaning a VAD plus a timer, treats both pauses the same way, which is a common reason agents interrupt people mid-sentence. Semantic turn detection models address this by looking at what was said, not just whether sound is present. They take the running transcript, sometimes the audio too, and classify whether the turn is actually complete.

The two work together rather than competing. VAD supplies candidate boundaries cheaply and continuously; the turn detector rules on the boundaries that matter. An agent without VAD has nothing to hand the turn detector in the first place.

What Is Silero VAD?

Silero VAD is the open-source neural VAD that shows up across modern voice stacks, from voice agent frameworks to transcription pipelines built around Whisper. It's MIT-licensed, about 2 MB, and fast enough to be a rounding error in any latency budget. The project has been iterating since 2020, with the current v6 models released in 2025.

The library exposes the smoothing parameters described earlier directly: threshold, minimum speech duration, minimum silence duration, and speech padding. In practice, those four numbers are where nearly all deployment effort goes.

Here's how we implemented it for the speech detection above:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
# pip install silero-vad torch torchaudio
from silero_vad import load_silero_vad, read_audio, VADIterator

SAMPLE_RATE = 16000
CHUNK = 512  # 32 ms at 16 kHz -- one VAD frame

model = load_silero_vad()
wav = read_audio("utterance.wav", sampling_rate=SAMPLE_RATE)

# The smoothing state machine wraps the raw per-frame probabilities.
# These four numbers are where nearly all VAD tuning happens:
vad = VADIterator(
    model,
    sampling_rate=SAMPLE_RATE,
    threshold=0.5,                 # probability above which a frame counts as speech
    min_silence_duration_ms=300,   # silence needed before a segment closes
    speech_pad_ms=100,             # padding kept on each side of a segment
)

for i in range(0, len(wav) - CHUNK, CHUNK):
    chunk = wav[i : i + CHUNK]

    prob = model(chunk, SAMPLE_RATE).item()   # raw, jittery per-frame probability
    event = vad(chunk, return_seconds=True)   # smoothed, timestamped turn events

    if event:
        print(f"{event}  (p={prob:.2f})")
        # -> {'start': 0.54}  (p=0.94)
        # -> {'end': 1.71}    (p=0.08)

vad.reset_states()

The model itself rarely needs touching; the defaults detect speech well, and the real work is matching the state machine to your acoustic environment and conversation style.

Where Else Does VAD Show Up?

Turn-taking for AI agents is the newest job, but VAD has been part of production audio systems for decades:

• Transcription pipelines, where long recordings get split at speech boundaries and silence gets stripped before batch STT, cutting compute and removing the silent stretches that cause hallucinated text
• Telephony and VoIP, where codecs stop transmitting during silence and substitute comfort noise, a technique called discontinuous transmission that meaningfully reduces bandwidth over a call
• Meeting and calling apps, for active speaker indicators, auto-mute suggestions, and talk-time analytics
• Speaker diarization, since working out who spoke when starts with detecting that anyone spoke at all
• Wake word and command systems, where the recognizer only runs on segments the VAD flags as speech

Do You Need To Build VAD Yourself?

Almost never. The detection problem is well solved by small open models, and Silero covers the large majority of cases without modification.

The work that remains is integration and tuning, like wiring VAD events into your STT, endpointing, and interruption logic, then adjusting thresholds and silence windows against real audio from your users (because phone calls, laptop mics, and cars all behave differently).

Real-time platforms have started absorbing that integration work too. Stream's voice and video SDKs, for example, ship speech detection and turn handling as part of the agent tooling, so VAD arrives already wired into the transport layer rather than as another component to assemble.

Wherever it comes from, it's worth testing deliberately. Have real users interrupt the agent, pause mid-number, and talk in noisy rooms. VAD failures are invisible in a quiet demo and obvious the moment the audio gets realistic.