How Does a Jitter Buffer Stop Video Calls From Sounding Choppy?

TL;DR

• Choppy call audio is almost always jitter (packets arriving at uneven times), not a bandwidth problem.
• A jitter buffer holds incoming packets for a few tens of milliseconds, reorders them, and plays them out at a steady rate, trading a little latency for smooth audio.
• Buffer size is a tradeoff: bigger targets smooth over more jitter but add lag. WebRTC's NetEQ resizes itself automatically instead of using one fixed value.
• Video needs its own jitter buffer too, but tolerates missed deadlines better than audio. A frozen frame is less noticeable than an audio glitch.

"I thi.. we sho..ld prob..ly push the lau.. to Thursday. Hello?" Everyone has sat through that call. The voice on the other end keeps cutting out, goes robotic for a second, and snaps back as if nothing happened. It's tempting to blame the connection, but a voice stream only needs a few dozen kilobits per second.

The problem is usually timing. The packets are arriving, but not necessarily in the right order.

The fix is simple: a jitter buffer. Every real-time audio system has one.

How Is Call Audio Turned Into Packets?

Before the network can mangle anything, the sender turns continuous audio into discrete packets. The client captures the microphone input, chops the signal into fixed-size frames, encodes each frame with a codec like Opus, and stamps it with a sequence number and a timestamp before handing it to the network. That's what the real-time transport protocol (RTP), the protocol underneath nearly every calling product, does.

Nearly every calling product now builds this pipeline on WebRTC, the open real-time framework that ships in every browser. WebRTC also settles the numbers. Whatever the microphone captures gets resampled to 48 kHz, enough to cover everything human ears can hear, and Opus encodes it in 20 ms frames. Shorter frames waste a growing share of bandwidth on packet headers. Longer ones add latency and make every lost packet cost more audio.

const SAMPLE_RATE = 48000;  // WebRTC's native rate
const FRAME_MS = 20;        // Opus's default frame duration
const FRAME_SAMPLES = SAMPLE_RATE * FRAME_MS / 1000;  // 960 samples

So one packet carries 20 ms of sound, and the sender emits one every 20 ms on a fixed cadence, numbered 0, 1, 2, 3. Everything downstream reasons in those terms. Frame #n covers samples n×960 through (n+1)×960, and nothing else about the audio matters to the buffer.

What Does the Network Do to Those Packets?

An IP network does three unhelpful things to that tidy stream.

• Every packet takes time to cross the network. That base delay comes from distance and router hops, and by itself, it's harmless. A call with a constant 80 ms of delay sounds perfectly smooth.
• The time varies per packet. Packet #101 might take 25 ms, while #102 takes 90 ms because #102 sat behind someone's video upload in a router queue. This variation is jitter, and jitter is the specific problem a jitter buffer solves.
• Some packets never arrive at all. Real-time audio rides on UDP, which doesn't retransmit, because a resent packet would show up after its moment to be played had passed.

Jitter also causes packets to arrive out of order. Say #101 leaves the sender at time zero and #102 leaves 20 ms later. If #101 sits in a congested queue for 90 ms, it arrives at the 90 ms mark. If #102 only sits for 25 ms, it arrives at 45 ms, ahead of a packet sent before it. No router swaps packets on purpose. A later packet just sometimes waits less than an earlier one, which is why the network model below has no separate reordering step.

We can model all of this in just a few lines:

send(packet) {
  const now = performance.now();

  // Queueing jitter: -ln(U) is a standard way to draw an exponential.
  const queueing = -Math.log(1 - Math.random()) * this.jitterMs;

  const transit = this.baseDelayMs + queueing + this.#spikeExtra(now);
  const lost = Math.random() * 100 < this.lossPct;

  const record = { seq: packet.seq, sentAt: now,
                   arriveAt: now + transit, transitMs: transit, lost };

  if (lost) {
    // The receiver is told nothing. Silence is what loss looks like.
    setTimeout(() => this.onDrop(packet, record), transit * 0.5);
  } else {
    setTimeout(() => this.onDeliver(packet, record), transit);
  }
}

The exponential draw is the classic model for queueing delay. Most packets incur a small extra wait, and a few incur a nasty one; that long tail is what makes buffer sizing interesting.

Also note that a lost packet triggers no action on the receiving side. Loss looks exactly like a very late packet, and the receiver can't tell the two apart until it stops waiting.

What Does the Buffer Do When a Packet Arrives?

The receiving side lives under a hard constraint. The sound card needs the next 20 ms of audio every 20 ms, exactly on time, for the whole call. Miss a deadline, and the listener hears a glitch. So the buffer's job is to convert variable network delay into constant total delay.

It does that by separating when a packet arrives from when it plays. Arrival just files the packet in a map keyed by its sequence number. Playing occurs at a fixed time, say, 100 ms after the sender produces the frame, and that 100 ms is the target delay.

A packet that crosses the network in 30 ms sits in the buffer for 70 ms. One that takes 85 ms sits for 15 ms. The network delay varies, the buffer wait varies by the same amount in the other direction, and the total is always 100 ms. The only packets that still cause trouble are the ones slower than the target itself, because their moment to play passes while they're still in flight.

insert(packet, transitMs) {
  // Feed the jitter estimator regardless of the packet's fate.
  this.transits.push(transitMs);
  if (this.transits.length > 250) this.transits.shift(); // ~5 s window
  if (this.mode === 'adaptive') this.#maybeRetarget();

  // Too late: the playout head has already moved past this frame.
  // Its slot got concealment. A jitter buffer never rewinds.
  if (this.nextSeq !== null && packet.seq < this.nextSeq) {
    this.stats.late++;
    return 'late';
  }

  if (this.packets.has(packet.seq)) return 'duplicate';

  // The normal case. Out-of-order arrival needs no special handling:
  // the Map keyed by seq "sorts" for free.
  this.packets.set(packet.seq, packet);
  this.maxSeq = Math.max(this.maxSeq, packet.seq);
  if (this.nextSeq === null) this.nextSeq = packet.seq; // stream anchor
  return 'buffered';
}

Out-of-order arrival requires no special handling at all, because packets are stored by sequence number rather than arrival order, and playout requests "seq N" when N's moment comes. And a packet that arrives after its slot has already played gets discarded. Playback has moved on.

What Happens When It's Time To Play a Frame?

Every 20 ms, the playout side calls pop() and must get a value back because the sound card won't wait. (Here, that clock is a Web Audio lookahead scheduler, since JavaScript timers on their own are far too sloppy for audio.)

pop() {
  const depth = this.depthMs;
  const target = this.targetMs;

  // Empty buffer: stop consuming and let arrivals pile back up to the
  // target before resuming. One clean pause instead of many glitches.
  if (this.maxSeq < this.nextSeq && !this.prebuffering) {
    this.prebuffering = true;
    this.stats.underruns++;
  }
  if (this.prebuffering) {
    if (depth >= target) this.prebuffering = false;
    else return { status: 'prebuffer', depthMs: depth, targetMs: target };
  }

  const packet = this.packets.get(this.nextSeq);
  const seq = this.nextSeq;
  this.nextSeq++; // deadline passed, the head advances no matter what

  if (packet) {
    this.packets.delete(seq);
    return { status: 'hit', packet, seq };
  }

  // The frame's moment came and its packet isn't here. Playout must
  // not stall, so the caller conceals: 20 ms of plausible filler.
  return { status: 'miss', seq };
}

pop() has three possible answers:

• A hit means the packet was waiting. Its 20 ms of samples go to the speaker and the playout head advances.
• A miss means the frame's deadline arrived and its packet didn't, whether it's still in flight or gone forever. The head advances anyway, and the caller synthesizes filler audio for the slot.
• Prebuffering means the buffer runs completely dry, so playout holds still and lets arrivals pile back up to the target before resuming. That turns what would have been a string of scattered glitches into one longer pause.

How Big Should the Jitter Buffer Be?

A bigger target means late packets still make their slot, so playback gets smoother. But a bigger target also means every packet plays later, so the conversation gets laggier. Somewhere past roughly 200 ms of one-way delay, people start talking over each other. You want the smallest target that stays smooth on the network you actually have.

There are three basic approaches to setting it.

• No real buffer, meaning a target of a single frame, plays everything as soon as possible. Latency is great, and on a jittery network the audio is terrible.
• A fixed target uses a hand-picked value. 100 ms flattens most Wi-Fi jitter. 400 ms sounds flawless and adds enough delay to wreck a two-way conversation.
• An adaptive target measures the network and resizes itself, which is what WebRTC's NetEQ does.

Every crackle in that clip is a packet that missed its 20 ms deadline.

What the second clip can't convey is the cost. Added delay is inaudible in one-way playback and only shows up when two people try to take turns talking.

Does Video Need a Jitter Buffer Too?

Yes. A video call runs two jitter buffers side by side, one for the audio track and one for the video track. Video packets cross the same network and experience the same jitter, so the video buffer holds and reorders them the same way the audio buffer does.

Three things work differently on the video side.

• A video frame is much bigger than a packet, so each frame arrives as several packets. The buffer must collect all of them before the decoder can use the frame, which means a single late packet can hold up the whole frame.
• Video can ask the sender to resend a lost packet and still use it. Audio doesn't, because the resent packet would arrive after its moment to play had passed. A frame that's still being assembled can wait those extra milliseconds.
• A missed deadline looks different. When audio misses, you hear a gap or a glitch. When video misses, the screen keeps showing the last frame a little longer, and most people never notice a brief freeze.

That last difference is why audio gets the tighter buffer, and why "the call is choppy" complaints are almost always about sound.

Where Does the Buffer Sit In a Real Call?

In a production WebRTC stack, you don't build any of this.

NetEQ, WebRTC's audio jitter buffer, handles adaptive sizing, time-stretching, and concealment on every receiving track, and the video path has its own frame assembly and de-jittering. What you can do is observe it and nudge it. The inbound RTP stats from getStats() expose jitter, jitterBufferDelay, and jitterBufferEmittedCount (divide the last two for the average buffer delay), plus concealedSamples and concealmentEvents on audio tracks. If concealment counts climb while bandwidth looks fine, jitter is the problem. And the jitterBufferTarget attribute on RTCRtpReceiver lets you manually trade latency for smoothness when you need to.

The same applies if you build on Stream. The Video SDKs run on WebRTC, so adaptive jitter buffering is already in effect in every call, and call stats expose the jitter, loss, and latency numbers you'd want to see when a user reports choppy audio. Understanding the buffer mostly matters for debugging, because it tells you which of those numbers to read and what they actually mean.