How Does Adaptive Bitrate Keep Video Calls Smooth When Your Network Drops?

TL;DR

• A call's network capacity changes constantly. If the sender keeps sending at the old rate, packets queue up and the video freezes.
• The sender spots this from receiver feedback on packet delay, then cuts its bitrate by lowering detail, resolution, or frame rate.
• Audio is sent ahead of video, and in group calls the SFU forwards a lower simulcast layer to viewers on weak connections.
• Quality returns in steps once the network recovers. Adaptive bitrate can't fix a dead connection or the first few seconds of a sudden drop.

Video calls aren't static. Your underlying network, whether that's Wi-Fi or cellular, is constantly fluctuating underneath your call. Someone else also starts a call, or streaming YouTube, or your laptop decides now is the best time to sync a backup in the background. A second ago, the connection could carry 2 Mbps of video, and now it can carry one. The software has only a few hundred milliseconds to do something, and live video means nothing is stored to play while it does.

Adaptive bitrate is how the call copes. When the network can carry less, the sender immediately cuts how much video data it sends, so packets stop piling up in queues along the path. The picture gets softer or less fluid for a while, but the timing stays in sync. Audio stays continuous, and the video that arrives is current.

What Is Adaptive Bitrate?

Adaptive bitrate means the rate at which video is sent adjusts while the stream is running to match what the network can carry at that moment.

For a livestream or a video on demand, that works by encoding the video at several bitrates in advance and cutting each one into segments a few seconds long. The player watches how quickly the last segments arrived and picks which bitrate to fetch next.

A call has to do it live. Your laptop encodes one stream, and when the network changes, the sender tells the encoder to produce a smaller or larger one starting with the next frame. That feedback comes back several times a second, so the change lands within a few hundred milliseconds.

Why Does a Bandwidth Drop Freeze a Video Call?

Because the sender keeps sending at the old rate, and the extra data has nowhere to go but a queue.

Say your laptop is sending 2 Mbps of video and the connection drops to 1 Mbps. Every second, the laptop hands the network a megabit more than it can deliver. That extra sits in a buffer, in your router or somewhere at the ISP, waiting behind everything already there.

The delay grows fast:

• After 200 ms, 200 kilobits are queued. At 1 Mbps, that queue takes 200 ms to drain, so every new packet now waits about 200 ms longer than it did before the drop.
• After 1 s, a full megabit is queued, and each packet arrives about a second late.
• The delay keeps growing one second per second until the buffer fills, and then packets get dropped instead.

Your colleague's laptop can't wait that out. A streaming player holds seconds of video in a buffer, so a late chunk goes unnoticed. A live call can't do that, because anything it holds back puts the conversation that far behind.

So the phone keeps each packet in a jitter buffer for only tens of milliseconds, enough to smooth out small variations in arrival. A frame that turns up 800 ms late is well past that window. Packets that were dropped mean the frame can't be decoded at all, and every frame that depends on it is useless too, so the decoder asks for a fresh keyframe. That's the freeze you see.

Cutting the bitrate stops the pile-up at the source. If the laptop drops to 850 kbps for a moment, below the new capacity, the queue drains, the delay drops, and the call continues at the lower quality.

Here's an example: A two-person call at 720p with the sender putting out about 1 Mbps and its bandwidth estimate sitting at 1.9 Mbps. Then the sender's upload is capped at 300 kbps for 30 seconds:

Two seconds after the cap went on, the sender was still pushing out 720p frames, and the round-trip time it was measuring had gone from about 20 ms to just over a second. That second is the queue. Packet loss was at 30 percent, which is the queue overflowing, and two seconds later it was at 74 percent.

By then, the estimate had dropped from 1.9 Mbps to 205 kbps, and the encoder had been told to aim for 47 kbps. Two seconds later, the round-trip time was back to 18 ms because the sender had cut so far that the queue had drained. At that point, the picture was one frame a second, but the pile-up was over.

How Does the Call Know the Network Has Slowed Down?

It watches what happens to its own packets. The receiving side sends RTCP feedback several times a second, listing which packets arrived and when. From that, the sender figures out what the network is doing right now.

The main signal is delay that keeps growing. Suppose your laptop sends a burst of packets every 33 ms, one per frame at 30 fps, and the bursts start arriving 34, 35, 36 ms apart. Each one is spending a little longer in a queue than the one before. Google Congestion Control, the algorithm in libwebrtc and so in every major browser, fits a line through those delay measurements and treats a rising slope as overuse. It catches the problem while the queue is still building, before anything has been dropped.

On overuse, the controller cuts its estimate to 85 percent of the rate that was actually getting through and sets that as its new target. Packet loss is the second signal, and sustained loss pulls the estimate down too, but loss is a late warning. A queue has to overflow before anything is dropped, so the delay signal does most of the work.

What Actually Changes When the Bitrate Goes Down?

The encoder gets a lower target and can either compress each frame harder, encode fewer pixels, encode fewer frames, or a mix of the three.

The first row is why a lower bitrate doesn't automatically mean a lower resolution. A 720p stream at 400 kbps is still 720p, just with less detail in each frame. Our bandwidth table puts a good 720p conference picture in VP9 at about 930 kbps and 480p at about 310 kbps, but those are quality targets, and an encoder will go well under them before it changes the frame size.

Going back to our example, it did all three things:

• It dropped frames first, because that's the one change an encoder can make on the very next frame. Two seconds after the cap, its target was 47 kbps, and for the next four seconds it stayed at 720p and managed 16 frames a second, then 10, then 1.
• It lowered the resolution next. Eight seconds in, it switched to 960x540, and on the way back up it returned to 720p as soon as its budget passed 400 kbps.
• It compressed harder once it had settled. Three seconds after the switch, it was back at 30 frames per second on about 150 kbps, a 540p picture on a seventh of the bits it had been spending on 720p.

Which of the three the encoder leans on depends on what's in the frame. For a person talking, motion matters more than fine detail, so it keeps the frame rate and gives up resolution. For a shared spreadsheet, the text has to stay readable, so it keeps the resolution and lets the frame rate fall to a few frames a second. The browser exposes this as the sender's degradationPreference, and a content hint on the track tells the encoder which kind of content it's looking at.

The codec decides how far any of this has to go. AV1 delivers more quality per bit than H.264, which is why we publish in AV1 from devices that can encode it efficiently and fall back to VP9 or H.264 on the rest.

How Does the Call Keep Video From Getting in the Way of Audio?

By pacing video packets onto the network instead of sending each frame in one go, and by putting audio at the front of the queue.

A video encoder produces bursts. At 5 Mbps and 30 fps, each frame is about 21 kB, or 18 RTP packets, and if the sender pushed them all out the moment the frame was ready, you'd get a 167 Mbps burst every 33 ms followed by 32 ms of nothing. The average rate is fine, but those bursts fill buffers along the path, and if one burst takes 100 ms to get through, every audio packet queued behind it is 100 ms late.

Packets go into a queue and are released onto the network at a steady rate, with a separate queue per track. Audio has the highest priority, then retransmissions, then video and error correction, then padding. A voice packet never waits behind a large video frame.

The pacer also enforces the bandwidth budget. That budget covers more than the picture. It also includes:

• The RTP, UDP, and IP headers on every packet.
• Retransmissions of packets the receiver reported lost.
• Forward error correction, the extra packets sent so the receiver can rebuild a lost one without asking.

Adding repair traffic without cutting media to make room for it worsens congestion, so the controller's target has to cover it all.

When even the lowest video layer won't fit, the best option is to stop sending video and keep the audio going. Stream SDKs do this by default. Low-bandwidth handling pauses some or all remote video, marks the paused participant's tile with an icon, and resumes video as conditions improve.

In a Group Call, Does One Bad Connection Lower Everyone's Quality?

Each participant's connection is handled independently.

That's possible because of simulcast. Your laptop publishes three encodings of the same camera at once, at full, half, and quarter resolution (Stream SDKs label the layers f, h, and q). The SFU receives all three and picks which one to forward to each viewer. Nothing is re-encoded on the server. Instead, it chooses between existing streams.

Which layer a viewer gets depends on their download capacity and on how big your video is on their screen. There's no point sending 720p to a thumbnail, so Dynascale gives a small tile the half- or quarter-layer and switches it back to full when the tile goes full screen. It works the other way too. If nobody is showing your video at full size, our servers tell your laptop not to encode and upload the full layer.

When a viewer's bandwidth estimate can't keep up with the layer it's receiving, it asks the SFU for a lower one, then asks for a higher one again as conditions improve.

The direction of the drop decides what happens:

• Your download drops. The SFU forwards smaller layers of everyone else to you. Nobody else notices.
• Your upload drops. Your laptop can't keep publishing the full layer, so it stops, and everyone watching you gets a half- or quarter-layer instead.

With twelve tiles on screen and a shrinking connection, the SFU has to fit everyone into what you can carry, so it pauses some videos and keeps the rest at a usable quality.

How Does the Picture Improve When the Network Recovers?

The controller keeps measuring after it backs off, and it raises its estimate in steps instead of jumping back to where it was.

While the estimate is well below the highest rate the connection carried before the drop, it grows by about 8 percent a second. Once it gets close to that old rate, it switches to small fixed increments, so it doesn't overshoot into the same queue it just got out of. If a step makes the delay start rising again, it backs off again, and the climb starts over from there.

In the example above, the cap came off at about 58 seconds. It took 18 seconds to get back above 500 kbps, where the encoder went back to 720p, and 29 seconds to get the sender back to its full 1 Mbps. The steps got smaller once it passed 1.6 Mbps, close to the 1.9 Mbps it had estimated before the drop.

Layer switching in a group call also has built-in margins. To move a viewer up a layer, the bandwidth has to sit above that layer's requirement by a margin, 20 percent, say, for a sustained period, and it has to fall well below before the viewer is moved down. Without that, a connection hovering around a threshold would flip sharp, blurry, sharp every few seconds, and that's worse to watch than a steady picture one step down.

What Can't Adaptive Bitrate Fix?

A dead connection, and the first few seconds of a sudden network change.

If no packets get through, sending fewer of them changes nothing. The call has to wait for the connection to come back, and what matters then is how fast the client reconnects, which our SDKs handle with a fast failover and reconnect protocol.

It also can't react instantly. Feedback has to travel back to the sender, the estimator needs a few reports before it's sure, and the encoder needs a frame or two to change its output. In our example, the viewer's picture went from 30 frames a second to 10 in the first second after the cap, to 4 in the next, then froze for 3.8 seconds while the sender still pushed 720p frames into a queue that had already overflowed. Eight seconds in, the encoder dropped to 540p, and the picture moved again.

Over the whole 30 seconds, the viewer saw eight freezes totaling 7.3 seconds, and 5.2 of those seconds were in the first eight. Once the sender had settled under the cap, the rest were short ones, a second or less, where packets got lost. The jitter buffer absorbs the small stuff, packets arriving 20 ms early or late, but not a sustained mismatch between sending rate and capacity, and making it bigger to try only adds delay to everything.

Nor does a lower bitrate bring back packets that are already gone. Those need retransmission or forward error correction, and a retransmission is only worth asking for if it can arrive before its frame is due to play. On a long round trip, it can't, and the receiver conceals the loss or requests a keyframe instead.

How Do You See This Happening in Your Own Calls?

Every call on the Stream platform reports its stats while it runs. The call stats report emits every two seconds by default, or every second if you ask it to, with the resolution, frame rate, and bytes sent for each video stream, the sender's bandwidth estimate, and a quality limitation reason that says whether the encoder is being held back by bandwidth, by CPU, or not at all. The charts in this piece are those numbers. In this two-person call, the SDK published a single stream with three temporal layers, so everything in the charts is the encoder adapting on its own. In a bigger call, you'd also see the f, h, and q layers come and go as the SFU asks for them.

The dashboard shows the same data for every call after the fact, with jitter, round-trip time, and packet loss alongside it, and the quality and latency guide covers the target resolutions, codec selection, and bandwidth numbers behind the defaults. If you're starting from scratch, the Video SDK docs are the place to begin. Simulcast, Dynascale, and the low-bandwidth fallback are on from the first call you make.