WebRTC Buffers

What Are Buffers?

Generally, a buffer refers to a dedicated area of memory used to store data before it is processed or temporarily used. Buffers can act as a buffer between a fast producer of data (e.g., a network connection) and a slower consumer (e.g., a processing algorithm). This prevents the consumer from being overwhelmed by data and allows it to work at its own pace. In some other cases, programs can access data quickly when needed by pre-loading data into a buffer, improving responsiveness and reducing waiting times.

Buffers play a crucial role in ensuring smooth and uninterrupted WebRTC calls. While some important buffers are mentioned here, other internal buffers may also be used for processing data in video systems.

Intro to Jitter Buffer

The jitter buffer acts as a temporary storage space for incoming audio and video data, specifically addressing the challenge of network jitter. Internet data delivery isn't always consistent. Packets can arrive out of order, delayed, or even disappear entirely. This variability in arrival times is called ”jitter.” Jitter disrupts the smooth playback of audio and video, causing audible glitches, stuttering video, and overall choppy communication.

The buffer can also help with packet loss. If a packet is missing entirely, the buffer can sometimes use techniques like packet loss concealment (as seen in the Media Resilience lesson) to estimate its content and maintain playback continuity. The jitter buffer can have either a static size or an adaptive size. As their names suggest, a static jitter buffer is a fixed-size buffer, while a dynamic one adapts to the current network conditions. During high amounts of jitter, an adaptive buffer size increases to accommodate more variability. This can slightly increase latency, but it ensures smoother playback.

In general, a jitter buffer should add a buffer of between 20 milliseconds and 60 milliseconds. The larger the buffer, the more latency is introduced into the call, affecting the user’s experience.

How Jitter Buffer Works

The jitter buffer sits at the receiving end of the network connection. As packets arrive, the buffer stores them regardless of their arrival time. The buffer then analyses the arrival patterns and estimates the actual playback time for each packet based on its original sequence. Once enough packets are available to reconstruct a frame (for video) or a short audio segment; the buffer releases them in the intended order and at the estimated playback times. Overall, this smooths out the playback, compensating for the jitter introduced by the network and creating a seamless experience for the user.

Video/Audio Buffers

The video buffer is not a separate buffer type but refers to the internal buffer within the jitter buffer specifically dedicated to video data. It performs tasks like reordering and potentially concealing lost video packets for smooth playback. Similarly, the "audio buffer" refers to the internal buffer within the jitter buffer dedicated to audio data. It manages audio flow to avoid glitches or delays and might adjust the playback rate for synchronization.

Outgoing Data Buffer

Underneath the hood, WebRTC uses the RTCDataChannel interface which represents a network channel that can be used for bidirectional peer-to-peer transfers of arbitrary data. Every data channel is linked to an RTCPeerConnection, and each peer connection can have up to a theoretical maximum of 65,534 data channels.

Each RTCDataChannel instance holds an outgoing data buffer that temporarily stores data before it's sent over the network. Unlike audio/video buffers, the data channel buffer isn't directly tied to managing network fluctuations. The outgoing data buffer prevents overwhelming the network by accumulating data and sending it in bursts when conditions are better. The buffer helps avoid congestion on the receiving end by holding data until the receiver signals readiness.

While you can't set the buffer size of an outgoing data buffer in an RTCDataChannel, you can access the bufferedAmount property to know how much data is currently queued. You can also subscribe to events which notify you when the buffer falls below a certain amount of data. To set this threshold, you can use the bufferedAmountLowThreshold property.

To get more information on RTCDataChannel and outgoing data buffers, you can check out Mozilla’s developer docs. Module 1 of WebRTC of the Brave also includes a lesson on working with RTCDataChannel which you can find here.

Conclusion

This lesson dived into the various types of buffers used in WebRTC. Buffers are crucial in the smooth operation of WebRTC and video calling.