Learn Audio and Video Codecs on WebRTC

In WebRTC, the selection and implementation of audio and video codecs are fundamental to achieving high-quality, efficient real-time communication. Codecs (coder-decoders) are algorithms that compress and decompress digital media, significantly affecting bandwidth usage, quality, and compatibility across devices and networks.

This lesson provides a comprehensive exploration of the codecs used in WebRTC applications, their characteristics, strengths, weaknesses, and the critical factors to consider when selecting the most appropriate codec for your specific use case. Understanding these elements will enable you to optimize your WebRTC applications for superior performance, wider compatibility, and enhanced user experiences.

Audio Codecs in WebRTC

Audio codecs are essential for compressing voice and sound data to manageable sizes while maintaining clarity and quality. WebRTC supports several audio codecs, each with unique characteristics suitable for different scenarios.

Opus: The Modern Standard

Opus has emerged as the gold standard for WebRTC audio due to its remarkable versatility and efficiency. Developed as an open standard by the Internet Engineering Task Force (IETF), Opus combines technologies from the SILK codec (used by Skype) and CELT (low-latency audio coding).

Key Characteristics of Opus

1. Versatility: Opus excels across a wide spectrum of audio applications, from low-bitrate voice communication (starting at 6 kbps in specification, though the effective floor in browsers is ~8-9 kbps) to high-fidelity stereo music streaming (up to 510 kbps).

2. Adaptive Bitrate: One of Opus's standout features is its ability to dynamically adjust quality based on network conditions:

   • During network congestion, it can operate at lower bitrates while maintaining intelligibility
   • When bandwidth is plentiful, it can seamlessly scale up to deliver higher quality audio
   • This adaptation happens in real-time without interruptions or artifacts

3. Low Algorithmic Latency: Opus achieves impressively low algorithmic delay (as little as 2.5ms in CELT mode), though typical implementations use 20ms frames with around 26.5ms total delay. This low latency is ideal for real-time communication where even small delays can disrupt natural conversation flow.

4. Error Resilience: Opus incorporates robust mechanisms to handle packet loss, maintaining audio quality even under challenging network conditions.

5. Universal Browser Support: All major browsers implement Opus as their primary WebRTC audio codec, ensuring excellent cross-platform compatibility.

Implementation Example

            // Setting preferred audio codec (Opus) with specific parameters
const audioTransceiver = pc.addTransceiver('audio');
audioTransceiver.setCodecPreferences([
  {
    mimeType: 'audio/opus',
    clockRate: 48000,
    channels: 2,  // Required for stereo=1 to work in Chrome/Edge
    sdpFmtpLine: 'minptime=10;useinbandfec=1;stereo=1'
  }
]);

// Note: For Chrome/Edge to actually use stereo, you may need to:
// 1. Set channels:2 as above
// 2. Disable DTX or add a second encoding
// 3. On Safari, you may need to additionally use SDP munging
//    to ensure codec preferences are respected
        

G.711: The Legacy Standard

G.711 is one of the oldest audio codecs still in common use today. Developed in 1972 as a telephony standard, it remains relevant in WebRTC for specific use cases, particularly where legacy compatibility is required.

Key Characteristics of G.711

1. Legacy Compatibility: G.711 serves as the common denominator in telephony systems and VoIP applications, making it essential for WebRTC applications that need to interface with traditional phone systems.

2. Simplicity: The codec uses a straightforward pulse code modulation (PCM) technique with two main variants:

   • μ-law (used primarily in North America and Japan)
   • A-law (used in Europe and most other countries)

3. Fixed Bitrate: Unlike Opus, G.711 operates at a fixed bitrate of 64 kbps, with no adaptation capabilities.

4. No Compression: G.711 offers minimal compression, which results in higher bandwidth usage but also means very low computational requirements and extremely low latency.

5. Limited Fidelity: With a sampling rate of 8 kHz, G.711 captures frequencies up to 4 kHz, which is sufficient for speech intelligibility but inadequate for music or complex audio.

Implementation Note

G.711 is typically included in WebRTC implementations by default and requires no special configuration, though it usually ranks lower in preference compared to more modern codecs.

Newer Audio Codecs: AAC and Enhanced Voice Services (EVS)

While Opus and G.711 are the primary audio codecs in WebRTC, other codecs have limited availability:

AAC (Advanced Audio Coding)

AAC offers high-quality audio compression and is widely used in digital audio broadcasting, streaming, and storage. In WebRTC, its support is very limited:

• Safari-Only Support: As of 2025, only Safari (macOS/iOS) exposes AAC for WebRTC (audio/mp4a-latm); Chromium removed it in M71 and Firefox does not signal AAC in its capabilities
• HTTPS Required: Safari requires HTTPS for AAC, and older iOS 15 devices need isac fallback support, making implementation and testing more complex
• Higher Quality at Lower Bitrates: Compared to older codecs, AAC delivers superior quality at comparable bitrates
• Content Delivery Integration: Potentially useful when WebRTC streams need to integrate with content delivery networks, but should be treated as proprietary/optional, not a core WebRTC codec

EVS (Enhanced Voice Services)

EVS is a successor to AMR-WB (Adaptive Multi-Rate Wideband) designed for 4G/5G voice services:

• Super-Wideband Audio: Supports frequencies up to 16 kHz
• Improved Speech Quality: Particularly effective for capturing speech nuances
• Limited WebRTC Support: Currently requires specialized implementations

Video Codecs in WebRTC

Video codecs significantly impact the quality, bandwidth consumption, and processing requirements of WebRTC applications. The ideal video codec selection depends on your specific use case, target audience, and platform requirements.

H.264/AVC: The Universal Standard

H.264/AVC (Advanced Video Coding) is arguably the most widely deployed video codec worldwide. Developed by the Joint Video Team of ITU-T VCEG and ISO/IEC MPEG, it balances compression efficiency with computational complexity.

Key Characteristics of H.264

1. Universal Compatibility: H.264 enjoys universal support across devices, browsers, and platforms, making it the safest choice for broad compatibility.

2. Hardware Acceleration: Most modern devices include dedicated hardware for H.264 encoding and decoding, significantly reducing CPU usage and power consumption.

3. Effective Compression: H.264 delivers reasonable quality at moderate bitrates, though newer codecs may achieve better efficiency.

4. Profile Flexibility: H.264 offers multiple profiles from Baseline (lowest complexity) to High (best quality) to accommodate different device capabilities.

5. Licensing Considerations: H.264 involves patents and may require licensing for commercial use, though web browsers handle this for most WebRTC applications.

Implementation Example

            // Setting H.264 as the preferred video codec
const videoTransceiver = pc.addTransceiver('video');
const codecs = RTCRtpSender.getCapabilities('video').codecs.filter(
  codec => codec.mimeType === 'video/H264'
);
videoTransceiver.setCodecPreferences(codecs);
        

VP8: The Open Alternative

VP8 is an open and royalty-free video codec developed by On2 Technologies and acquired by Google. It was the first mandatory video codec in WebRTC specifications.

Key Characteristics of VP8

1. Royalty-Free: VP8's open nature avoids the licensing concerns associated with H.264.

2. Comparable Quality: At similar bitrates, VP8 delivers quality comparable to H.264 Baseline profile.

3. Universal WebRTC Support: As a mandatory codec in the WebRTC specification, VP8 is supported by all WebRTC-compatible browsers.

4. Moderate Computational Requirements: While not as optimized for hardware as H.264, VP8 offers reasonable efficiency on modern devices.

5. Temporal Scalability: VP8 supports temporal scalability, allowing adaptation to changing network conditions.

VP9: The Enhanced Open Option

VP9 is Google's successor to VP8, offering significant improvements in compression efficiency while maintaining the royalty-free approach.

Key Characteristics of VP9

1. Superior Compression Efficiency: VP9 achieves approximately 50% bitrate reduction compared to VP8 for similar quality.

2. Scalability: Designed with scalability in mind, VP9 effectively handles video resolutions from low-resolution to 8K.

3. Mixed Hardware Support: While initially lacking hardware acceleration, VP9 support in hardware has increased significantly. Support varies by platform:

   • Chrome/Edge: Full encode and decode support
   • Firefox desktop: Software encode by default; hardware encode only with gfx.vp9.hw.enabled=true on modern GPUs
   • Firefox mobile: Decode-only support
   • Safari: Decode support since Safari 17.4; encode is not yet exposed to WebRTC
   • iOS: Hardware decode only; encode remains experimental

4. HDR Support: VP9 supports high dynamic range (HDR) content for enhanced visual quality.

5. Open Standard: Like VP8, VP9 remains a royalty-free alternative to proprietary codecs.

AV1: The Next-Generation Open Codec

AV1 (AOMedia Video 1) is the newest significant codec in the WebRTC ecosystem, developed by the Alliance for Open Media (including Google, Mozilla, Microsoft, Amazon, and others).

Key Characteristics of AV1

1. Exceptional Compression Efficiency: AV1 offers approximately 30% better compression than VP9 and 50% better than H.264.

2. Royalty-Free: Designed specifically to avoid patent licensing issues, AV1 is fully open and royalty-free.

3. Limited Browser Support: Major browsers have varying levels of support:

   • Chrome: Software encode since M90; hardware encode limited to newer Intel Arc/Snapdragon chips
   • Firefox/Edge: Primarily decode support
   • Safari: No AV1 support yet
   • Apple platforms: M3 chips support hardware decode (since macOS Ventura 14.4), but no encoding yet

4. Significant Computational Cost: AV1 encoding is 3-5× more CPU-intensive than VP9 on desktop CPUs, making it challenging for real-time applications on most devices without hardware acceleration.

5. Screen Sharing Optimization: AV1 includes special tools for screen content, making it excellent for screen sharing applications.

6. Low-Latency Design: AV1's RTP payload format (draft-ietf-avtcore-rtp-av1) and Annex E tools enable low-latency communication, though the computational cost remains a significant trade-off.

Implementation Note

AV1 support in WebRTC is still evolving. Check for current browser compatibility before relying on it exclusively:

            // Check if AV1 is supported
function isAV1Supported() {
  const capabilities = RTCRtpSender.getCapabilities('video');
  return capabilities.codecs.some(codec => 
    codec.mimeType.toLowerCase() === 'video/av1');
}

// Prioritize AV1 if available
if (isAV1Supported()) {
  const av1Codecs = RTCRtpSender.getCapabilities('video').codecs
    .filter(codec => codec.mimeType.toLowerCase() === 'video/av1');
  videoTransceiver.setCodecPreferences(av1Codecs);
}
        

SVC (Scalable Video Coding) in WebRTC

Modern WebRTC implementations are increasingly adopting Scalable Video Coding (SVC), which allows a single encoded bitstream to provide multiple resolution and framerate variants simultaneously.

Benefits of SVC in WebRTC

1. Adaptation Without Renegotiation: SVC allows receivers to adapt to network changes without requiring the sender to reencode or renegotiate.

2. Simulcast Alternative: While traditional simulcast requires encoding the same content multiple times at different qualities, SVC achieves similar results more efficiently.

3. Bandwidth Optimization: SVC is particularly valuable in group calls where participants have varying connection capabilities.

4. Reduced Latency: By eliminating the need for encoder reconfiguration during adaptation, SVC helps maintain low latency.

SVC Implementation in Different Codecs

• VP8: Supports temporal scalability natively, but requires setting the scalabilityMode field in RTCRtpEncodingParameters (e.g., "L1T3") to enable it in Chrome and Firefox
• VP9: Supports temporal scalability natively; Chrome adds spatial scalability with 2-3 layers when configured with appropriate scalability modes (e.g., "L2T3")
• H.264: H.264-SVC extensions exist theoretically but are not implemented in any mainstream browsers
• AV1: Includes native SVC support with all three dimensions (spatial, temporal, and quality scalability)

Example of enabling SVC:

            // Enable temporal scalability for VP8
const sender = pc.addTransceiver('video').sender;
const params = sender.getParameters();
params.encodings[0].scalabilityMode = 'L1T3'; // 1 spatial layer, 3 temporal layers
await sender.setParameters(params);
        

Codec Negotiation in WebRTC

WebRTC uses the Session Description Protocol (SDP) to negotiate which codecs will be used in a communication session. Understanding this process is crucial for ensuring optimal codec selection.

Codec Negotiation Process

1. Offer Creation: The initiating peer creates an offer containing its supported codecs and their parameters.

2. Offer Handling: The receiving peer processes the offer, comparing the offered codecs with its own capabilities.

3. Answer Generation: The receiving peer creates an answer containing the intersection of supported codecs.

4. Codec Selection: The communication uses the mutually supported codec with the highest priority.

            // Sample code to force a specific codec order in the offer
const transceiver = pc.addTransceiver('video');
const capabilities = RTCRtpSender.getCapabilities('video');

// Clone the codecs array before sorting to avoid mutating the cached array
// Some browsers cache this array, so sorting in place can cause issues
const preferred = capabilities.codecs.slice().sort((a, b) => {
  const codecA = a.mimeType.toLowerCase();
  const codecB = b.mimeType.toLowerCase();

  if (codecA.includes('vp9')) return -1;
  if (codecB.includes('vp9')) return 1;
  if (codecA.includes('h264')) return -1;
  if (codecB.includes('h264')) return 1;
  if (codecA.includes('vp8')) return -1;
  if (codecB.includes('vp8')) return 1;

  return 0;
});

transceiver.setCodecPreferences(preferred);
        

Codec Negotiation Diagram

Choosing the Right Codec: Decision Framework

Selecting the appropriate codec requires balancing multiple factors. Here's a comprehensive decision framework to guide your codec selection:

1. Network Considerations

Bandwidth Constraints:

• Limited Bandwidth: Prioritize newer, more efficient codecs (AV1, VP9) for video; Opus with lower bitrate settings for audio
• Variable Bandwidth: Select codecs with strong adaptive bitrate capabilities (Opus for audio, VP9 or AV1 with SVC for video)
• Mobile Networks: Consider both bandwidth efficiency and power consumption; hardware-accelerated H.264 often provides the best balance

Network Stability:

• High Packet Loss: Choose codecs with strong error resilience (Opus with Forward Error Correction for audio)
• Highly Variable Conditions: Implement SVC or simulcast approaches for video

2. Compatibility and Platform Support

Browser and Device Support:

• Broad Compatibility: H.264 for video, Opus for audio
• Modern Browsers Only: VP9 or AV1 can be considered for better efficiency
• Mobile Devices: Prioritize hardware-accelerated codecs to reduce power consumption

Legacy System Integration:

• PSTN/Telephony Integration: Include G.711 support
• Enterprise Video Systems: Ensure H.264 compatibility

A Compatibility Matrix

*Safari/iOS: VP9 decode-only, experimental encode; Chrome/Firefox/Edge: AV1 primarily decode support, limited hardware encode; Opus: Mobile hardware typically accelerates decoding only, encoding remains CPU-based

3. Quality and Use Case Requirements

Use Case Optimization:

• Video Conferencing: Balance quality and bandwidth; H.264/VP8 for compatibility, VP9/AV1 for efficiency
• Live Streaming: Prioritize quality and bandwidth efficiency; consider VP9 or AV1
• Screen Sharing: VP9 or AV1 with screen content tools
• Voice-Only Communication: Opus at appropriate bitrates (20-40 kbps)
• Music or High-Fidelity Audio: Opus at higher bitrates (64-128 kbps)

Specific Quality Requirements:

• Resolution Requirements: Higher resolutions benefit more from advanced codecs
• Framerate Needs: Applications requiring high framerates (gaming, sports) benefit from temporal scalability

4. Processing Power and Battery Considerations

• Low-Power Devices: Prioritize hardware-accelerated codecs (typically H.264)
• High-Performance Environments: Can consider more computationally intensive options (AV1)
• Mobile Optimization: Balance quality and power consumption

5. Licensing and Cost Factors

• Open Source Projects: Consider royalty-free options (VP8, VP9, AV1)
• Commercial Applications: Evaluate potential licensing implications of H.264/H.265
  • H.264 Licensing Exception: MPEG-LA offers royalty waivers for browser-to-browser WebRTC calls with fewer than 100,000 concurrent users (under MPEG-LA AVC/H.264 License 5.10), which benefits many small to medium applications
• Browser-Based Applications: Browser vendors typically handle codec licensing for standard use cases

Practical Implementation Strategies

Adaptive Codec Selection

Rather than selecting a single codec, consider implementing an adaptive approach:

            function selectOptimalCodecs(connection, deviceCapabilities, networkQuality) {
  const videoTransceiver = connection.addTransceiver('video');
  const audioTransceiver = connection.addTransceiver('audio');

  // Get available codecs
  const videoCodecs = RTCRtpSender.getCapabilities('video').codecs;
  const audioCodecs = RTCRtpSender.getCapabilities('audio').codecs;

  // Determine optimal video codec based on network and device
  let optimalVideoCodecs;
  if (networkQuality === 'poor' && deviceCapabilities === 'low') {
    // Prioritize H.264 for hardware acceleration on low-end devices
    optimalVideoCodecs = videoCodecs.filter(c => c.mimeType.toLowerCase().includes('h264'));
  } else if (networkQuality === 'poor' && deviceCapabilities === 'high') {
    // For poor network but capable device, use VP9
    optimalVideoCodecs = videoCodecs.filter(c => c.mimeType.toLowerCase().includes('vp9'));
  } else if (networkQuality === 'good' && isAV1Supported() && deviceCapabilities === 'high') {
    // Use AV1 only for good conditions AND high-end devices due to CPU cost
    optimalVideoCodecs = videoCodecs.filter(c => c.mimeType.toLowerCase().includes('av1'));
  } else {
    // Default to VP9 for good balance
    optimalVideoCodecs = videoCodecs.filter(c => c.mimeType.toLowerCase().includes('vp9'));
  }

  // Fall back to VP8 if preferred codec isn't available
  if (optimalVideoCodecs.length === 0) {
    optimalVideoCodecs = videoCodecs.filter(c => c.mimeType.toLowerCase().includes('vp8'));
  }

  // For audio, Opus is almost always the best choice
  const opusCodecs = audioCodecs.filter(c => c.mimeType.toLowerCase().includes('opus'));

  // Apply selections
  if (optimalVideoCodecs.length > 0) {
    videoTransceiver.setCodecPreferences(optimalVideoCodecs);
  }

  if (opusCodecs.length > 0) {
    audioTransceiver.setCodecPreferences(opusCodecs);
  }

  // Note: On Safari, you may need to additionally use SDP munging 
  // to ensure codec preferences are respected
}
        

Monitoring and Adaptation

Implement real-time monitoring and adaptation to optimize codec parameters:

            // Example of monitoring and adapting audio codec parameters
function monitorAndAdaptAudioCodec(rtcPeerConnection, audioSender) {
  // Monitor network conditions
  setInterval(async () => {
    const stats = await rtcPeerConnection.getStats(audioSender);
    let packetLossRate = 0;
    let availableBitrate = 0;

    // Calculate packet loss using remote-inbound-rtp stats
    // (outbound-rtp doesn't know about lost packets)
    stats.forEach(report => {
      if (report.type === 'remote-inbound-rtp') {
        const packetsLost = report.packetsLost || 0;
        const packetsReceived = report.packetsReceived || 1;
        packetLossRate = packetsLost / (packetsReceived + packetsLost);
      }

      if (report.type === 'candidate-pair' && report.state === 'succeeded') {
        availableBitrate = report.availableOutgoingBitrate;
      }
    });

    // Adapt Opus parameters based on conditions
    const parameters = audioSender.getParameters();

    // Find Opus codec in parameters
    const opusEncodingIdx = parameters.encodings.findIndex(e => 
      parameters.codecs.find(c => 
        c.payloadType === e.codecPayloadType && 
        c.mimeType.toLowerCase() === 'audio/opus'
      )
    );

    if (opusEncodingIdx >= 0) {
      // Adjust FEC based on packet loss
      if (packetLossRate > 0.05) {
        // Enable FEC for packet loss > 5%
        parameters.encodings[opusEncodingIdx].fec = { ssrc: parameters.encodings[opusEncodingIdx].ssrc };
      }

      // Adjust bitrate based on available bandwidth
      if (availableBitrate > 0) {
        // Leave headroom for other traffic
        const targetBitrate = Math.min(128000, availableBitrate * 0.7);
        parameters.encodings[opusEncodingIdx].maxBitrate = targetBitrate;
      }

      // Note: We're using standard RTCRtpEncodingParameters rather than
      // non-standard properties like 'networkPriority' which are
      // Chromium-only and behind flags

      // Apply the changes
      audioSender.setParameters(parameters);
    }
  }, 2000); // Check every 2 seconds
}
        

Conclusion

The selection of audio and video codecs in WebRTC applications is a critical decision that significantly impacts quality, efficiency, and compatibility. As we've explored in this lesson, each codec offers distinct advantages and trade-offs, making the optimal choice highly dependent on your specific use case, target audience, and technical constraints.

Key takeaways from this lesson include:

1. Balanced Approach: There is rarely a one-size-fits-all solution for codec selection. The best approach typically involves understanding your requirements and choosing codecs that provide the optimal balance of efficiency, quality, and compatibility.

2. Future-Proofing: The codec landscape continues to evolve, with newer options like AV1 promising significant improvements. Designing your WebRTC application with the flexibility to adopt these advancements ensures long-term viability.

3. Adaptive Strategies: Rather than static codec selection, consider implementing adaptive strategies that can optimize the communication experience based on real-time conditions.

4. Testing is Essential: Theory can guide codec selection, but real-world testing with your actual user base and network conditions is irreplaceable for validating your codec choices.

By applying the knowledge gained in this lesson, you'll be better equipped to make informed decisions about codec selection and implementation in your WebRTC applications, ultimately delivering superior real-time communication experiences to your users.

Additional Resources

1. Mozilla Developer Network: WebRTC Audio Codecs
2. Mozilla Developer Network: WebRTC Video Codecs
3. WebRTC Samples: Codec Constraints
4. Opus Codec Official Documentation
5. AV1 Codec Alliance Website
6. WebRTC Audio and Video Codec Requirements RFC