Stream's Livestreaming product now supports 4K at 60fps. Plenty of providers support 4K, but very few, if any, offer 4K at 60fps over WebRTC. We wanted to share some details about the choices that made this possible.
What Is a Livestream and Why Is 4K60 So Difficult?
Unlike your usual video call, where all members have their cameras and microphones enabled, a livestream usually only has one main feed (with many different qualities) from the creator and all others receiving this feed.
Since most people prefer building their livestream on external platforms (such as OBS) instead of simply using their camera, you also need to build an ingress mechanism to allow that.
Additionally, you can have an egress (like RTMP) which allows you to forward the same stream to other livestreaming platforms like YouTube or Twitch. The last part is often HLS, which allows you to create a separate livestream over HTTP instead of connecting through WebRTC. HLS, however, gives up the latency advantage you get on WebRTC.
Even though it might seem like video calls probably hold more people and more streams, these are actually somewhat easy to manage. For one, people usually sit fairly static and their windows on the screen are quite small, so the resolution can be reduced and the bitrate can be adapted without affecting the overall call quality.
Livestreams, on the other hand, usually involve a lot of activity on the screen and every user subscribes to the full resolution (as it's likely the only thing on their screen). Most livestreams often go out at 1080p (very few at 4K) and 30fps (some at 60).
Now here's the interesting part: 4K isn't 2x the pixels of 1080p. It's 4x, and it needs roughly 3x the bitrate (bitrate needed does not scale linearly). And 60fps needs another 1.8x to 2x over 30fps, which is less than double because frames that are closer together in time are more similar to each other, so each individual frame is cheaper to describe. Put those together, and a 4K 60fps livestream takes roughly 5x more bitrate than a 1080p 30fps livestream. So a team that has built 1080p 30fps livestreaming now needs to scale their infrastructure demands to five times the original to allow this to happen. Those are pre-optimisation figures, what comes straight out of the encoder before any tuning, and bringing them down is most of what the rest of this post is about.
Choosing the Right Codec
Codecs help you reduce the bitrate required for sending video to more plausible levels. A video codec's job is to avoid sending the same thing twice. It sends a full frame occasionally (a keyframe), then for every frame after that, it sends only what changed.
Choosing the right codec is essential to having a decent livestreaming product because you need to make sure of two things that often contradict each other: the first is that everyone should be able to view the livestream, and the second is that it should need the least bitrate possible.
For our usual video calls, we use the VP9 codec. While VP9 isn't exactly state-of-the-art, this is where we felt comfortable drawing the line between compatibility and bitrate.
However, for ingress-based (e.g. with OBS) livestreaming, we needed to maximise compatibility and make sure hardware decoding was an option. Hence, we went with the H.264 codec to achieve this. H.264 has a fixed-function decoder in essentially every device that has ever shipped a screen. If you send a stream a device can't decode in hardware, you fall back to software decode (which drains the battery and drops frames) and can severely degrade the experience.
The main drawback of this was that we would need upwards of 40 Mbps of bitrate for every device subscribed to 4K 60fps since modern codecs are way better in terms of bitrate, but H.264 is still heavy on the network end.
Using VMAF for Judging Compression
One way you can close a gap like that is VMAF, a metric Netflix open-sourced that predicts how a compressed video will score with human viewers by comparing each frame against the original and combining several quality measures into one number from 0 to 100.
We can use VMAF to judge how much we can compress without the stream losing significant quality. We picked a target of VMAF 85 and treated it as the constraint, then went looking for the lowest bitrate that still hit it. That got us to 28 Mbps for 4K 60fps, which is where we ship it today.
The shape of that curve is why we target a score instead of a bitrate. On the steep left side, every bit you add is a bit the viewer can see. Later, it flattens, so you're paying for encoder output that nobody's eyes will resolve. It also means the answer moves with the content, which is what we wanted, because static content clears 85 at a fraction of the bitrate a concert needs, and a fixed bitrate would have to be set for the concert and left there.
One thing worth being clear about is that none of this happens live. VMAF needs the original to compare against, and it is far too slow to run on a 4K 60fps stream in flight, so we run it beforehand on representative footage to work out the settings, and those settings are what go into production.
Not Everyone Gets 4K 60fps
28 Mbps is not a number we send to every viewer. Every stream goes out as a ladder of bitrates, and each viewer sits on the highest rung their connection and their hardware can actually hold. If their bandwidth drops, they move down a rung. This adaptive streaming ensures we don't unnecessarily send viewers what they don't need, or something they simply can't consume because they don't have the bandwidth or the screen resolution necessary.
How Many Viewers Can Watch at 4K 60fps?
We've benchmarked our 4K solution to 10,000 concurrent viewers. Most times, only a small fraction of users will actually be streaming on 4K60 due to both device or data constraints. In terms of infrastructure for Stream, egress gets used up first, and a viewer on the top rung costs several times what a viewer on the benchmark configuration costs, so the ceiling comes down roughly in proportion.
We overprovisioned all the way to benchmark levels before the first real 4K 60fps stream went out, and it turned out to need noticeably less than we had built for, since we tested for everyone using 4K and most real-world streams simply don't have that many people doing so.
Future Improvements
In the future, we still want to upgrade the livestreams to AV1, which cuts down on the bitrate significantly. Two things are in the way though, and only one of them is about viewers' devices.
The first is encoding. At 4K 60fps, AV1 encoding on our ingress path is software only. Software AV1 at 4K 60fps is fine for a one-off. It just isn't something we can turn on by default for every stream that shows up.
The second is decoding, and this is where the backup stream comes in. Most modern devices handle AV1 fine, meaning iPhone 15 and up, Galaxy S23 and up, etc., but most is not all. There are also devices that decode AV1 perfectly well and never announce that they can, which from our side is the same thing as not supporting it. So the plan is dual encoding, AV1 with an H.264 backup stream, which is already on our public roadmap.
For now, 4K 60fps is available on request rather than generally available. A stream at this quality is worth setting up with us properly rather than just switching on, so if you have something you want to put out at 4K 60, contact sales and we'll get you running. We're curious to see what people do with it.