Inside Stream AI Moderation for Live Video & Audio

TL;DR

• Stream AI Moderation runs on a copy of the call. It samples camera frames and live captions and scores them with the same models as chat.
• It acts right after a violation is detected, since WebRTC is too fast to block a frame first.
• Rules escalate in steps. Repeated violations lead to a warning, then a mute or blur, then a kick. Flagged calls go to the same review queue as chat.
• Other video setups work, too. Send frames and captions to the Check API, and apply mute and kick yourself.

The challenge: Live video does not give you a queue. A frame that leaves the camera on WebRTC is on someone else's screen in a few hundred milliseconds. You cannot hold the segment, score it, and then decide whether to publish it. By the time a model returns, the moment has already aired.

Stream's live video and audio moderation is designed around that fact. We do not try to beat glass-to-glass latency with a blocking classifier. We fork a cheap copy of the live call, sample it, score the samples against the same policy engine as chat, and act on the participant (mute, blur, warn, kick, or end the call) while a human can still review what happened.

This piece covers the architecture of that path: live video keyframes, live audio captions, uploaded files, and the shared policy they all use.

Block or Correct?

When you moderate a livestream, you can either block content before it airs or correct it afterward. That choice is real, but your streaming protocol mostly makes it for you, before you ever pick a model.

• LL-HLS / DASH give you seconds of player delay. A check that finishes inside that window can keep a segment off the CDN. Nothing aired.
• WebRTC gives you a few hundred milliseconds. Interactive calls and co-hosted livestreams run here. The blocking window is effectively gone. For example, a check that returns in 300 ms is still scoring a frame the audience already saw.

Stream Video is a WebRTC product. Built-in call moderation is therefore a correcting path with live enforcement instead of a pre-publish holdback. What viewers have already seen or heard can't be recalled, so the system focuses on stopping what comes next. It can take the participant off camera and keep a record for moderators to review.

How Moderation Runs Alongside the Call

At ingest, the media path to viewers is left alone. Moderation never sits in the SFU forwarding loop. A second path peels off two things:

1. Video keyframes: still JPEGs from each participant's camera, captured by the same frame recorder Stream already exposes for calls.
2. Audio as text: live closed captions from the call's speech-to-text, scored as speech, not as a raw waveform.

participants
    │
    ├─► WebRTC / SFU  ──────────────────────────► viewers
    │
    └─► moderation copy
            ├─ frame recorder  → image models
            └─ live captions   → text models
                    │
                    ▼
              call policy
                    │
                    ├─► mute / blur / warn / kick / end
                    └─► review queue (same dashboard as chat)

Video owns capture and enforcement. Moderation owns models, rules, and the review queue. The call keeps going while those two talk.

If you run your own media stack, the same decision layer is available. You send frames and captions in, you get the same rule evaluation and the same queue. You handle mute and kick yourself.

Live Video: Score Frames

A live call is 15-30 frames per second per camera. Scoring every frame would be expensive and mostly redundant. Common harms on camera (nudity, weapons, hate symbols) persist across many frames. A one-frame flash is usually noise.

So we sample.

The frame recorder takes a still from each participant on an interval you set: 2 to 60 seconds, typically a few seconds, default 720p. For example, on a four-person call at a 5-second interval that is four images every five seconds.

Each still is scored with the same image categories as an uploaded photo. This includes explicit content, nudity, violence, weapons, hate symbols, and the rest of the image policy. There is no separate "video network" on the live path. A live keyframe is an image that happens to have a call and a participant attached to it.

This is what keeps live video on the same policy surface as chat images and uploaded clips. You tune a category once, and then the live call, the chat attachment, and the file upload all see it.

Additionally, a single dirty frame is not enough to kick someone. Call rules count consecutive matching keyframes. Three frames in a row above your confidence threshold is a different signal from one ambiguous still. A clean frame resets the count. That is how you avoid acting on a blurry thumbnail or a split-second reflection.

The sampling interval sets how much coverage a call gets. A high-risk livestream can sample as often as every 2 seconds, while a large webinar with a controlled stage can sample less often. The same models score every still, whatever the interval.

Live Audio: Moderate Speech

Platform rules are written about what people say, such as threats, slurs, and solicitation. A classifier that only looks at the raw audio signal can't tell what was said, so it doesn't map well to those rules.

Built-in audio moderation uses the call's live captions. As someone speaks, caption segments arrive as text. Each segment is scored with the same text engines you already use for chat (the fast NLP path or an LLM with your own harm definitions).

The same consecutive-hit rule applies. Two caption segments in a row that match hate speech is a trigger. A clean segment resets the count. You are not punishing someone for a single mistranscribed word.

This is why live audio and live video can share a rule. A call rule is just "if the camera trips or the speech trips, escalate." Chat text, live speech, and image frames are different inputs into one decision.

Now, this path does not wait for the call recording to finish, and it does not run a separate audio-event model on the live mix. After-the-fact recordings and full transcripts still exist for investigation, but they are not how the live call is policed.

How Call Rules Escalate

A chat message can be blocked before it posts. A live participant cannot. So, the useful response is a sequence.

A typical call rule looks like this:

You can also end the whole call, send a warning only, or take no in-call action and only flag the session for review.

A cooldown sits on the rule so the same condition does not fire every sampled frame. Violation counts reset when the session ends. If you need to deal with a user who is bad across many calls, that is a user rule on a rolling window. (For example, ban after five call-rule hits in 24 hours.)

This is easy to get wrong if you bolt a generic image API onto WebRTC, since the model is not the product. The product is "three strikes, then they are gone, and a moderator can see the frames."

Same Queue as Chat

Flagged calls land in the same Moderation review queue as flagged messages and images. The reviewer sees the participant, the frames or caption that tripped the rule, and the action already taken.

That is deliberate. Trust and safety teams do not want a video console, a chat console, and a third tool for livestreams. Chat, Feeds, and Video already share users and policy, so the queue should, too.

Moderators can still kick or end a call from the dashboard after the fact. Automatic actions respond while a violation is still happening, and moderators handle satire, appeals, and cases the model is unsure about.

Live Calls vs. Uploaded Files

Live is not the only video Stream moderates. Chat and Feeds also carry file attachments.

Uploaded video is accepted immediately, then sampled in the background. Frames are taken across the file (about one per second, thinned on long videos so the whole file is still covered), scored with the same image categories, and the file takes the strictest frame. A five-minute clip uses the same model as a short one, with frames sampled across the whole file.

Uploaded audio is transcribed, then the transcript is scored as text. It uses the same text policy as chat and live captions. The attachment is available while that runs, and the verdict catches up.

The deadline changes, but the labels do not.

If the Call Is Not on Stream Video

The built-in path assumes Stream is already capturing frames and captions. If you run your own SFU or livestream pipeline, you send those samples to the Check API as an external call with images for keyframes and text for captions.

Rules, consecutive counts, the review queue, and the trigger events are the same. Mute and kick are not. Stream cannot mute a room it does not host. Your backend applies the action when the rule fires.

Integration comes down to sending those samples to the Check API. Stream doesn't need your video stream, only the frames and caption text.

What You Configure

You configure three things, and they stay out of the request path:

1. How often to look: frame interval and resolution on the call type. Tighter interval, more coverage, more work per participant.
2. What counts as a hit: harm labels and minimum confidence on keyframes; harm labels or your own LLM definitions on captions; how many consecutive hits before the rule fires.
3. What happens next: the action sequence, the cooldown, and whether a user rule should ban repeat offenders across calls.

With frame recording set to auto-on for a call type and a call rule in place, the live path is active. Stream Video captures the frames and captions, the models score them, and any action the rule calls for runs during the call. The evidence stays in the review queue.

The Job

Stream's live moderation works within the limits of WebRTC instead of trying to get around them.

It samples the call at low cost, scores those samples with the same models used across the rest of the platform, and uses its time to act on the participant. The first frame has already reached viewers by then, so the goal is to stop what comes next.