Architecting Real-Time Moderation for Live Video

TL;DR

• Live moderation splits into two paths: a fast, shallow check that blocks content before it ships, and a slower, deeper check that corrects the decision after the fact.
• Your delivery protocol sets the size of that window before any model runs. WebRTC's sub-second delivery leaves almost no time to block; LL-HLS's few seconds of delay leaves room to work with.
• Keep the blocking path inside its window by scoring fewer frames, compressing the model, and running it at the edge; give the correcting path the time and context (multimodal fusion, human review) the blocking path doesn't have.
• Track precision, recall, and F1 by category instead of one accuracy number, and keep an immutable decision log.

A creator goes live on your platform. It's a shopping stream with thousands of viewers and chat running fast.

Twenty minutes in, something crosses a line. A counterfeit item on camera... a slur in the chat... a claim the policy doesn't allow.

Viewers have already watched it happen by the time anyone notices, and it's your platform's rules on the line, not just the creator's.

That's what makes live stream moderation different from every other content type. There's no queue to hold a violation in, and no way to recall what's already gone out.

This piece covers the architecture for it: a fast, shallow path that beats your delivery window and blocks content outright; a slower, deeper path that corrects the call after the fact; and the protocols, models, and queues that make both work.

What You Must Decide: Whether To Block Now or Correct After

The core idea is to sort every check you run by its deadline rather than by what it looks at.

That means you're not building one moderation pipeline for chat, another for video, and another for audio. Instead, you ask a timeline question about every check: how long do I have before a viewer sees this, and can the check answer inside that time?

The time you have is the gap between content reaching your servers and a player showing it, and your protocols and product decisions set this number: a few seconds of segment delay on LL-HLS, or just a few hundred milliseconds on WebRTC, plus whatever small hold you can add without viewers feeling it.

• A check that beats the window can block the content. Its verdict arrives while the content is still only on your side, so acting on it means the chat message never appears or the video segment never gets published. Nothing was seen, so there's nothing to undo.
• A check that misses the window has to act after viewers have watched the content. This means you're correcting the decision, not the violation. Whatever was seen was seen, and no correction resets someone's eyeballs.

So you effectively end up with two streams of moderation, one fast and shallow, the other slow and deep. The same live video gets both treatments at once: a small image model scores sampled frames quickly enough to hold a segment back, while a temporal model that needs a few seconds of clip runs behind it and flags more subtle problems.

Chat works the same way, with a lightweight classifier gating each message and a slower context check reconsidering it.

So the design work splits into three pieces:

1. The window, which your protocols fix before any classifier runs.
2. The blocking path, everything that has to beat the window.
3. The correcting path, everything allowed to miss it as long as it can still act.

The window comes first because you inherit it rather than choose it.

The Window Your Protocols Set

Protocols are what define livestreams, and they matter to moderation in different ways:

• Delivery protocols, server-to-player, set the window itself. This is where the time in the timeline question comes from.
• Ingest protocols, encoder-to-server, decide what condition the stream arrives in. They don't change the window, but a stream that stalls on the way in leaves nothing for either path to work with.

Most of the latency spread comes down to TCP versus UDP. RTMP runs over TCP, so one lost packet holds up everything queued behind it until retransmission. That's head-of-line blocking, and on a congested stream it turns into stalls and jitter. The UDP protocols skip the missing packet and move on, since a frame that shows up late isn't worth waiting for in a live stream.

Here's a breakdown of low-latency streaming and the different tradeoffs:

On the ingest side, RTMP is still the default because nearly every encoder speaks it. On the delivery side, the choice decides how much room the blocking path gets:

• LL-HLS: seconds of delay, cheap scale, real working room. Segments ride the same CDN caches as static content, so big passive audiences stay affordable, and players run a few seconds behind your encoder.
• WebRTC: sub-second delivery, per-viewer cost, almost no window. Every viewer holds a session with a selective forwarding unit (SFU), and a frame is on screens before most checks return.

In practice, most platforms mix them: WebRTC for the hosts and guests on stage, and LL-HLS via a CDN for everyone watching.

But we can see immediately how these decisions matter for moderation. With LL-HLS, a violation caught now stops the next segment from ever being published, or ends the stream before any player requests it, holding the violation's airtime to seconds.

But if you deliver entirely through WebRTC, your options are limited, as blocking mostly means acting on the frames that follow. Sub-second delivery is a real feature; you just pay for part of it out of the moderation budget.

One Stream In, Two Paths Out

At ingest, you fork the stream. One copy goes straight to transcode and delivery. The other feeds moderation, and nothing that happens to it can slow the first copy down.

On the moderation copy, the two paths map onto straightforward engineering:

• The blocking path runs synchronously, inline with the content. A verdict comes back before the message posts or the segment ships.
• The correcting path runs asynchronously, behind a queue. Verdicts come back as webhooks, on their own time.

What Runs Inline

A semantic filter scoring each chat message before it posts, a small image model scoring sampled frames before the next segment ships. There's deliberately little infrastructure here, because every extra hop spends window time.

The inline check doesn't have to live on your servers either. You can put it on the device: a light model screens every frame locally, and only suspicious 2-6 second clips, keyframes, or compact embeddings get uploaded for a deeper look. The blocking path sits next to the content, and the correcting path picks up at the upload.

What Runs Behind the Queue

Ingestion publishes frames and chat events to a bus like Kafka, rules and heavy models consume them from a stateful processor like Flink, and decisions flow back to enforcement on their own schedule. A model that takes ten seconds, or a review backlog forty items deep, delays a correction and touches nothing else.

The consumers can be a whole pipeline of their own. In AWS's generative-AI moderation pattern, a segment landing in S3 starts a Step Functions state machine that fans out: Transcribe on the audio, Rekognition on sampled frames for labels and moderation flags, a Claude model for frame summaries, and Titan embeddings. Everything collects in OpenSearch, where a Bedrock LLM reads the metadata against the policy prompt.

The worry with acting after the fact is how long after. The Amazon IVS variant gives real numbers: under 500 ms for the Rekognition call on frames below 1 MB, about two seconds from frame to notification through the rules engine, and the stream stops automatically when a rule fires. A correction that lands in two seconds means the violation aired for two seconds, not the rest of the hour.

In short: cheap checks sit next to the content, expensive checks sit behind a queue, and enforcement flows back into delivery.

The Blocking Path: Beating the Window

Everything on this path answers the same timeline question: Can this check return a verdict before the window closes?

The work splits into four pieces, in order:

1. Find where the time actually goes.
2. Score fewer frames.
3. Make the model fit the budget.
4. Move the whole thing next to the stream.

Find Where the Time Actually Goes

A software profiler measures the model, not the pipeline. It can report a 5 ms model while the full pipeline runs 50 ms behind. The number to watch is glass-to-glass: how long from a photon hitting the camera sensor to a processed result coming out the other end.

Much of that time is spent before inference runs. Sensor exposure takes 1-33 ms depending on light, a rolling shutter reads rows out one at a time, and an encoder with B-frames enabled buffers several frames before it can emit anything. By the time the blocking path sees a frame, part of the budget is gone.

Instrument every stage and fix the biggest contributors first. They're often decode, transport, and buffering rather than the model.

Score Fewer Frames

Most violations persist across many consecutive frames; a nudity or weapon detection doesn't appear and vanish inside 33 ms. AWS samples at two frames per second and describes that as typical.

On top of sampling, drop the frames that carry nothing new:

• Embedding similarity. AWS's generative-AI pattern embeds each sampled frame, runs a cosine similarity check against the previous one, and discards any frame too close to its predecessor.
• Image hashing. Amazon IVS skips a frame if its hash is near the previous frame's for the same channel.

Sampling plus deduplication can cut inference volume by an order of magnitude. The saving applies to both paths, since the correcting path's heavy models see the same reduced stream.

Make the Model Fit the Budget

Once the model is the bottleneck, the goal is fewer operations per frame. Three compression techniques:

• INT8 quantization maps weights and activations from 32-bit floats to 8-bit integers. The model shrinks roughly 4x, and Tensor Cores run the integer math 2-4x faster than FP32. It costs some accuracy, so teams train with quantization awareness and keep the sensitive first and last layers in FP16.
• Structured pruning removes whole filters, channels, or attention heads. Zeroing individual weights produces sparse matrices that standard GPUs can't accelerate; removing entire filters preserves the dense matrix operations the hardware runs fast.
• Temporal shift modules give a 2D model temporal reasoning without 3D convolutions. They shift a fraction of the feature-map channels along the time axis before a normal 2D convolution, so the model reasons about motion at close to 2D speed.

Past the model, several pipeline settings reduce buffering:

When a downstream stage falls behind, a leaky queue drops the oldest frames instead of building a backlog. A verdict on a frame from two seconds ago arrives too late to act on. SRT makes the same tradeoff at the transport layer when it drops stale packets.

Run It Next to the Stream

Pure cloud makes sense when the model is too big to quantize for edge hardware and the application can tolerate more than 300 ms of round-trip latency. Running at the edge keeps raw frames on the device, which helps privacy, and ships events or short clips instead of full video, which cuts egress cost.

In practice, almost every production system lands on the same hybrid: a quantized detector at the edge screens everything, and ambiguous frames escalate to a heavier model in the cloud. That's the blocking and correcting split expressed in hardware: the edge model does the blocking, and the cloud model does the correcting.

Cold starts are the operational detail to plan for. TensorRT engine builds and model loads can take 30 to 300 seconds, so pre-warm with a synthetic inference at startup and pin model files to NVMe. Otherwise, the first stream after a deploy goes unmoderated.

The Correcting Path: Getting the Call Right

Everything here has already missed the window, so a verdict can take seconds or minutes instead of milliseconds. That time goes to bigger models, more context, and human judgment.

The work splits into two pieces:

1. Score the modalities together, with context the inline checks don't have.
2. Route the judgment calls to humans, without putting one in the request path.

Score the Modalities Together

Blocking-path checks look at one track at a time, because that's what fits inside the window. Each classifier answers a narrow question: what's visible in this frame, what was said in the last few seconds of audio, what's in this chat message.

Per-track checks miss harm that only exists in the combination. A chat message that's innocuous on its own but aimed at the person on screen passes the text check, because the text check can't see the screen. A pitch that calls an item authentic passes the audio check, footage of the item passes the frame check, but together they count as a counterfeit sale. Each classifier answered its own question correctly, and the violation went through.

Scoring the tracks jointly closes that gap, and multimodal ML has three standard ways to combine them:

• Early fusion concatenates feature vectors straight after the encoders. It catches low-level cross-modal signals but needs expensive joint training.
• Late fusion combines each modality's final scores with learned weights. Modular and easy to maintain, but the interaction between modalities is lost.
• Hybrid fusion does both, running cross-modal attention over raw features while correlating the intermediate scores. It captures the most context and is the most work to build and run.

The joint score still needs context from outside the content. The same phrase can be fine in one channel and a violation in another. This is context collapse. A semantic embedding check runs in 10-50 ms, but the final call needs user history, channel type, and conversation flow. Aggregating those signals server-side cuts false positives and makes the system harder to game.

Route the Judgment Calls to Humans

Satire, political speech, and harassment phrased politely end up in front of a person. Review time is unpredictable, so a human inside a synchronous pipeline backs the queue up behind them. Redis's write-up on oversight patterns lays out the three ways to wire people in:

• Human-in-the-loop gates on explicit approval. The latency is unbounded, so it can't sit on a live path.
• Human-on-the-loop lets the system act in real time while people monitor with veto power. Live moderation runs this way, and the cost is a short exposure window before an override lands.
• Human-out-of-the-loop runs autonomously inside policy bounds, with people involved only at design and evaluation time.

Escalation routes on confidence. The same write-up pairs a trust score for model agreement and classification confidence with a risk score that flags severity regardless of confidence. High trust triggers automated action. Low trust or moderate risk goes to review.

You can't predict when a reviewer will get to an item, so the system saves the whole event before it escalates: the flagged clip, the transcript, the model scores, and the conversation around it. The reviewer opens the case with everything already in front of them.

Redis Streams works as that queue. Escalations sit as persistent logs you can query later, and a sorted set ranks them by urgency, scored on viewer count, creator tier, and severity. Whatever's most urgent gets seen first.

When the reviewer makes the call, the item flips from pending to approved or rejected. That status change triggers enforcement asynchronously, and the decision becomes a labeled example for active learning, so the models learn the edge cases and send fewer of them to review over time.

Stream Moderation is built around the same loop. Flagged content lands in a review queue sorted by severity; each severity level can have its own action, and webhooks fire whether the flag came from a model, a blocklist, or a person. The Check API takes text, images, video, or a user profile, so chat and video share one queue and one enforcement layer.

Measuring Both Paths

Plain accuracy will flatter the system because moderation data is heavily imbalanced. Almost everything is fine, so a model can post a high accuracy number while missing most of the abuse.

Moderation metrics are what to track instead:

Split all three by category, because a system strong on spam can be weak on bullying, and by language and region, because models are best in the languages they were trained on. Operationally, track system latency, throughput at peak, review time per item, reviewer agreement rate, and escalation rate. A rising rate of escalation usually indicates ambiguous policy rather than a weak model.

The Audit Trail Is a Legal Requirement Now

Under the EU's Digital Services Act, a live platform owes users a notice-and-action mechanism, prompt removal of illegal content once notified, and a statement of reasons for every removal. Annual transparency reports have to describe the automated systems in use and disclose their accuracy and error rates.

The penalties are serious money. The Commission can impose a fine of up to 6% of global annual turnover for serious infringements, plus periodic penalties of up to 5% of average daily worldwide turnover for each day of delay in compliance. Enforcement started in December 2025 with the first DSA non-compliance fine, €120 million against X, for transparency failures rather than slow removal, and well below the cap, but the framework is clearly live.

The useful part is that compliance mostly asks for things this architecture already produces: a sub-second automated path for clearly illegal content, which is the blocking path, and an immutable log of every automated and human decision. The event log the correcting path keeps for review queues and active learning is the same artifact a regulator asks for.

The Whole System, Job by Job

Everything in this article comes down to three jobs: pick your window, make sure the inline checks beat it, and give the deeper checks a way to act once they're done.

• Pick your window deliberately. Ingest over SRT so a creator's bad hotel wifi doesn't stall the stream, deliver LL-HLS through a CDN to viewers, and save WebRTC for the hosts and guests who actually need it. Every bit of delivery latency you cut is time your inline checks lose, so sub-second delivery is a cost you should choose on purpose.
• Make sure the inline checks beat it. Sample around two frames per second, throw away near-duplicate frames with embedding similarity or hashing, and run a quantized model at the edge with zero-copy buffers and leaky queues. Warm it up before traffic hits, otherwise the first stream after a deploy goes out unmoderated.
• Give the deep checks time and a way back in. Put them behind Kafka so nothing slow touches delivery, run the fused multimodal scoring and LLM policy checks on their own schedule, and act through webhooks. Anything the models aren't sure about goes to a review queue in Redis Streams, ranked by urgency, and every human decision goes back into training.
• Watch both sides and write everything down. Track precision and recall by category and language instead of one headline accuracy number, keep an eye on review time and escalation rate, and log every decision somewhere immutable. That log is also what the DSA asks you to produce.

Where Stream Fits

The whole design keeps coming back to the same question: Does this check beat the window or miss it?

Beat it, and you can block. Miss it, and you're correcting the decision, so the check needs a way to act on the stream, the replay, and the account.

Stream covers a lot of these pieces on shared infrastructure: chat, feeds, and video with AI moderation built in, a Check API that takes text, images, video, or user profiles, review queues ranked by severity with webhooks, and the open-source Vision Agents SDK for running models at the edge. It doesn't make the tradeoffs go away, but you end up tuning a pipeline instead of building one.

If you want to try it, the moderation docs cover setup for chat and video. Wiring the Check API into chat is the smallest version of the blocking path, and a sensible place to start before the video pipeline.