How Do I Integrate AI Content Moderation Into a VOD Pipeline Using a Video API?

TL;DR

• Moderation runs as an asynchronous step between upload and publication, and the video stays private until a decision comes back.
• The checks cover everything a viewer can see or hear: frames, speech, on-screen text, and the title, description, captions, and thumbnail.
• A verified webhook starts each check, and every job is tagged with the content version so a late result for an old file can't approve a new one.
• Your backend maps each recommendation to approved, needs review, or rejected, and it signs playback tokens only after encoding is done and every required check is approved.

Video APIs are great, but relatively simple, pieces of technology. They take a user's upload, encode it with H.264, store it somewhere like S3, and deliver it to viewers via HLS and a CDN. They do that for any video file they're given. They have nothing to do with the video's actual content.

But what if that video contains nudity, graphic violence, or hate speech? Nothing in that pipeline asks whether the video breaks your community guidelines, so if playback opens as soon as encoding finishes, your viewers are the first people to see a video with who-knows-what in it.

This post shows how to add AI content moderation to a VOD pipeline. Moderation runs as an asynchronous step between upload and publication, and the video stays private until a decision comes back. Your video API still handles ingest, encoding, and delivery. The moderation service scores the frames, speech, and text. Your backend publishes the video only after it records an approval.

Where Does Moderation Fit in a VOD Pipeline?

Content moderation sits between when the upload is stored and when your backend starts granting playback access. The video API doesn't need to know. Your backend creates a record for each upload, starts the checks, and is the only part of the system that can make a video watchable.

The uploaded video becomes watchable only at the last step, when your backend creates an approved decision for the current version and starts signing playback tokens.

Encoding status and moderation status are kept in separate fields, because they finish at different times and fail for different reasons. The content version number increments each time the uploader replaces the file, so a late result for an old file can't approve the new one.

{
  "video_id": "video_123",
  "asset_id": "mux_asset_456",
  "playback_id": "mux_signed_playback_789",
  "content_version": 2,
  "encoding_status": "processing",
  "moderation_status": "pending",
  "required_checks": ["video_and_metadata", "transcript"],
  "completed_checks": [],
  "reviewed_version": null
}

The required_checks array lists every check this upload needs, and completed_checks fills in as results arrive. The reviewed_version field records which content version the decision applies to. Your playback endpoint reads this record, and nothing else, when it decides whether to sign a token.

Publishing only after approval is proactive moderation. With reactive moderation, the video goes live first and gets reviewed after user reports or a later scan, so everyone who watched before the takedown has already seen it. Reactive moderation gets uploads live sooner. Holding the video fits better when user-generated content comes from accounts you don't know yet.

The Stream video moderation engine accepts a video immediately and analyzes it in the background. Our video moderation docs say the verdict arrives "a few seconds to a few minutes later, depending on length." In chat and feeds, the message or post is updated when analysis finishes. In a VOD pipeline, your backend keeps the video private until then.

Which Parts of a Video Need to Be Checked?

Everything a viewer can see or hear needs checking, including frames, speech, on-screen text, and the title, description, captions, and thumbnail the uploader supplied. Each needs a different kind of analysis, and content moderation that only checks frames will miss a spoken threat or an abusive title.

Frame analysis is computer vision. The Stream engine samples one frame per second by default, scores each frame with the same categories and confidence model we use for image moderation, and gives the video the strictest verdict of any frame. For long videos, we thin the sample rate so the whole file is still covered.

The sampling rate decides what the model gets a chance to see:

• At one frame per second, a two-second shot appears in about two sampled frames.
• At one frame every five seconds, the same shot can fall entirely between two samples, and the model never sees it.

You can make sampling denser in the dashboard video policy, but that means more work per video. Video moderation best practices suggest sampling more densely only when short spikes of policy-breaking content matter.

Speech and text are natural language processing problems, and there are several ways to check them:

• Keyword filters and regex blocklists catch known slurs, but they have no contextual understanding, so they miss a threat phrased politely and flag a word used innocently.
• The Stream NLP engine rates toxic categories by severity.
• The Stream LLM text engine takes rules written in plain language plus a description of your platform, and large language models judge each sentence in context.

LLMs take longer per check, but the video is already waiting on frame analysis, so the extra time doesn't change when it can publish.

Generative AI has made deepfakes and other AI-generated content easy to produce. If your content policies cover GenAI uploads or misinformation in the footage, route reports about them to human moderators. If the misinformation is spoken or written, you can describe it as a custom rule for the LLM engine.

How Do You Send a Video From Your Video API to a Moderation Service?

Start the check from a verified webhook, give the moderation service a URL it can fetch while viewers still can't, and tag every job with the content version it's checking. With Mux as the video API, you'll need to:

• Create the asset with a signed playback policy, the highest static rendition, and auto-generated captions, so you have an MP4 file and a transcript to hand over later.
• Receive Mux webhooks, verify the mux-signature header, store the event, and respond right away. Mux treats a delivery attempt as failed after 5 seconds and keeps retrying for 24 hours.
• Wait for video.asset.static_rendition.ready before you submit the video. Static renditions take longer to build than the HLS version, so video.asset.ready doesn't mean the MP4 exists yet.
• Sign a playback token for the MP4 URL with an expiry that covers analysis plus a retry or two (e.g., an hour), and include that URL in the check.
• When video.asset.track.ready fires for the generated captions track, fetch the plain-text transcript with its own signed token and submit it as a second check.

Mux documents that duplicate webhook messages can arrive even after your server responds with a 2xx, so record each event ID and skip any ID you've already processed. Do the same with ours. Every retry we send has the same X-Webhook-Id header, and our webhooks guide says to deduplicate on that ID.

The check is a single call through our Node SDK. The entity_id includes the content version, so the result maps back to the exact file that was checked. The config_key names the content moderation policy you created with upsertConfig, with the video, image, and text engines turned on.

const videoUrl = "https://stream.mux.com/" + video.playback_id + "/highest.mp4?token=" + muxToken;

const response = await client.moderation.check({
  entity_type: "vod_video",
  entity_id: video.video_id + ":v" + video.content_version,
  entity_creator_id: video.uploader_id,
  moderation_payload: {
    texts: [video.title, video.description],
    images: [thumbnailUrl],
    videos: [videoUrl],
  },
  config_key: "vod_uploads",
});

await db.saveModerationTask(video.video_id, video.content_version, response.task_id, response.status);

The check response comes back with status set to partial while video analysis is still running, along with a task_id and a review queue item. The recommended_action in that first response only reflects the engines that have already finished. Store the task ID on the record and leave moderation_status at pending until the completion webhook arrives.

How Do You Turn Moderation Results Into a Publish Decision?

Map each recommendation to a state on your record:

• A keep becomes approved
• A flag becomes needs review
• A remove becomes rejected.

We send moderation_check.completed once the check for an entity has finished, after both the synchronous and asynchronous engines, and we send it even when nothing was flagged. The payload includes entity_id, entity_type, final recommended_action, and review_queue_item_id. Your handler verifies the signature, drops repeat deliveries, queues the work, and returns 200.

app.post("/webhooks/stream", express.raw({ type: "application/json" }), async (req, res) => {
  let event;
  try {
    event = client.verifyAndParseWebhook(req.body, req.headers["x-signature"]);
  } catch (err) {
    return res.status(401).end();
  }

  const isNew = await db.recordWebhookId(req.headers["x-webhook-id"]);
  if (isNew && event.type === "moderation_check.completed") {
    await jobs.enqueue("apply-moderation-result", {
      entityId: event.entity_id,
      entityType: event.entity_type,
      action: event.recommended_action,
      reviewItemId: event.review_queue_item_id,
    });
  }

  res.status(200).end();
});

The route uses express.raw because verifyAndParseWebhook hashes the raw request body. The queued job reads the version from the entity_id, ignores the result if the record has moved on to a newer version, and otherwise marks that check complete.

Each video category has a confidence threshold from 1 to 100, and higher values produce fewer false positives. Start with the defaults and raise the threshold on any category that over-flags.

A confidence score says how sure the model is that a label applies. It doesn't say how harmful the content is, so a surgery tutorial and a gore clip can both score high for visually disturbing imagery. Flagged videos go to human review in our dashboard's review queue. When a moderator acts on one, we send review_queue_item.updated with the action in the payload, and your backend turns that into approved or rejected.

Before it signs a playback token, your backend ensures encoding is done, and every required check is approved for the current version.

// Application pseudocode: run before signing any playback token
function isPublishable(video) {
  const allChecksDone = video.required_checks.every(function (check) {
    return video.completed_checks.includes(check);
  });
  return (
    video.encoding_status === "ready" &&
    allChecksDone &&
    video.moderation_status === "approved" &&
    video.reviewed_version === video.content_version
  );
}

An empty or partial result never counts as approval. If a check errors or times out, the video stays private while the job retries, and repeated failures go to manual review. Apply the same rule to thumbnails, storyboards, and transcript URLs. With a signed playback ID, Mux serves each of those only with a valid token, so your backend controls them the same way it controls playback.

What Does Stream Handle in This Pipeline, and What Stays in Your Backend?

Stream runs content moderation analysis, policies, and review tools. Your backend keeps the record and the decision to publish. Stream returns a recommendation and doesn't change anything on your video API, so holding a video back and releasing it is code you write.

The audio moderation engine transcribes audio and checks the transcript against the same harm categories as our text engines, with its own configuration. Audio files must be no larger than 25 MB; larger files are skipped without moderation. At 128 kbps, a common AAC bitrate, 25 MB holds about 26 minutes of audio. At 64 kbps, it holds about 52 minutes.

If your VOD library is made of recordings from calls or livestreams on our Video API, we're also the source of the files. Our Video API covers calls, livestreams, and their recordings, and it sends call.recording_ready when a recording is processed and call.transcription_ready when a transcription is ready.

How Do You Know Moderation Is Working After Launch?

Moderation is working when you can see how long uploads wait, how often the model is wrong in each direction, and how much harmful content gets past it. You'll want to track:

• Time from upload to publish, split between videos approved on the first pass and videos that went to review.
• The age of the oldest item in the review queue.
• How often moderators overturn a flag, which gives you a false positive rate per category.
• Violations found after approval, from user reports and audits.
• Checks that errored or timed out.

Our Logs & Analytics API returns every moderation action, who took it, and when, and you can export logs for compliance. When an uploader disputes a decision, our Appeals API lets them submit an appeal and lets moderators accept or reject it. After you change a policy, Evaluations re-runs a saved batch of up to 1,000 items so you can see what improved or drifted.

You can set up the policy, engines, and review queue for this pipeline with our AI content moderation API, and our Moderation docs include a Node getting-started guide that takes you from creating a policy to your first check call.