Image Moderation

Stream's image moderation engine detects harmful or inappropriate images. By default it runs asynchronously: images are processed in the background so they don't add latency, and actions are applied as soon as the analysis is complete. To get the verdict in the same request instead, moderate images synchronously.

Supported Categories

These are the categories shown in the dashboard image policy editor. Set an action and a confidence threshold per category, and narrow a category to some of its subcategories with subclassifications.

The subcategories depend on the image classifier enabled for your app, so your app uses one of the two lists below. The dashboard always shows the list that applies to your app.

Subcategories (list A)

Subcategories (list B)

Images have no severity levels: each category is matched by its confidence score. queryModerationRules returns your app's subcategories as ai_image_subclassifications.

To moderate text inside images, such as a phone number in a screenshot, add ocr_rules. Extracted text is classified with the AI Text labels.

Supported Formats

Image moderation accepts the following formats:

• JPEG (.jpg, .jpeg)
• PNG (.png)
• GIF (.gif)*. For animated GIFs, only the first frame is moderated.
• WebP (.webp)*
• SVG (.svg)*
• HEIC/HEIF (.heic, .heif)**

* Converted to JPEG or PNG for moderation only. The stored file is not changed.

** Converted to JPEG when uploaded to Stream's CDN, so the stored file is a JPEG and is moderated as one. HEIC/HEIF images hosted outside Stream's CDN are not moderated.

The following MIME types are accepted:

• image/jpeg, image/jpg
• image/png
• image/gif
• image/webp
• image/svg+xml

The format is detected from the file contents, falling back to the Content-Type header. Images in other formats are not moderated. This includes BMP, which can be uploaded to Stream's CDN but is not converted, as well as TIFF and AVIF. To keep unsupported uploads out entirely in Stream Chat, configure the app's image upload allowlist or blocklist from the dashboard. This is a Stream Chat upload setting, not part of the Moderation API; for standalone moderation, filter file types in your own upload flow.

Confidence Scores

For each category, set a confidence threshold between 1 and 100. An action triggers only when the engine's confidence for that category is at or above the threshold.

• A higher threshold (80-90) only matches highly confident detections. It catches less and produces fewer false positives.
• A lower threshold (40-50) catches more, including borderline images, and produces more false positives.

For strict enforcement, such as explicit content on a platform for teens, lower the threshold so more is detected. For categories where false positives hurt more than misses, raise it. Start with the default (50) and adjust from what you see in the review queue.

Configuration

Set up image moderation with ai_image_config on a policy. min_confidence is required on every image rule. The scale is 0-100 and the default is 50. A higher value flags less: 90 only matches highly confident detections.

await client.moderation.upsertConfig({
  key: "chat:messaging",
  ai_image_config: {
    rules: [
      { label: "Explicit", action: "remove", min_confidence: 70 },
      { label: "Violence", action: "flag", min_confidence: 50 },
    ],
    ocr_rules: [{ label: "PII", action: "flag" }],
  },
});

Image rule actions are flag, shadow, remove, bounce, bounce_flag, and bounce_remove. subclassifications narrows a label to its L2 children. queryModerationRules returns ai_image_label_definitions and ai_image_subclassifications for the exact strings.

Send images

Send image URLs in moderation_payload.images (up to 30) with the Check API. Stream Chat and Feeds send attached images for you.

const response = await client.moderation.check({
  entity_type: "listing",
  entity_id: "listing-123",
  entity_creator_id: "user-42",
  config_key: "marketplace",
  moderation_payload: {
    images: ["https://cdn.example.com/listings/123/photo-1.jpg"],
  },
});

To send image bytes instead of URLs, use Analyze.

Images on your own CDN

Image moderation works with any CDN, as long as Stream's servers can download the image with a plain GET. Signed URLs work if they are still valid when Stream fetches them. Private S3 objects use a stream+s3:// URL. client.moderation.checkS3Access() checks that the configured IAM role can read them; pass s3_url to test one object.

To moderate a large set of existing images, client.moderation.bulkImageModeration({ csv_file }) takes a CSV of image URLs and returns a task_id. Completion arrives as export.bulk_image_moderation.success.

How It Works

By default, image moderation is asynchronous:

1. The image is accepted immediately to maintain low latency.
2. Analysis begins in the background, and the Check response has status: "pending".
3. The engine evaluates the image against all configured categories.
4. If a confidence threshold is met, the configured action is applied. Stream Chat and Feeds update the message or post.
5. Flagged images appear in the review queue, and a review_queue_item.new webhook is sent.

Synchronous moderation

To hold an image until it's checked, turn on Process images synchronously in the dashboard image policy, or set async: false on ai_image_config. Check then returns the image verdict in its response, and Stream Chat applies the action before the message is delivered.

await client.moderation.upsertConfig({
  key: "marketplace",
  ai_image_config: {
    async: false,
    rules: [{ label: "Explicit", action: "remove", min_confidence: 50 }],
  },
});

Synchronous moderation adds the analysis time to the request. An image that was already moderated on upload reuses that verdict.