Moderate any content

Stream's moderation APIs can be used to moderate custom content such as user profiles, avatars, and other content types. This guide shows you how to use Stream's moderation APIs to moderate custom content.

The overall process of moderating custom content is as follows:

1. Create a moderation policy
2. Submit content for moderation
3. Process the recommended action
4. Monitor results in the Dashboard
5. Receive notifications of dashboard actions

flowchart TD
    A[Create Moderation Policy] -->|Configure policy settings| B[Submit Content]
    B -->|Use Check API| C[Receive Recommendation]
    C -->|Analyze result| D{Take Action}
    D -->|Flag| E[Review Queue]
    D -->|Remove| F[Content removed]
    D -->|Shadow| G[Shown only to the author]
    E --> H[View in Dashboard]
    F --> H
    G --> H
    H -->|Actions taken| I[Webhook Notifications]

    style A fill:#e1f3ff
    style B fill:#e1f3ff
    style C fill:#e1f3ff
    style D fill:#ffe1e1
    style E fill:#f0e1ff
    style F fill:#f0e1ff
    style G fill:#f0e1ff
    style H fill:#e1ffe4
    style I fill:#fff3e1

Let's go through each step in detail.

Create a moderation policy

A moderation policy is a set of rules that define how content should be moderated in your application. See Configuration for details on setting up moderation policies.

Submit custom content to the moderation Check API

You can use the moderation Check API to submit custom content to the moderation engine and receive a recommended action.

response = client.moderation.check(
    entity_type="entity_type",
    entity_id="entity_id",
    entity_creator_id="entity_creator_id",
    moderation_payload=ModerationPayload(
        texts=["this is bullshit", "f*ck you as***le"],
        images=["example.com/test.jpg"],
    ),
    config_key="custom:my-content-platform-1",
)

In this example:

• entity_type is the type of the entity you are moderating. You can provide any generic identifier for the entity type e.g., user:profile, user:avatar, user:cover_image, etc.
• entity_id is the unique identifier for the entity you are moderating.
• entity_creator_id is the unique identifier for the creator of the entity you are moderating.
• texts is the array of texts you want to moderate.
• images is the array of images you want to moderate.
• custom:my-content-platform-1 is the unique identifier for the moderation policy you created earlier.

Query Flagged Content

You can use the moderation QueryReviewQueue API to query flagged content. This is useful if you want to review the flagged content in your app.

response = client.moderation.query_review_queue(
    filter={"entity_type": "entity_type"},
    sort=[{"field": "created_at", "direction": -1}],
)

Monitoring Moderated Content

You can monitor flagged or removed content from the dashboard. There is one content queue. Filter it by entity type when you want users, text, or media.

The available actions are:

• Mark Reviewed: This action indicates that you have reviewed the content and determined it doesn't require further action. It helps keep track of which contents have been addressed by moderators.
• Delete: We provide Delete action button out-of-the-box, but actual logic to delete the content needs to be handled by your backend via webhook.

Webhook Integration

You can receive webhook notifications for actions taken on the content from the dashboard. Please refer to the Webhook Integration guide for more details. E.g., if moderator presses delete button for specific content on the dashboard, you will receive a webhook notification with the action details. And accordingly you can delete the content from your platform.