Chat Moderation

Integrating moderation in Stream Chat is a straightforward process that requires no code changes. You simply need to configure moderation policies through the dashboard, and Stream will automatically handle content moderation for your chat application. This guide will walk you through setting up moderation policies and monitoring flagged content through the Stream dashboard.

Configure Moderation Policy

A moderation policy is a set of rules that define how content should be moderated in your application. We have a detailed guide on What Is Moderation Policy.

Please also follow the following guides on creating a moderation policy for chat:

Auto Moderation on Message

Client side integration

If you are using any of our Chat SDKs, you can immediately start seeing the moderation in action. Think of it as having a highly efficient moderation team working 24/7 behind the scenes—except this one runs on algorithms instead of coffee! ☕️

Server side integration

If you are using the SendMessage API from server side, then by default messages will be skipped for moderation. You can enable moderation for messages by setting the force_moderation option to true.

const { message } = await channel.sendMessage(
  {
    text,
    attachments,
    user_id: sentBy,
  },
  {
    force_moderation: true,
  },
);

Its important to note that if the message is blocked, SendMessage API will NOT return an error. But instead the message object will have a type of error. The example response is as following:

{
  "message": {
    "id": "2b56e00f-4149-465f-b1e7-62d892788f85",
    "type": "error",
    "text": "Message was blocked by moderation policies",
    "moderation": { "action": "remove", "original_text": "F*** you motherf***er" }
    "user": {
      "id": "little-lake-2",
      "name": "little-lake-2",
      ...
    },
    "cid": "messaging:first",
    "created_at": "2024-11-21T13:13:38.071535Z",
    "updated_at": "2024-11-21T13:13:38.071535Z",
    "shadowed": false,
    ...
  },
  "duration": "136.06ms"
}

When a message is blocked by moderation, no WebSocket events (message.new) will be triggered for that message. This means other users in the channel will not receive any notification or see the blocked message in their chat interface. The message is effectively prevented from being delivered to any recipients.

Monitoring Moderated Content

This concludes the setup for moderation. You can try sending a message in your chat application to see how the moderation policy works in real-time. Remember to monitor and adjust your policies as needed to maintain a safe and positive environment for your users. You can monitor all the flagged or blocked content from dashboard. You have access to three separate queues on the dashboard.

Users Queue

This queue contains all the users who were flagged by another user or users who have at least one flagged content. You as a moderator can take certain actions on user entirely or the content posted by that user. All the list of actions will be available when you hover over a user in the list. The available actions on user are as following:

Mark Reviewed: This action indicates that you have reviewed the user’s profile or content and determined that no further action is needed at this time. It helps keep track of which users have been assessed by moderators, ensuring efficient management of the queue. This will also mark all the content from this user as reviewed.
Permanently Ban User: This action permanently restricts the user from accessing or participating in the platform. It’s typically used for severe or repeated violations of community guidelines. When a user is permanently banned, they are unable to log in, post content, or interact with other users. This action should be used judiciously, as it’s a final measure for handling problematic users.
Temporarily Ban User: This action temporarily restricts a user’s access to the platform for a specified period of time. It’s often used as a warning or corrective measure for less severe violations. During the ban period, the user cannot log in or interact with the platform. This allows them time to reflect on their behavior while giving moderators a chance to review the situation before deciding on further action.
Delete the User
Delete all the content from the user

Text Queue

This queue contains all the text contents that have been flagged or blocked by the moderation system. As a moderator, you can review these contents and take appropriate actions. The available actions for each content in the Text Queue are:

Mark Reviewed: This action indicates that you have reviewed the content and determined it doesn’t require further action. It helps keep track of which contents have been addressed by moderators.
Delete: This action removes the message entirely from the platform.
Unblock Message: If a message was automatically blocked by the moderation system, but upon review you determine it’s actually acceptable, you can use this action to unblock it. This allows the message to be visible and accessible in the chat.

These actions provide moderators with the flexibility to handle different situations appropriately, ensuring a fair and safe environment for all users.

Media Queue

The Media Queue is specifically designed for handling images, videos, and other media content that has been flagged or blocked by the moderation system. This queue allows moderators to review visual content that may violate community guidelines or pose potential risks. Similar to the Text Queue, moderators can take various actions on the items in this queue, such as marking them as reviewed, deleting inappropriate content, or unblocking media that was mistakenly flagged.

User-Driven Actions

In addition to automated moderation, Stream’s Chat SDK provides methods for users to take moderation actions against other users. These actions help maintain community standards by allowing users to report inappropriate behavior or content. For detailed information on effects of these actions, please check the documentation for Chat Moderation Tools. Here we cover some basic examples on how to use these actions.

Flag User

Users can flag another user with a reason. This will trigger a review in the moderation dashboard.

If you are using Chat UI SDK, users can flag the user from user actions menu.

If you have your own custom UI, you can use the following API to flag the user:

await client.moderation.flagUser("user_id", "reason", {
  user_id: "user_id", // User ID who is flagging the user, required only for server-side integration only.
});

Flag Message

Users can flag a message, which will trigger a review in the moderation dashboard.

If you are using Chat UI SDK, users can flag the message from message actions menu.

If you have your own custom UI, you can use the following API to flag the message:

await client.moderation.flagMessage("message_id", "reason", {
  user_id: "user_id", // User ID who is flagging the message, required only for server-side integration only.
});

Ban User

A user with moderator permissions can ban a user from the chat. For more details, please check the documentation for Banning Users.

await client.banUser("target_user_id", {
  banned_by_id: "user_id", // User ID who is banning the user,
});

Unban User

A user with moderator permissions can unban a user from the chat.

await client.unbanUser("target_user_id", {
  unbanned_by_id: "user_id", // User ID who is unbanning the user,
});

Block User

A user can choose to block another user, which will prevent the blocked user from sending any message to the user. Please check the documentation for Blocking Users for more details.

await client.blockUser("blocked_user_id", "blocking_user_id");

Unblock User

A user can choose to unblock another user, which will allow the blocked user to send messages to the user.

await client.unblockUser("blocked_user_id", "blocking_user_id");

Chat Regulation Tools

Besides the moderation tools, Stream Chat also offers a set of regulation tools that can help you comply with regulations and protect your users’ privacy.

Slow Mode

For live events or concerts, you can sometimes have so many users that the sheer volume of messages overloads the browser or mobile device. This can cause the UI to freeze, high CPU usage, and degraded user experience. Stream offers 3 features to help with this. Slow mode helps reduce noise on a channel by limiting users to a maximum of 1 message per cooldown interval.

The cooldown interval is configurable and can be anything between 1 and 120 seconds. For instance, if you enable slow mode and set the cooldown interval to 30 seconds a user will be able to post at most 1 message every 30 seconds.

Please check the documentation for Slow Mode for more details.

Pre-Send Webhook

Sometimes you want to have more control over what users are allowed to post and either discard or edit their messages when they do not meet your content guidelines.

This webhook gets called before the message reaches the API and therefore before it is saved to the channel and observable by other users. This allows you to intercept a message and make sure that inappropriate content will never be displayed to other users.

You can use this webhook to enforce any of these rules:

Scrub PII from messages (ie. social security numbers, credit cards, etc.)
Restrict contact information sharing (ie. discard phone numbers, emails)
Validate messages do not include complex words that are best matched using a regular expression (regex)

Please check the documentation for Pre-Send Webhook for more details.

How Auto-Moderation Works

Feeds Moderation