Moderation for Chat

Moderation is essential for a good user experience on chat. There are also requirements from DSA and the app stores to take into account.

Stream has advanced AI moderation capabilities for text, images, video and audio. Before we launched moderation, customers often struggled with the cost and difficulty of integrating external moderation APIs. You can now setup moderation in minutes at an affordable price point.

There are 4 layers of moderation:

• Limits / chat features: Restrict what's allowed. Moderator permissions, disabling links or images, slow mode, enforce_unique_usernames, slash commands etc.
• Simple: Blocklist, regex, domain allow/block, email allow/block
• User actions: Flag, mute, ban etc.
• AI moderation: AI on text, images, video, audio

Let's go over each of these and show what's supported.

Limits & Chat features

Disabling the permission to post links or add attachments

You can control links and attachments by revoking the relevant permissions for a role. The permissions to manage are:

• add-links - ability to post messages containing URLs
• create-attachment - ability to add attachments to messages
• upload-attachment - ability to upload files/images

Update the grants for a channel type to remove these permissions from a role:

// Remove link and attachment permissions for channel_member role
await client.updateChannelType("messaging", {
  grants: {
    channel_member: [
      "read-channel",
      "create-message",
      "update-message-owner",
      "delete-message-owner",
      // "add-links" - removed to disable links
      // "create-attachment" - removed to disable attachments
      // "upload-attachment" - removed to disable uploads
    ],
  },
});

For more details on permissions, see User Permissions.

Image & Video file types

You can restrict which file types users can upload using image_upload_config and file_upload_config. This allows you to set allowed or blocked file extensions and MIME types, as well as size limits.

Both configs accept the following fields:

// Restrict images to common formats only
await client.updateAppSettings({
  image_upload_config: {
    allowed_file_extensions: [".jpg", ".jpeg", ".png", ".gif", ".webp"],
    allowed_mime_types: ["image/jpeg", "image/png", "image/gif", "image/webp"],
    size_limit: 5 * 1024 * 1024, // 5MB
  },
  // Restrict file uploads to documents only
  file_upload_config: {
    allowed_file_extensions: [".pdf", ".doc", ".docx"],
    allowed_mime_types: ["application/pdf", "application/msword"],
    size_limit: 10 * 1024 * 1024, // 10MB
  },
});

For more details, see App Settings.

Giving moderators more permissions

Moderators have elevated permissions like the ability to ban users, delete messages, and more. You can assign moderator roles to users at the channel level or across all channels.

Add a Moderator to a Channel:

// Add a member with moderator role
await channel.addMembers([
  { user_id: "james_bond", channel_role: "channel_moderator" },
]);

Make a User a Moderator Across All Channels:

To grant a user moderator permissions across all channels in your app, update their global role:

await client.partialUpdateUser({
  id: "james_bond",
  set: { role: "admin" },
});

For more details on permissions, see User Permissions.

Slow mode

Slow mode helps reduce noise on a channel by limiting users to a maximum of 1 message per cooldown interval (1-120 seconds). Moderators, admins, and server-side API calls are not restricted.

// enable slow mode and set cooldown to 1s
await channel.enableSlowMode(1);

// increase cooldown to 30s
await channel.enableSlowMode(30);

// disable slow mode
await channel.disableSlowMode();

For more details, see Slow Mode & Throttling.

Enforce unique usernames

This setting prevents users from using duplicate usernames. When enabled with app, it enforces uniqueness across the entire app. With team, it only enforces within the same team.

// Enable uniqueness constraints on App level
await client.updateAppSettings({
  enforce_unique_usernames: "app",
});

// Enable uniqueness constraints on Team level
await client.updateAppSettings({
  enforce_unique_usernames: "team",
});

Slash commands for banning

Stream Chat supports built-in slash commands like /ban and /unban for quick moderation actions. Enable these commands on your channel type:

// Enable ban/unban commands for a channel type
await client.updateChannelType("messaging", {
  commands: ["giphy", "ban", "unban", "mute", "unmute", "flag"],
});

Simple Moderation features

Blocklist

A blocklist is a list of words used to block or flag chat messages.

For setup and API usage, see Word, Regex, Email, and Domain Filters.

Regex

Regex filters allow you to match and moderate messages using regular expressions. This is useful for filtering patterns like phone numbers, URLs, or complex word variations. Configure regex filters via the Stream dashboard under 'Blocklist & Regex Filters'.

For detailed configuration, see Word, Regex, Email, and Domain Filters.

Email/domain allow or block

You can configure domain and email filters to control what URLs and email addresses can be shared in messages. Set up allowlists or blocklists for specific domains via the Stream dashboard.

For detailed configuration, see Word, Regex, Email, and Domain Filters.

User Actions

Flag

Any user can flag a message or user. Flagged content is added to your moderation review queue on the Stream Dashboard.

// Flag a message
const flag = await client.flagMessage(messageId);

// Flag with a reason and custom data
const flag = await client.flagMessage(messageId, {
  reason: "spammy_user",
  custom: {
    user_comment: "This user is spamming.",
  },
});

Reasons & custom data

You can enhance flags by associating them with a specific reason and custom data. It is advisable to utilise a slug or keyword as a designated reason for easy translation or other forms of display customisation.

The custom data can encompass any object, offering supplementary metadata to the flag.

The Query Message Flags endpoint retrieves both reasons and custom data, and the reason can also be utilised for filtering these flags.

// flag with a reason
let flag = await client.flagMessage(messageID, {
  reason: "spammy_user",
});

// flag with a reason and additional custom data
flag = await client.flagMessage(messageID, {
  reason: "spammy_user",
  custom: {
    user_comment: "This user is spamming the homepage.",
    page: "homepage",
  },
});

// flag with only custom data
flag = await client.flagMessage(messageID, {
  custom: {
    page: "homepage",
  },
});

Query Flagged Messages

If you prefer to build your own in-app moderation dashboard, rather than use the Stream dashboard, you can query flagged messages using the QueryReviewQueue API endpoint.

Both server-authenticated and user-authenticated clients can use this method. For client-side requests, the user needs moderator or admin permissions.

const response = await client.moderation.queryReviewQueue(
  { entity_type: "stream:chat:v1:message" },
  [{ field: "created_at", direction: -1 }],
  { next: null },
);

for (const item of response.items) {
  console.log(item.message.id);
  console.log(item.message.text);
  console.log(item.message.type);
  console.log(item.message.created_at);
}

console.log(next); // <-- next cursor for pagination

Please refer to the Moderation API documentation for more details.

Mute

Users can mute other users. Mutes are stored at the user level and returned when connectUser is called. Messages from muted users are still delivered via websocket but not via push notifications.

See Mute in the Moderation API for full documentation and SDK examples.

Block

The user block feature allows users to control their 1-on-1 interactions within the chat application by blocking other users.

await client.blockUser("user-to-block");

How blocking impacts chat

When a user is blocked, several changes occur:

• Direct Communication Termination: When a user blocks another user, communication in all 1-on-1 channels are hidden for the blocking user.
• Adding to Channels: If a blocked user tries to add the blocking user to a channel as a member, the action is ignored. The channel will not include the blocking user but will have the remaining members.
• Push Notifications: The blocking user will not receive push notifications from blocked users for 1-on-1 channels.
• Channel Events: The blocking user will not receive any events from blocked users in 1-on-1 channels (e.g., message.new).
• Group Channels: Group channels are unaffected by the block. Both the blocking and blocked users can participate, receive push notifications, and events in group channels.
• Query Channels: When hidden channels are requested, 1-on-1 channels with blocked users will be returned with a blocked:true flag and all the messages.
• Active Chats and Unread Counts: Blocked users will not appear in the blocking user's list of active chats. Messages from blocked users will not contribute to unread counts.
• Unblocking Users: After unblocking, all previous messages in 1-on-1 channels become visible again, including those sent during the block period.
• Hidden Channels: Channels with only the blocked and blocking users are marked as hidden for the blocking user by default. If a blocked user sends a message in a hidden channel, the channel remains hidden for the blocking user.
• Group Channel Messages: Messages from blocked users will still appear when retrieving messages from a group channel.
• WebSocket Connection: When connecting to the WebSocket, the blocking user receives a list of users they have blocked (user.blocked_users). This is only available for the blocking user's own account.
• Message Actions: Actions such as sending, updating, reacting to, and deleting messages will still work in blocked channels. However, since the channels are hidden, these actions will not be visible to the blocking user.

Block User

Any user is allowed to block another user. Blocked users are stored at the user level and returned with the rest of the user information when connectUser is called. A user will be blocked until the user is unblocked.

await client.blockUser("user-to-block");

Unblock user

await client.unBlockUser(blockedUser);

List of Blocked Users

const resp = await client.getBlockedUsers();

Server Side

Block User:

await client.BlockUsersAsync(new BlockUsersRequest
{
    BlockedUserID = user2.Id,
    UserID = user1.Id,
});

Unblock user:

await client.UnblockUsersAsync(new UnblockUsersRequest
{
    BlockedUserID = user2.Id,
    UserID = user1.Id,
});

Get List of Blocked Users:

await client.GetBlockedUsersAsync(user1.ID);

AI moderation

AI moderation can detect over 40 harms in 50+ different languages. In addition to these classification models LLM based moderation is also supported. Moderation APIs are available at additional costs. It's priced to be cost-effective and typically is a fraction of the cost of other moderation APIs.

Read the full AI moderation docs.

Ban

Users can be banned from an app entirely or from a channel. When banned, they cannot post messages until the ban is removed or expires. You can also apply IP bans and optionally delete the user's messages.

See Ban in the Moderation API for full documentation, including shadow bans, query endpoints, and SDK examples.