Stream Chat

Integrating moderation in Stream Chat is a straightforward process that requires no code changes. You simply need to configure moderation policies through the dashboard, and Stream will automatically handle content moderation for your chat application. This guide will walk you through setting up moderation policies and monitoring flagged content through the Stream dashboard.

Configure Moderation Policy

A moderation policy is a set of rules that define how content should be moderated in your application. See Configuration for details on setting up moderation policies.

• Step 2: Set Up Policy for Text Moderation
• Step 4: Test Moderation Policy

Auto Moderation on Message

Client side integration

If you are using any of our Chat SDKs, you can immediately start seeing the moderation in action. Think of it as having a highly efficient moderation team working 24/7 behind the scenes—except this one runs on algorithms instead of coffee! ☕️

Server side integration

If you are using the SendMessage API from server side, then by default messages will be skipped for moderation. You can enable moderation for messages by setting the force_moderation option to true.

const { message } = await channel.sendMessage(
  {
    text,
    attachments,
    user_id: sentBy,
  },
  {
    force_moderation: true,
  },
);

message = channel.send_message(
    {"text": text, "attachments": attachments, "user_id": sent_by},
    force_moderation=True,
)

msg := &getstream.MessageRequest{
    Text:       text,
    UserID:     getstream.PtrTo(sentBy),
}
response, err := client.Chat().SendMessage(ctx, channelType, channelID, getstream.SendMessageRequest{
    Message:         *msg,
    ForceModeration: getstream.PtrTo(true),
})

client.chat().sendMessage(channelType, channelID, SendMessageRequest.builder()
    .message(MessageRequest.builder()
        .text(text)
        .userId(sentBy)
        .build())
    .forceModeration(true)
    .build()).execute();

$client->chat()->sendMessage($channelType, $channelID, new SendMessageRequest(
    message: new MessageRequest(
        text: $text,
        userId: $sentBy,
    ),
    forceModeration: true,
));

client.chat.send_message(channel_type, channel_id,
  GetStream::Generated::Models::SendMessageRequest.new(
    message: GetStream::Generated::Models::MessageRequest.new(
      text: text,
      user_id: sent_by
    ),
    force_moderation: true
  )
)

await client.Chat.SendMessageAsync(channelType, channelID, new SendMessageRequest
{
    Message = new MessageRequest
    {
        Text = text,
        UserID = sentBy,
    },
    ForceModeration = true,
});

Its important to note that if the message is blocked, SendMessage API will NOT return an error. But instead the message object will have a type of error. The moderation object in the response includes the action taken and detailed information about why the message was moderated. The example response is as following:

{
  "message": {
    "id": "2b56e00f-4149-465f-b1e7-62d892788f85",
    "type": "error",
    "text": "Message was blocked by moderation policies",
    "moderation": {
      "action": "remove",
      "original_text": "F*** you motherf***er",
      "text_harms": ["toxicity", "insult"],
      "blocklists_matched": ["profanity-list"]
    },
    "user": {
      "id": "little-lake-2",
      "name": "little-lake-2",
      ...
    },
    "cid": "messaging:first",
    "created_at": "2024-11-21T13:13:38.071535Z",
    "updated_at": "2024-11-21T13:13:38.071535Z",
    "shadowed": false,
    ...
  },
  "duration": "136.06ms"
}

The moderation object contains the following fields:

When a message is blocked by moderation, no WebSocket events (message.new) will be triggered for that message. This means other users in the channel will not receive any notification or see the blocked message in their chat interface. The message is effectively prevented from being delivered to any recipients.

User-Driven Actions

In addition to automated moderation, Stream's Chat SDK provides methods for users to take moderation actions against other users. These actions help maintain community standards by allowing users to report inappropriate behavior or content. For detailed information on effects of these actions, please check the documentation for Chat Moderation Tools. Here we cover some basic examples on how to use these actions.

Flag User

Users can flag another user with a reason. This will trigger a review in the moderation dashboard.

If you are using Chat UI SDK, users can flag the user from user actions menu.

If you have your own custom UI, you can use the following API to flag the user:

await client.moderation.flag({
  entity_type: "stream:user",
  entity_id: "user_id",
  reason: "reason",
  user_id: "user_id", // User ID who is flagging the user, required only for server-side integration.
});

client.moderation().flag(
    entity_type="stream:user",
    entity_id="user_id",
    reason="reason",
    user_id="user_id",  # User ID who is flagging the user, required only for server-side integration.
)

response, err := client.Moderation().Flag(ctx, &getstream.FlagRequest{
    EntityType: "stream:user",
    EntityID:   "user_id",
    Reason:     getstream.PtrTo("reason"),
})

var response = client.moderation()
    .flag(FlagRequest.builder()
        .entityType("stream:user")
        .entityID("user_id")
        .reason("reason")
        .build())
    .execute()
    .getData();

$response = $client->moderation()->flag(new FlagRequest(
    entityType: 'stream:user',
    entityID: 'user_id',
    reason: 'reason',
));

response = client.moderation.flag(
  GetStream::Generated::Models::FlagRequest.new(
    entity_type: "stream:user",
    entity_id: "user_id",
    reason: "reason",
  )
)

var response = await client.Moderation.FlagAsync(new FlagRequest
{
    EntityType = "stream:user",
    EntityID = "user_id",
    Reason = "reason",
});

Flag Message

Users can flag a message, which will trigger a review in the moderation dashboard.

If you are using Chat UI SDK, users can flag the message from message actions menu.

If you have your own custom UI, you can use the following API to flag the message:

await client.moderation.flag({
  entity_type: "stream:chat:v1:message",
  entity_id: "message_id",
  reason: "reason",
  user_id: "user_id", // User ID who is flagging the message, required only for server-side integration.
});

client.moderation().flag(
    entity_type="stream:chat:v1:message",
    entity_id="message_id",
    reason="reason",
    user_id="user_id",  # User ID who is flagging the message, required only for server-side integration.
)

response, err := client.Moderation().Flag(ctx, &getstream.FlagRequest{
    EntityType: "stream:chat:v1:message",
    EntityID:   "message_id",
    Reason:     getstream.PtrTo("reason"),
})

var response = client.moderation()
    .flag(FlagRequest.builder()
        .entityType("stream:chat:v1:message")
        .entityID("message_id")
        .reason("reason")
        .build())
    .execute()
    .getData();

$response = $client->moderation()->flag(new FlagRequest(
    entityType: 'stream:chat:v1:message',
    entityID: 'message_id',
    reason: 'reason',
));

response = client.moderation.flag(
  GetStream::Generated::Models::FlagRequest.new(
    entity_type: "stream:chat:v1:message",
    entity_id: "message_id",
    reason: "reason",
  )
)

var response = await client.Moderation.FlagAsync(new FlagRequest
{
    EntityType = "stream:chat:v1:message",
    EntityID = "message_id",
    Reason = "reason",
});

Ban User

A user with moderator permissions can ban a user from the chat. For more details, please check the documentation for Banning Users.

await client.ban({
  target_user_id: "target_user_id",
  banned_by_id: "user_id", // User ID who is banning the user.
});

client.ban(
    target_user_id="target_user_id",
    banned_by_id="user_id",  # User ID who is banning the user.
)

response, err := client.Ban(ctx, &getstream.BanRequest{
    TargetUserID: "target_user_id",
    BannedByID:   getstream.PtrTo("user_id"), // User ID who is banning the user.
})

var response = client.ban(BanRequest.builder()
    .targetUserID("target_user_id")
    .bannedByID("user_id") // User ID who is banning the user.
    .build())
    .execute()
    .getData();

$response = $client->ban(new BanRequest(
    targetUserID: 'target_user_id',
    bannedByID: 'user_id', // User ID who is banning the user.
));

response = client.ban(
  GetStream::Generated::Models::BanRequest.new(
    target_user_id: "target_user_id",
    banned_by_id: "user_id", # User ID who is banning the user.
  )
)

var response = await client.BanAsync(new BanRequest
{
    TargetUserID = "target_user_id",
    BannedByID = "user_id", // User ID who is banning the user.
});

Unban User

A user with moderator permissions can unban a user from the chat.

await client.unban({
  target_user_id: "target_user_id",
});

client.unban(
    target_user_id="target_user_id",
)

response, err := client.Unban(ctx, &getstream.UnbanRequest{
    TargetUserID: "target_user_id",
})

var response = client.unban(UnbanRequest.builder()
    .targetUserID("target_user_id")
    .build())
    .execute()
    .getData();

$response = $client->unban(new UnbanRequest(
    targetUserID: 'target_user_id',
));

response = client.unban(
  GetStream::Generated::Models::UnbanRequest.new(
    target_user_id: "target_user_id",
  )
)

var response = await client.UnbanAsync(new UnbanRequest
{
    TargetUserID = "target_user_id",
});