Reviewing Content

Moderators can review all the content where moderation actions have been taken. This could include any content flagged by the moderation engines or manually flagged by the users of the application. The following guides will help you understand how to review content in different products.

Before we jump into the different review queues, let’s understand what the review queues are and how they are used. A review queue is a collection of content that has been flagged by the moderation engines or manually flagged by the users of the application. Moderators are expected to review this queue and take action on the content regularly. The following diagram shows the workflow of content moderation:

graph TD
    A[User posts content] --> B[Moderation Engine]
    B -->|Content is safe| D[Content Published]
    B -->|Content is harmful| C[Review Queue - Inbox]
    E[User flags content] --> C[Review Queue - Inbox]
    C -->|Moderator decides content is safe| G[Safe Content]
    G --> D[Content Published]
    G --> F[Review Queue - Reviewed]
    C -->|Moderator decides content is harmful| F[Review Queue - Reviewed]

On the dashboard, we provide three separate review queues for moderators to facilitate different workflows:

Users Queue: For reviewing flagged users.
Text Content Queue: For reviewing flagged text.
Media Content Queue: For reviewing flagged media.

These queues are available on the sidebar of the dashboard.

Users Queue

The Users Queue is a critical tool for moderators, allowing them to review users who have:

Posted content flagged by the moderation engines.
Been reported by other users.
Been manually flagged by moderators.

For smaller moderation teams (1 or 2 moderators), this queue is particularly efficient for managing flagged users.

Columns In Users Queue

Let’s go through each column in the Users Queue:

User: The user who posted the content.
Action: You will see major action count and minor action count here:
- Major Actions: Number of pieces of content posted by this user that were deleted by the moderator.
- Minor Actions: Number of times this user was banned by the moderator.
Reporter: Who reported this user.
- If the user was reported or flagged by another user, you will see the username or user ID of the reporter.
- If the user was reported by some internal rule, you will see “Automod” as the reporter. E.g., if any content posted by this user was caught by the automod system.
Reviewer: The moderator who reviewed this user. This column will be empty if the user has yet to be reviewed.
Reason: The reason for flagging the user (this is provided by the reporter).
Date: The date when the user was flagged.

Actions On Users Queue

When you hover over a user in the Users Queue, you will see a set of actions that you can take on the user:

Mark Reviewed: This action means that you have reviewed the user and all the flagged content posted by this user, and no further action is needed. This moves the user to the Reviewed tab. Please note that marking a user as reviewed will mark all the content posted by this user as reviewed as well.
Delete User:
- This action will delete the user from the platform. Please note that this is not a hard delete. Please learn more about Soft Delete vs Hard Delete in Managing Users Guide.
- You can choose to delete the user’s content as well as part of this action.
Permanently Ban User: This will permanently ban the user from the platform. When a user is banned, they will not be allowed to post content until the ban is removed or expired.
24h Global Ban: This will temporarily ban the user from the platform for 24 hours.

User Details Screen

When you click on a user in the Users Queue, you will be taken to the User Details Screen. On this screen, you will see the same action buttons as you saw in the Users Queue. You can also find some basic information about the user here, such as their name, username, profile picture, and whether this user is banned or not.

The tabs on the right side of the screen will help you get some context about the user. Let’s go through each tab:

Logs Tab

This tab shows all the history of moderation-related activity around this user. Let’s take a look at examples of a couple of logs:

Log related to flagged content and actions taken:
- User posted content on 2024-09-23 with the text “I am sure you are a virgin :laughing:”
- The ID of this content or entity is f72bfe33-c606-43a6-8d79-b0f86946813c. In the case of chat, this ID will be the message ID. In the case of the activity feed, this ID will be the activity ID or reaction ID.
- This text was flagged by “AI Mod”. The moderation engines that flagged this content were:
  - profanity_en_2020_v1 blocklist
  - “AI Text” harm engine classified this content as “Hatred” and “Sexually Explicit”.
- This content (feeds reaction in this case) was deleted by the moderator on “Nov 1st 2024”.
Log related to user reporting:
The current user was reported by another user, Noa Walker, giving a reason as “Blood” on “September 2nd 2024”.

Content Tab

This tab shows all the content posted by this user that was flagged by the moderation engines or manually flagged by other users. You can click on any content to see more details about it. The explanation can be found in the Text Queue Details document. While reviewing the user, you can go through the content and take actions on it before marking the user as reviewed or taking any other actions.

Notes Tab

This tab shows all the notes added by the moderators about this user. You can add notes about the user here, which will help you identify the user in the future.

Use Cases:

Tracking Behavior: Add notes about any suspicious or inappropriate behavior observed, which can help in identifying patterns over time.
User Warnings: Document any warnings given to the user, including the reason and date, to maintain a record of their conduct.
Context for Actions: Provide context for any actions taken against the user, such as temporary bans or content deletions, to inform future moderation decisions.
Positive Feedback: Record any positive interactions or contributions by the user, which can be useful for recognizing good behavior or contributions to the community.
Follow-up Actions: Note any follow-up actions required, such as monitoring the user’s activity for a certain period or re-evaluating their status after a temporary ban.

Actions Tab

This tab shows all other contents or users reported by this user. This information can be helpful in identifying whether the user is misusing the moderation system.

Text Queue

Moderators can review flagged text content individually by navigating to the Text Queue through the Text tab on the sidebar. This page displays all flagged or blocked text content. The Text Queue is particularly useful for:

Reviewing specific pieces of text content flagged by moderation engines or reported by users.
Identifying patterns in flagged text content to understand common issues or repeated offenses.
Understanding the context and details of specific text entries.

Labels on Text Queue

The labels on the content give you information about the moderation engines that flagged this content. The following table lists all the labels and their descriptions:

Label	Description
	This content was flagged by the AI Moderation engine. You can find explicit labels for the engines that flagged this content in the same list.
	This content was flagged by the blocklist “profanity_en_2020_v1”.
	This content was flagged by the AI Text harm engine as “Vulgarity”.
	This content was manually flagged by a moderator, whose ID is mentioned in the label.
	This content was flagged by a user and provided a reason in the label.
	This content was flagged by the AI Image Moderation engine as “Sexually Explicit”.
	This content was blocked by the moderation engine.
	This content belongs to the chat product, basically a message.
	This content belongs to the feed product, basically an activity.
	This content belongs to the feed product, basically a reaction.

Actions on Text Queue

When you hover over a piece of content in the Text Queue, you will see a set of actions that you can take on the content:

Mark Reviewed: This action means that you have reviewed the content and no further action is needed.
- If the content was blocked by the moderation engine, it will stay blocked.
- If the content was only flagged by the moderation engine, it will stay visible to the users.
Delete: This will delete the content from the platform. This performs a soft delete.
Unblock: If the content was blocked by the moderation engine, you can unblock it.
Permanent Ban: This will permanently ban the user from the platform. When a user is banned, they will not be allowed to post content until the ban is removed or expired.
24h Global Ban: This will temporarily ban the user from the platform for 24 hours.
Channel Ban: This action is available only for the chat product. This will temporarily ban the user from the channel on which the message was sent, and so the user will not be able to send messages to that channel until the ban is removed.
24h Channel Ban: This action is available only for the chat product. This will temporarily ban the user from the channel on which the message was sent, and so the user will not be able to send messages to that channel for 24 hours.

Context Modals

When you click on a piece of content in the Text Queue, it will open the text context modal. This modal gives you access to the following information which can help you understand the context of the content and take actions on it.

Depending on the product type, the context modal will be different. Below are the links to all the context modals:

Message Context Modal: Applicable only for the chat product.
Activity Context Modal: Applicable only for Activity Feed product.
Reaction Context Modal: Applicable only for Activity Feed product.
Custom Content Context Modal: Applicable only for custom content moderation.

Media Queue

The Media Queue is designed for reviewing flagged images, videos, and other media content. Moderators can take actions such as marking content as reviewed, deleting inappropriate content, or unblocking media that was mistakenly flagged.

Actions on Media Queue

When you hover over a piece of content in the Media Queue, you will see a set of actions that you can take on the content. These actions are the same as the actions you saw in the Text Queue.

Context Modals

When you click on a piece of content in the Media Queue, it will open the media context modal. This modal gives you access to the following information which can help you understand the context of the content and take actions on it. Depending on the product type, the context modal will be different. Below are the links to all the context modals:

Message Context Modal: Applicable only for the chat product.
Activity Context Modal: Applicable only for Activity Feed product.
Reaction Context Modal: Applicable only for Activity Feed product.

This section is applicable only for the chat product.

When you click on a piece of content in the Text Queue, it will open the message context modal. This modal gives you access to the following information which can help you understand the context of the content and take actions on it:

The whole discussion in which this message was posted.
The user who posted this message, including their past moderation history, e.g., number of times the user was banned before, number of times this user’s content was deleted before, etc.
All the action buttons which you saw in the Text Queue.
All the notes previously added by the moderators about this user.
Navigational buttons to go to previous and next messages within the review queue.
Button to copy the message context URL to clipboard. This URL can be shared with other moderators to review the message in context.

This section is applicable only for Activity Feed product.

When you click on a piece of content in the Text Queue, it will open the activity context modal. This modal gives you access to the following information which can help you understand the context of the content and take actions on it:

Text and image content of the activity that got flagged.
The user who posted this activity, including their past moderation history, e.g., number of times the user was banned before, number of times this user’s content was deleted before, etc.
All the comments/reactions on this activity.
All the action buttons which you saw in the Text Queue.
All the notes previously added by the moderators about this user.
Navigational buttons to go to previous and next messages within the review queue.
Button to copy the message context URL to clipboard. This URL can be shared with other moderators to review the message in context.

This section is applicable only for Activity Feed product.

When you click on a piece of content in the Text Queue, it will open the reaction context modal. This modal gives you access to the following information which can help you understand the context of the content and take actions on it:

Text and image content of the reaction that got flagged.
The user who posted this reaction, including their past moderation history, e.g., number of times the user was banned before, number of times this user’s content was deleted before, etc.
All the comments/reactions surrounding this reaction. This will help you understand the context of the reaction.
All the action buttons which you saw in the Text Queue.
All the notes previously added by the moderators about this user.
Navigational buttons to go to previous and next messages within the review queue.
Button to copy the message context URL to clipboard. This URL can be shared with other moderators to review the message in context.

When you click on a piece of content in the Text Queue, it will open the custom content context modal. This modal gives you access to the following information which can help you understand the context of the content and take actions on it:

Text and image content of the activity that got flagged.
All the action buttons which you saw in the Text Queue.
All the notes previously added by the moderators about this user.
Navigational buttons to go to previous and next messages within the review queue.
Button to copy the message context URL to clipboard. This URL can be shared with other moderators to review the content.

Configuring Moderation Policy

Moderator Roles and Permissions