await client.moderation.check(
"entity_type",
"entity_id",
"entity_creator_id",
{
texts: ['this is bullshit', "f*ck you as***le"],
images: ['example.com/test.jpg']
},
"config_key",
{
"force_sync": true
}
)
Moderation APIs
Overview
Moderation APIs allow you to integrate moderation capabilities into your product or application. You can use a combination of these APIs along with Stream’s Moderation Dashboard to prevent app users from posting harmful content on your platform. Additionally, you can build a custom dashboard using these APIs.
Let’s examine each of the moderation APIs provided by Stream.
Prerequisites
- Make sure you have created the app on Stream. If not, please refer to Guide to Getting Started with Stream
- Following API documentation presents examples from our javascript client. Please refer to our guide on Stream API and client integration to setup client.
Moderation Check Endpoint
The Moderation Check Endpoint enables real-time content moderation by analyzing submitted content and providing actionable recommendations. This comprehensive API supports multiple content formats, including text messages, images, and video content.
The process follows these simple steps:
- Set up your moderation policy with defined rules and actions (see Policy Setup)
- Submit content for review along with your policy key
- The API analyzes the content using configured moderation engines and rules
- Receive an immediate recommendation based on the analysis
By integrating this endpoint into your application workflow, you can efficiently maintain content quality and ensure compliance with your platform’s standards while providing a safe environment for your users.
API Usage
In the context of the Moderation API, the term "entity" refers to a piece of content that you send for moderation. This can be text, an image, and/or a video.
Request Params
| key | required | type | description |
|---|---|---|---|
| entity_type | true | string | This is identifier for the type of content you are sending for moderation. This helps with categorizing content on dashboard. E.g., if you have multiple products then you can set unique entity_type for content coming from each product. It could be any string. |
| entity_id | true | string | Unique identifier for entity |
| entity_creator_id | true | string | Unique identifier for user who created this entity. Generally this is the user id of the app user. |
| moderation_payload | true | object | Entity or content to be moderated |
| config_key | true | string | Key of the config to be used for moderation. Please check Config Setup API for details |
Response
| key | type | description |
|---|---|---|
| status | “complete” | “partial” |
| task_id | string | Id of the task running the async moderation. You can check the status of the task itself using GetTask endpoint. |
| recommended_action | “flag”, “remove”, “keep” | Final result of moderation which suggest what action should be taken for moderated entity.
|
| item | object | This is basically a json representation of the review queue item accessible on Stream dashboard. You can use this to compose your own dashboard UI. |
Upsert Config
We provide a separate API endpoint to enable or disable specific harm engines. Stream supports various moderation engines, both in-house and from subprocessors, such as toxicity detection, platform circumvention prevention, Bodyguard AI, and AWS Rekognition. We plan to introduce more in the future. Each engine processes content and provides relevant classifications. As a customer, you don’t need to handle each result individually—that would be overwhelming. Instead, we’ve simplified the process by allowing you to map these classifications to a few values representing “Recommended Actions”.
These recommended actions are handled automatically in Stream’s Chat and Activity Feeds products. For custom moderation, you’ll receive the final recommended action through the check endpoint (TODO: add link), allowing you to take appropriate action on the content.
API Usage
await client.moderation.upsertConfig({
key: "unique_config_key",
bodyguard_config: {
rules: [
{label: "SCAM", action: "flag"},
{label: "SEXUAL_HARASSMENT", severity_rules [
{severity: "low", action: "flag"},
{severity: "medium", action: "flag"},
{severity: "high", action: "remove"},
{severity: "critical", action: "remove"}
]}
]
},
block_list_config: {
enabled: true,
rules: [
{
name: 'blocklistname',
action: 'remove',
},
],
},
automod_toxicity_config: {
enabled: true,
rules: [
{ "label": "identity_attack", "threshold": 0.5, "action": "flag" },
{ "label": "insult", "threshold": 0.5, "action": "remove" },
{ "label": "obscene", "threshold": 0.5, "action": "flag" },
{ "label": "sexual_explicit", "threshold": 0.5, "action": "flag" },
{ "label": "threat", "threshold": 0.5, "action": "flag" }
],
},
aws_rekognition_config: {
enabled: true,
rules: [{ label: 'Non-Explicit Nudity', action: 'flag' }],
},
automod_platform_circumvention_config: {
enabled: true,
rules: [{action: "remove", label: "platform_circumvention", threshold: 0.5}]
}
});
Request Parameters
| Key | Required | Type | Description |
|---|---|---|---|
| key | true | string | Unique identifier for the config. You’ll need to specify this key when sending content for moderation check. |
| bodyguard_config | false | object | Configuration for the Bodyguard AI engine. |
| block_list_config | false | object | Configuration for the block list. This allows you to disallow specific words or phrases on your platform. |
| automod_toxicity_config | false | object | Configuration for Stream’s internal toxicity engine. |
| aws_rekognition_config | false | object | Configuration for image moderation. Please refer todocsfor a list of all the harms. |
| automod_platform_circumvention_config | false | object | Configuration for Stream’s platform circumvention engine. It helps prevent users from moving to different platforms for communication. |
For out-of-the-box moderation in our Chat and Activity Feeds products, we use specific conventions for the configuration key:
- Config key for a specific chat channel:
chat:messaging:channel_id - Config key for a channel type:
chat:messaging - Config key for all channels of any type:
chat - Config key for the Activity Feeds product:
feeds:default
Get Config
API Usage
You can retrieve the config for specific key as following
await client.moderation.getConfig("config_key")
Flag Content
Flagging content marks it for review by moderators. Flagged items appear on the Stream dashboard, where moderators can take appropriate action.
await client.moderation.flag(
"entity_type",
"entity_id",
"entity_creator_id",
"reason for flag",
{
custom: {},
moderation_payload: { text: ['this is shit'] },
user_id: 'user_id' // only for server side usage
}
)
Request Params
| key | required | type | description |
|---|---|---|---|
| entity_type | true | string | This is identifier for the type of content you are sending for moderation. This helps with categorizing content on dashboard. E.g., if you have multiple products then you can set unique entity_type for content coming from each product. It could be any string. |
| entity_id | true | string | Unique identifier for entity |
| entity_creator_id | true | string | Unique identifier for user who created this entity. Generally this is the user id of the app user. |
| reason | true | string | Reason for flagging. This is necessary for any moderator to know the intent |
| options.moderation_payload | false | object | Content to be manually reviewed. This is used to display the content on dashboard for review. |
| options.custom | true | string | Any custom properties you may want to attach with this flag |
| options.user_id | false | string | Only needed if flag API is being used with server side usage. |
Response
| key | type | description |
|---|---|---|
| item_id | string | Id for the Review Queue Item which got created as a result of flagging. You can request the entire review queue object using GetReviewQueueItem endpoint [TODO: Add link] |
Flag a user
When a user/moderator flags a user, it ends up in “Users” review list (as shown in following screen
await client.moderation.flagUser("target_user_id", "spam")
// Internally this method simply calls the flag endpoint as following
// await client.moderation.flag(
// "stream:user",
// "target_user_id",
// "",
// "spam",
//)
Flag a message
await client.moderation.flagMessage("message_id", "spam")
// Internally this method simply calls the flag endpoint as following
// await client.moderation.flag(
// "stream:chat:v1:message",
// "message_id",
// "",
// "spam",
//)
Query Review Queue
This endpoint allows you to query the review queue. It supports a wide range of filters and sorting options, enabling you to build a dashboard that’s easy for moderators to use.
const filters = { entity_type: "some_entity_type" }
const sort = [{ field: "created_at", direction: -1 }]
const options = { next: null } // cursor pagination
const { items, next } = await client.moderation.queryReviewQueue(
filters,
sort,
options
)
Request Params
We support filtering by following params
| key | type | description |
|---|---|---|
| id | string | Query items by specific id |
| created_at | datetime | Query review items created on specific date |
| updated_at | datetime | |
| status | partial, complete | Query items for which moderation check has completed |
| entity_type | string | Query by entity type. E.g.,
|
| entity_id | string | |
| entity_creator_id | string | |
| user_report_reason | string | Query items which were flagged by users manually with given reason |
| reviewed | boolean | Query items which have been reviewed. Calling SubmitAction API on any item marks it as reviewed |
| has_text | boolean | Query items which has flagged text content |
| has_image | boolean | Query items which has flagged image content |
| has_video | boolean | Query items which has flagged video content |
| category | aws_rekognition, automod_toxicity, automod_platform_circumvention, automod_semantic_filters, bodyguard, block_list | Query items which are flagged by specific moderation engine. |
| label | string | Query items which were classified with certain label by moderation engine E.g., spam |
| reporter_type | moderator, user, automod | E.g., Query items which were flagged by user with moderator role |
| reporter_id | string | Query items which were flagged by specific user |
| date_range | string | Query items created between specific date range. E.g., { daterange: "2020-04-20T00:00:00_2035-04-20T00:00:00" } |
| recommended_action | remove, flag, shadow_block | Recommended Action by AI moderation |
We support following sort parameters
| key | type | description |
|---|---|---|
| id | string | |
| created_at | string | |
| updated_at | string |
Response
| key | type | description |
|---|---|---|
| items | array | List of review queue items |
| next | string | Next cursor for pagination purpose |
| stats | object | Counts of pending review queue items. E.g., { texts: 30, users: 5, media: 10 } |
Sample Response
{
"items": [{
"id": "a68277a4-a5c4-4f74-a039-ad92f93ac827",
"created_at": 1725276192603929000,
"updated_at": 1725276210096123000,
"entity_type": "stream:feeds:v2:activity",
"entity_id": "80f650aa-692e-11ef-8080-80016e37cdb0",
"moderation_payload": {
"texts": ["fuck you asshole. I really like her. We had an amazing date, but I am not sure if I feel attracted to her. I am confused. She is just not smart enough for me and also her face is pretty average. I don't want ugly babies. What should I do?"],
"images": ["https://github.com/user-attachments/assets/30a86186-6f76-4881-935a-2c547d904ed3"],
"custom": {
"feed": "user:1725276189"
}
},
"moderation_payload_hash": "3281929468518332009",
"has_image": true,
"has_video": false,
"has_text": true,
"status": "complete",
"recommended_action": "flag",
"completed_at": "2024-09-02T11:23:30.096683Z",
"languages": [],
"severity": 0,
"reviewed_at": null,
"reviewed_by": "",
"entity_creator": {
"id": "user-1725276189-5",
"role": "user",
"custom": {
"name": "Lucius Young"
},
"created_at": 1725276189438843000,
"updated_at": 1725276189438843000,
"banned": false,
"online": false,
"ban_count": 0,
"deleted_content_count": 0
},
"bans": [],
"flags": [{
"type": "block_list",
"entity_type": "stream:feeds:v2:activity",
"entity_id": "80f650aa-692e-11ef-8080-80016e37cdb0",
"moderation_payload": {
"texts": ["fuck you asshole. I really like her. We had an amazing date, but I am not sure if I feel attracted to her. I am confused. She is just not smart enough for me and also her face is pretty average. I don't want ugly babies. What should I do?"],
"images": ["https://github.com/user-attachments/assets/30a86186-6f76-4881-935a-2c547d904ed3"],
"custom": {
"feed": "user:1725276189"
}
},
"moderation_payload_hash": "3281929468518332009",
"labels": ["profanity_en_2020_v1"],
"result": [{
"text": "fuck you asshole. I really like her. We had an amazing date, but I am not sure if I feel attracted to her. I am confused. She is just not smart enough for me and also her face is pretty average. I don't want ugly babies. What should I do?",
"action": "flag",
"labels": ["profanity_en_2020_v1"],
"provider_name": "block_list"
}],
"review_queue_item_id": "a68277a4-a5c4-4f74-a039-ad92f93ac827",
"created_at": 1725276192742036000,
"updated_at": 1725276192742036000
}, {
"type": "automod_toxicity",
"entity_type": "stream:feeds:v2:activity",
"entity_id": "80f650aa-692e-11ef-8080-80016e37cdb0",
"moderation_payload": {
"texts": ["fuck you asshole. I really like her. We had an amazing date, but I am not sure if I feel attracted to her. I am confused. She is just not smart enough for me and also her face is pretty average. I don't want ugly babies. What should I do?"],
"images": ["https://github.com/user-attachments/assets/30a86186-6f76-4881-935a-2c547d904ed3"],
"custom": {
"feed": "user:1725276189"
}
},
"moderation_payload_hash": "3281929468518332009",
"labels": ["obscene", "insult"],
"result": [{
"text": "fuck you asshole. I really like her. We had an amazing date, but I am not sure if I feel attracted to her. I am confused. She is just not smart enough for me and also her face is pretty average. I don't want ugly babies. What should I do?",
"action": "flag",
"labels": ["obscene", "insult"],
"provider_name": "automod_toxicity"
}],
"review_queue_item_id": "a68277a4-a5c4-4f74-a039-ad92f93ac827",
"created_at": 1725276192742038000,
"updated_at": 1725276192742038000
}, {
"type": "bodyguard",
"entity_type": "stream:feeds:v2:activity",
"entity_id": "80f650aa-692e-11ef-8080-80016e37cdb0",
"moderation_payload": {
"texts": ["fuck you asshole. I really like her. We had an amazing date, but I am not sure if I feel attracted to her. I am confused. She is just not smart enough for me and also her face is pretty average. I don't want ugly babies. What should I do?"],
"images": ["https://github.com/user-attachments/assets/30a86186-6f76-4881-935a-2c547d904ed3"],
"custom": {
"feed": "user:1725276189"
}
},
"moderation_payload_hash": "3281929468518332009",
"labels": ["INSULT", "SUPPORTIVE", "VULGARITY"],
"result": [{
"provider_name": "bodyguard",
"text": "fuck you asshole. I really like her. We had an amazing date, but I am not sure if I feel attracted to her. I am confused. She is just not smart enough for me and also her face is pretty average. I don't want ugly babies. What should I do?",
"action": "flag",
"labels": ["INSULT", "SUPPORTIVE", "VULGARITY"]
}],
"review_queue_item_id": "a68277a4-a5c4-4f74-a039-ad92f93ac827",
"created_at": 1725276192742039000,
"updated_at": 1725276192742039000
}],
"actions": [{
"id": "aed07d1f-e3e8-4237-bec0-f8d9bf591280",
"created_at": 1725276192749455000,
"type": "flag",
"user": null,
"reason": "",
"custom": {
"flag_type": "bodyguard"
},
"target_user_id": "user-1725276189-5",
"review_queue_item_id": "a68277a4-a5c4-4f74-a039-ad92f93ac827"
}, {
"id": "139b7c59-70c0-4e1d-96f0-e057a93a4833",
"created_at": 1725276192749455000,
"type": "flag",
"user": null,
"reason": "",
"custom": {
"flag_type": "automod_toxicity"
},
"target_user_id": "user-1725276189-5",
"review_queue_item_id": "a68277a4-a5c4-4f74-a039-ad92f93ac827"
}, {
"id": "b4660069-5855-4b20-a0fe-d769f190a3b7",
"created_at": 1725276192749455000,
"type": "flag",
"user": null,
"reason": "",
"custom": {
"flag_type": "block_list"
},
"target_user_id": "user-1725276189-5",
"review_queue_item_id": "a68277a4-a5c4-4f74-a039-ad92f93ac827"
}],
"content_changed": false
}, {
"id": "947712cf-fd61-4fd9-bdf3-f2c9064977a9",
"created_at": 1725276191973725000,
"updated_at": 1725276191973725000,
"entity_type": "stream:chat:v1:message",
"entity_id": "d10120bc-f3c6-4c47-a981-3b4a7b1cc748",
"moderation_payload": {
"texts": ["This is my phone number. +91 9958592028. Contact me for more information on whatsapp :)"],
"images": ["https://github.com/user-attachments/assets/06d6c129-6b6e-4be2-94fd-c0f1cdd77278"],
"custom": {
"original_message_type": "regular"
}
},
"moderation_payload_hash": "17739892842151117680",
"has_image": true,
"has_video": false,
"has_text": true,
"status": "complete",
"recommended_action": "remove",
"completed_at": null,
"languages": [],
"severity": 0,
"reviewed_at": null,
"reviewed_by": "",
"message": {
"id": "d10120bc-f3c6-4c47-a981-3b4a7b1cc748",
"text": "This is my phone number. +91 9958592028. Contact me for more information on whatsapp :)",
"html": "<p>This is my phone number. +91 9958592028. Contact me for more information on whatsapp :)</p>\n",
"type": "error",
"user": {
"id": "sophia2-76",
"role": "user",
"custom": {
"name": "Sophia Bailey",
"image": "https://xsgames.co/randomusers/assets/avatars/female/14.jpg"
},
"created_at": 1725275473485946000,
"updated_at": 1725275473485946000,
"banned": false,
"online": false
},
"attachments": [{
"type": "image",
"image_url": "https://github.com/user-attachments/assets/06d6c129-6b6e-4be2-94fd-c0f1cdd77278",
"asset_url": "https://github.com/user-attachments/assets/06d6c129-6b6e-4be2-94fd-c0f1cdd77278",
"custom": {}
}],
"latest_reactions": [],
"own_reactions": [],
"reaction_counts": {},
"reaction_scores": {},
"reaction_groups": {},
"reply_count": 0,
"deleted_reply_count": 0,
"cid": "messaging:sample-app-channel-2",
"created_at": 1725276192098227000,
"updated_at": 1725276192098227000,
"custom": {},
"shadowed": false,
"mentioned_users": [],
"silent": false,
"pinned": false,
"pinned_at": null,
"pinned_by": null,
"pin_expires": null
},
"entity_creator": {
"id": "sophia2-76",
"role": "user",
"custom": {
"name": "Sophia Bailey",
"image": "https://xsgames.co/randomusers/assets/avatars/female/14.jpg"
},
"created_at": 1725275473485946000,
"updated_at": 1725275473485946000,
"banned": false,
"online": false,
"ban_count": 0,
"deleted_content_count": 0
},
"bans": [],
"flags": [{
"type": "automod_platform_circumvention",
"entity_type": "stream:chat:v1:message",
"entity_id": "d10120bc-f3c6-4c47-a981-3b4a7b1cc748",
"moderation_payload": {
"texts": ["This is my phone number. +91 9958592028. Contact me for more information on whatsapp :)"],
"images": ["https://github.com/user-attachments/assets/06d6c129-6b6e-4be2-94fd-c0f1cdd77278"],
"custom": {
"original_message_type": "regular"
}
},
"moderation_payload_hash": "17739892842151117680",
"labels": ["platform_circumvention"],
"result": [{
"provider_name": "automod_platform_circumvention",
"text": "This is my phone number. +91 9958592028. Contact me for more information on whatsapp :)",
"action": "flag",
"labels": ["platform_circumvention"]
}],
"review_queue_item_id": "947712cf-fd61-4fd9-bdf3-f2c9064977a9",
"created_at": 1725276192082384000,
"updated_at": 1725276192082384000
}, {
"type": "bodyguard",
"entity_type": "stream:chat:v1:message",
"entity_id": "d10120bc-f3c6-4c47-a981-3b4a7b1cc748",
"moderation_payload": {
"texts": ["This is my phone number. +91 9958592028. Contact me for more information on whatsapp :)"],
"images": ["https://github.com/user-attachments/assets/06d6c129-6b6e-4be2-94fd-c0f1cdd77278"],
"custom": {
"original_message_type": "regular"
}
},
"moderation_payload_hash": "17739892842151117680",
"labels": ["SCAM", "PII"],
"result": [{
"text": "This is my phone number. +91 9958592028. Contact me for more information on whatsapp :)",
"action": "remove",
"labels": ["SCAM", "PII"],
"provider_name": "bodyguard"
}],
"review_queue_item_id": "947712cf-fd61-4fd9-bdf3-f2c9064977a9",
"created_at": 1725276192082387000,
"updated_at": 1725276192082387000
}, {
"type": "aws_rekognition",
"entity_type": "stream:chat:v1:message",
"entity_id": "d10120bc-f3c6-4c47-a981-3b4a7b1cc748",
"moderation_payload": {
"texts": ["This is my phone number. +91 9958592028. Contact me for more information on whatsapp :)"],
"images": ["https://github.com/user-attachments/assets/06d6c129-6b6e-4be2-94fd-c0f1cdd77278"],
"custom": {
"original_message_type": "regular"
}
},
"moderation_payload_hash": "17739892842151117680",
"labels": ["Weapons", "Violence", "Weapon Violence", "Graphic Violence"],
"result": [{
"labels": ["Weapons", "Violence", "Weapon Violence", "Graphic Violence"],
"provider_name": "aws_rekognition",
"image": "https://github.com/user-attachments/assets/06d6c129-6b6e-4be2-94fd-c0f1cdd77278",
"action": "flag"
}],
"review_queue_item_id": "947712cf-fd61-4fd9-bdf3-f2c9064977a9",
"created_at": 1725276222331622000,
"updated_at": 1725276222331622000
}],
"actions": [{
"id": "8bfb5b9c-2570-43f7-b9e3-ab5298170f71",
"created_at": 1725276222349316000,
"type": "flag",
"user": null,
"reason": "",
"custom": {
"flag_type": "aws_rekognition"
},
"target_user_id": "sophia2-76",
"review_queue_item_id": "947712cf-fd61-4fd9-bdf3-f2c9064977a9"
}, {
"id": "ffcfd5cf-0446-418d-852f-c4ec94f9994a",
"created_at": 1725276192092815000,
"type": "flag",
"user": null,
"reason": "",
"custom": {
"flag_type": "automod_platform_circumvention"
},
"target_user_id": "sophia2-76",
"review_queue_item_id": "947712cf-fd61-4fd9-bdf3-f2c9064977a9"
}, {
"id": "bff0c43e-0d36-421b-9012-91e681c175bc",
"created_at": 1725276192092815000,
"type": "flag",
"user": null,
"reason": "",
"custom": {
"flag_type": "bodyguard"
},
"target_user_id": "sophia2-76",
"review_queue_item_id": "947712cf-fd61-4fd9-bdf3-f2c9064977a9"
}],
"content_changed": false
}],
"stats": {
"media": 90,
"users": 51,
"texts": 181
},
"next": "W3sibmFtZSI6ImNyZWF0ZWRfYXQiLCJ2YWx1ZSI6IjIwMjQtMDktMDJUMTE6MjM6MTEuOTczNzI1WiIsImRpcmVjdGlvbiI6LTF9XQ==",
"duration": "141.27ms"
}
Submit Action
This endpoint allows you to perform specific actions on review queue items or flagged content. It’s the underlying API used on Stream’s moderation dashboard when a moderator deletes a flagged message or unblocks a blocked message.
This endpoint serves multiple purposes:
- For Stream’s chat or activity feeds products: It enables you to delete messages or ban users from your custom moderation dashboard.
- For in-house products: You can inform Stream about actions taken on review queue items, ensuring these actions are reflected in Stream’s moderation dashboard.
await client.moderation.submitAction("action_type", "item_id")
Request Params
| key | required | type | description |
|---|---|---|---|
| action_type | true | string | |
| item_id | true | string | |
| options | false | object |
If you are using Stream’s chat or feeds product, we have some set of predefined actions which you can use. These actions are used on Stream dashboard when you press one of the buttons highlighted in following screenshot.
Mark Item As Reviewed
Marks the review queue item as reviewed. Basically on dashboard this will simply move the item from Inbox tab to Reviewed tab. No specific action is taken on the item itself.
await client.moderation.submitAction("mark_reviewed", "item_id")
Ban a user
Bans the user entity or creator of flagged entity. You can globally ban the user or ban only for chat channel.
await client.moderation.submitAction(
"ban",
"item_id",
{
ban: {
reason: "spam",
channel_ban_only: false // default
}
}
)
Delete Chat Message
await client.moderation.submitAction(
"delete_message",
"item_id",
{ delete_message: { hard_delete: false } }
)
Delete Feeds Activity
await client.moderation.submitAction("delete_activity", "item_id")
Delete User
await client.moderation.submitAction("delete_user", "item_id", {
delete_user: {
hard_delete: false,
mark_messages_deleted: false,
delete_conversation_channels: false,
}
})
Delete Feeds Reaction
await client.moderation.submitAction("delete_reaction", "item_id", {
delete_reaction: {
hard_delete: false, // default
}
})
Unban User
await client.moderation.submitAction("unban", "item_id", {
unban: {
channel_cid: "channel_cid", // to unban channel banned user
}
})
Restore Chat Message or Activity or Reaction
await client.moderation.submitAction("restore", "item_id")
Unblock Message
await client.moderation.submitAction("unblock", "item_id")
Custom Check
The Custom Check endpoint enables you to submit your own AI moderation results for review by moderators on the dashboard. This API is currently being tested and not available to all users. If you are interested in using this API, please contact Stream Support Team.
On this page: