Implementing AI-Powered Content Moderation in Live Streaming

Almost 30% of internet users watch a livestream each week. If you are ever one of those 30%, you’ll notice the constant stream of video and chat going on from the sometimes thousands of viewers, all wanting to be a part of the experience.

And some choose to be a bad part of that experience. Livestream is one of the most challenging content moderation environments due to its real-time nature and high volume of simultaneous interactions. Livestreams require instant decision-making to prevent harmful content from reaching the audience. A single inappropriate comment can quickly spiral into a cascade of toxic behavior, disrupting the community experience and potentially causing lasting damage to both viewers and creators.

AI is an ideal moderator in these scenarios. It can quickly react to any incoming message in any language from thousands of users, removing it from the chat, and taking appropriate action against the offending account.

This is what you’ll build here. We’re going to create a livestream video and chat app that relays the chat messages to an AI model for moderation. If the message is toxic, it’ll be removed from the chat.

Developer Setup

This application will be based on the Stream Livestream tutorial. Follow the instructions there to start a Stream Livestream application, and then all the other code here will build on that.

Create a Stream Account

To get started, you'll need a Stream account and API credentials. Head over to Stream's signup page to create your free account.

Once you've created your account, follow these steps to set up your project:

1. Log into the Stream Dashboard
2. Click the "Create App" button in the top right corner
3. Give your app a name (e.g., "Livestream Moderation Demo")
4. Choose "Development" mode - this provides free API calls for testing
5. Click "Create App" to generate your project

After creating your app, you'll land on the app dashboard, where you can find your API credentials:

• The Stream API Key - Used to initialize the Stream client
• The API Secret - Required for backend token generation

Keep these credentials handy, as you'll need them throughout this tutorial. The API Secret should be kept secure and never exposed in your frontend code.

Creating Our Livestream App

Following the Livestream tutorial will get you to the point where you’ll have a working livestream from OBS to a React app using Stream. We want to add chat to this livestream so the audience can take part and interact with each other.

To do that, we’ll need to make a few additions to the livestream code. The first part is to install the other Stream libraries required:

1
yarn add stream-chat stream-chat-react

Then import them in our App.tsx:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
// App.tsx

import {
 LivestreamPlayer,
 StreamVideo,
 StreamVideoClient,
 User as VideoUser,
 StreamCall,
} from "@stream-io/video-react-sdk";
import { useState, useEffect } from 'react';
import type {
 User,
 ChannelSort,
 ChannelFilters,
 ChannelOptions,
 StreamChat
} from 'stream-chat';
import {
 Chat,
 Channel,
 ChannelList,
 MessageInput,
 MessageList,
 Thread,
 Window,
} from 'stream-chat-react';
import 'stream-chat-react/dist/css/v2/index.css';

From here, we want to set up our Stream users and the environment variables we need to connect to Stream:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
// App.tsx

// your Stream app information
const userId = "alice";

const user: User = {
 id: userId,
 name: userId,
 image: 'https://getstream.io/random_png/?id=alice&name=Alice',
};

const videoUser: User = { type: "anonymous" };

const apiKey = 'STREAM_API_KEY';
const viewerToken = 'LIVESTREAM_TOKEN';
const callId = 'LIVESTREAM_CALL_ID';
const channelId = 'livestream';

Then we will create our App. To start, we want to set up some state variables to allow us to pass around the token, chat client, video client, and channel instances throughout our application, ensuring proper state management and component communication.

We also have two useEffects. The first one handles token fetching from our backend server, making a POST request to retrieve the authentication token needed for Stream's services. This token is essential for securing our chat and video connections. The second useEffect initializes both our video and chat clients once we have the token, establishing the connection to Stream's services and creating our livestream channel.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
// App.tsx

export default function App() {
 const [token, setToken] = useState<string | null>(null);
 const [chatClient, setChatClient] = useState(null);
 const [videoClient, setVideoClient] = useState(null);
 const [channel, setChannel] = useState(null);

 useEffect(() => {
   const fetchToken = async () => {
     try {
       const response = await fetch('http://localhost:3001/get-token', {
         method: 'POST',
         headers: {
           'Content-Type': 'application/json',
         },
         body: JSON.stringify({ user_id: userId }),
       });
       const data = await response.json();
       console.log(data);
       setToken(data.token);
     } catch (error) {
       console.error('Error fetching token:', error);
     }
   };

   fetchToken();

 }, []);

useEffect(() => {
   if (!token) return;

   const vClient = new StreamVideoClient({
     apiKey,
     user: videoUser,
     token: viewerToken
   });
   setVideoClient(vClient);

   const cClient = new StreamChat(apiKey);
   cClient.connectUser(user, token);
   setChatClient(cClient);

   const channel = cClient.channel('livestream', channelId, {
     name: 'Livestream Chat',
     members: [userId],
   });

   setChannel(channel);

   return () => {
     cClient.disconnectUser();
     vClient.disconnectUser();
   };
 }, [token]);

Next, we need a new way to handle messages rather than the default from Stream. This handleSubmit function is the function that is going to be called from the overrideSubmitHandler of our MessageInput:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
// App.tsx
 const handleSubmit = async (message: { text: string }) => {
   if (!channel) return;

   try {
     // Send to your backend first
     const response = await fetch('http://localhost:3001/messages', {
       method: 'POST',
       headers: {
         'Content-Type': 'application/json',
       },
       body: JSON.stringify({
         text: message.text,
         channelId: channel.id,
         userId: userId,
       }),
     });
     if (!response.ok) {
       throw new Error('Failed to process message');
     }
     // Get the processed message from backend
     const processedMessage = await response.json();
     console.log(processedMessage);

     // Send to Stream channel (either original or processed message)
     return channel.sendMessage({
       text: processedMessage.text || message.text,

     });
   } catch (error) {
     console.error('Error processing message:', error);
     throw error;
   }
 };

This function acts as a message interceptor that first sends the message to a backend server for processing (such as content moderation), then sends either the processed or original message to the Stream channel, maintaining a single source of truth for message handling.

Finally, we render our application by combining Stream's video and chat components in a split layout. The video section displays the livestream using StreamCall and LivestreamPlayer components, while the chat section implements a full-featured chat interface with channel list, message list, message input, and threading capabilities, all connected through our previously initialized clients.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
// App.tsx

 if (!chatClient || !videoClient) return null;

 const call = videoClient.call('livestream', callId);

 return (
   <>
   <div>
   <StreamCall call={call}>
   <StreamVideo client={videoClient}>

       <LivestreamPlayer callType="livestream" callId={callId} />

   </StreamVideo>
   </StreamCall>
   </div>
   <div>
     <Chat client={chatClient}>
     <ChannelList filters={filters} sort={sort} options={options} />
     <Channel>
       <Window>
         <MessageList />
         <MessageInput   overrideSubmitHandler={handleSubmit} />
       </Window>
       <Thread />
     </Channel>
   </Chat>
   </div>
   </>
 );
}

We can then run this code with:

1
yarn run dev

Depending on what you’re streaming, you’ll see something like this:

If we were to start sending messages, those messages would fail as we currently don’t have the endpoint for the handleSubmit function. Let’s build that next.

Building Our Livestream Moderation Server

On the backend, we need a server that will do three things:

1. Allow us to create a Stream user token. This creates a JWT token that will authenticate the user with Stream's services, ensuring secure access to chat features.
2. Take the POST input from our Livestream chat frontend. This receives incoming chat messages through a RESTful endpoint, allowing the frontend to communicate with our moderation system.
3. Call an LLM endpoint to perform the moderation. This sends message content to an LLM API that evaluates and potentially modifies messages based on moderation rules before they reach the chat. Here, we’re going to use Claude from Anthropic.

This can live as part of our React app above, but it is easier to build this independently, giving you more control over how you build out the server. Let’s break it down in the same way as our frontend code. First, we’ll install our dependencies:

1
npm install express core @anthropic-ai stream-chat dotenv

Create an app.js file and add the imports to the top, along with some initial setup:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
// app.js

import express from 'express';
import cors from 'cors';
import Anthropic from '@anthropic-ai/sdk';
import { StreamChat } from 'stream-chat';
import dotenv from 'dotenv';

dotenv.config();

const app = express();
const port = 3001;

app.use(cors());
app.use(express.json());

This sets Express to use cors to enable cross-origin resource sharing. This allows our frontend application running on a different port to communicate with our backend API. app.use(express.json()) configures Express to automatically parse incoming JSON payloads in request bodies.

Then, we’ll initialize our clients using our API keys (which we’ll pull from a .env file using dotenv):

1
2
3
4
5
6
7
8
9
10
11
// app.js

// Initialize Anthropic client
const anthropic = new Anthropic({
 apiKey: process.env.ANTHROPIC_API_KEY,
});

const chatClient = new StreamChat(
 process.env.STREAM_API_KEY,
 process.env.STREAM_API_SECRET
);

We can then set up the first of our two endpoints. This is the /get-token endpoint, which generates a user-specific Stream authentication token using the Stream client's createToken method. This method creates a JWT containing the user's identity and permissions for Stream's services.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
// app.js

app.post('/get-token', async (req, res) => {
 try {
   const { user_id } = req.body;
   const token = chatClient.createToken(user_id);
   res.json({ token });
 } catch (error) {
   console.error('Error creating token:', error);
   res.status(500).json({ error: 'Failed to create token' });
 }
});

app.listen(port, () => {
 console.log(`Server running on port ${port}`);
});

Then the all-important /messages endpoint:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
// app.js

app.post('/messages', async (req, res) => {
 try {
   const { text, channelId, userId } = req.body;

   // Call Claude for content moderation
   const msg = await anthropic.messages.create({
     model: "claude-3-5-sonnet-20241022",
     max_tokens: 1024,
     messages: [{
       role: "user",
       content: `If the message is inappropriate, replace it with [Message removed]. If it's fine, return it as-is: "${text}"`
     }],
   });

   const moderatedText = msg.content[0].text; // Extract the actual response text
   console.log('moderatedText', moderatedText);
   // Send back the moderated message
   res.json({
     text: moderatedText,
     channelId,
     userId,
     moderated: moderatedText !== text
   });

 } catch (error) {
   console.error('Error moderating message:', error);
   res.status(500).json({ error: 'Failed to moderate message' });
 }
});

This endpoint serves as the core moderation pipeline for our chat system. It receives incoming messages, processes them through Claude's AI moderation using a specific prompt, and returns either the original or modified message based on the AI's evaluation.

The endpoint maintains message context by preserving the channelId and userId, while adding a moderation flag to track whether the message was altered. All of this happens synchronously, ensuring real-time moderation without introducing noticeable latency to the chat experience.

Finally, we can start the server:

1
2
3
4
5
// app.js

app.listen(port, () => {
 console.log(`Server running on port ${port}`);
});

We run this code with:

1
node app.js

Now, we’re in a position to start moderating our chat. Let’s say our user adds a nice welcome message to the chat. That message will go through no problem:

But, then she gets excited, and wants the audience to “Make some … noise” [insert your preferred profanity]. Well, then, it gets shut down:

Instead of the message, the LLM returns [Message removed] immediately, and the profanity is never shared with the audience.

This can happen in real-time, synchronously, across thousands of users. There will be no queue, as there would be if human moderators on a trust and safety team had to tag each message, and messages will not be shown and then deleted. To change the level of profanity you are happy with, you have to change the prompt.

This moderation (and others, such as image moderation and different types of text moderation) is currently built into Stream, with AI chat integrations and AI moderation. However, it is important to understand what is happening under the hood, whether you choose the in-built solutions or build your own. AI is an excellent opportunity for moderation, allowing you to create live streams for thousands of viewers, safe in the knowledge that your chat can stay clean and positive for all those watching.