Filtering Profanity in Chat with React Native

Unfortunately, there will always be users who tend toward using profanity within chat. As a developer, there will be a time that will come when you need to moderate this type of behavior. There’s simply no way you will be able to manually watch all of the chat rooms and ban users because of their bad behavior. And even if you’re ready to implement an automated solution, blacklisting profanities won’t cover everything – something will always slip through the cracks.

Unfortunately, there will always be users who tend toward using profanity within chat. As a developer, there will be a time that will come when you need to moderate this type of behavior. There's simply no way you will be able to manually watch all of the chat rooms and ban users because of their bad behavior. And even if you're ready to implement an automated solution, blacklisting profanities won't cover everything – something will always slip through the cracks.

In this tutorial, we'll be looking at how to add automatic profanity filtering and moderation to a chat app using an off-the-shelf Machine Learning solution for analyzing text. We will be using the Translator Text API from Microsoft Cognitive Services for analyzing text. For chat, we'll use Stream Chat along with React Native to create the app, and Gifted Chat to generate the chat UI. Alternatively, you can also use our React Native Chat components.

Prerequisites

Basic knowledge of React, React Native, and Node.js is required to follow this tutorial.

This tutorial also assumes that you already have prior experience working with Stream Chat in React Native. If you're new to React Native or Stream Chat, be sure to check out my previous tutorial – Using Gifted Chat UI with React Native.

The following package versions were used in creating and testing this tutorial:

  • Node 11.2.0
  • Yarn 1.13.0
  • React Native CLI 2.0.1
  • React Native 0.59.9
  • Stream Chat 0.13.3 (or above)

You will also need to have an account for the following services:

  • Stream Chat - for implementing the chat feature. Sign up for an account if you don't have one already.
  • ngrok - we use this to expose the Node.js server (for working with Stream Chat) to the internet.
  • Microsoft Azure - allows us to use Cognitive Services. When you sign up for an account.

React Native App Overview

In this tutorial, we will update an existing React Native chat app that was built with Stream Chat and Gifted Chat UI. The current app has the following features:

  • Sending and receiving messages.
  • Viewing the list of room members and their online presence.

We will then add the following features:

  • Masking the profanities.
  • Warning the user when they use profanities and banning them from sending messages if they continue.

Here's what the app will look like:

From the gif above, you can see that profanities are masked using asterisks. On the 'user's first and second offense, they get an alert that they will get banned if they continue to use colorful language. On the third offense, they will get banned from sending messages in all the chat rooms that they're a member of.

Banned users will still be able to receive messages, but 'they'll also have a "banned" marker next to their name when viewing the members of the room:

Setting up Webhooks on Stream Chat

This section assumes that you already have an existing Stream app instance:

Scroll down to the Chat Events section and activate webhooks. At this point, you 'don't have a live URL yet. You can go ahead and expose port 5000 using ngrok and paste the HTTPS URL here:

It won't work yet, because nothing is running on port 5000. But that's where we're going to run the server later on, so it should be fine. Once you're done, click on Save to save the changes.

Setting up the Translator Text API

The Translator Text API is part of Microsoft Cognitive Services, a collection of services that brings machine learning capabilities to your apps via a simple API.

It's primarily used for translating text from one language to another. Though it also has a nice feature for dealing with profanity in text. That is what we will be using in this tutorial.

This section assumes that you already have a Microsoft Azure account, and that you're already logged in to the Azure Portal.

The first step is to search for "Cognitive services":

Once you're inside Cognitive Services, click on the Add button:

This allows you to select the specific Cognitive Services API that you want to use. In this case, 'we'll be using Translator Text so go ahead and search for it:

Click on the first dropdown option that shows up, and you'll be redirected to the following page. When you see this, click on the create button:

That will redirect you to the page for creating the resource. Enter the following details:

  • Name: RNStream
  • Subscription: Pay-As-You-Go
    • Pricing Tier: F0 (note: I've selected S0 in the screenshot simply because I already = have an existing resource that has selected F0 for the very same service)
  • Resource Group: stream

Note that if you don't have an existing resource group, you can click on the Create new link below it and enter its name:

Once the resource is created, you'll see something like this:

You can view the API key that you can use for making requests by clicking on the Show access keys link. You can also see the quantity remaining for your selected pricing tier. If you're on the free tier, you won't get charged. If you go over the quota, the API will stop working, and you'll receive a notification to switch to another tier.

Bootstrapping the App

Since we will be updating an existing app in this tutorial, I've already prepared a starter project which contains all the code of the current app. Execute the following commands on your terminal to set it up:

https://gist.github.com/anchetaWern/f3abbf64dd4ada93ec078babaf93036f

Aside from that, React Native Config also has additional setup necessary.

Next, you need to update the .env and server/.env file with your Stream Chat and Cognitive Services API key.

https://gist.github.com/anchetaWern/2cfd148c853ef7f72cafcd3ae904a910

https://gist.github.com/anchetaWern/ef1267f597d3704f0267484ceeb854a9

Once that's done, you should be able to follow the instructions on the Running the App section so you can try out the basic chat functionality. Running the app now should also prove useful as you go through this tutorial. That way, you can see the results as 'you're adding the code.

Updating the App

Now, we're ready to update the app. We'll first work on updating the server before we proceed to the actual app.

Updating the Server

The first thing that we need to do is add the axios configuration for making requests to the Translator Text API. This is where we supply the API key via the header:

https://gist.github.com/anchetaWern/eeca537c1c3db158c2d7bce8f63e67e8

Next, we need to update the Express middleware that we use, based on whether the x-signature exists in the header or not. This header value is sent by 'Stream's server whenever the webhook is triggered. Currently, the request body is automatically parsed to a JSON object. That's not going to work because the function provided by Stream's API for verifying whether or not a request came from them requires us to pass the request body as plain text. That's what the code below does. If the x-signature header exists, it uses bodyParser.text(), otherwise, it uses bodyParser.json():

https://gist.github.com/anchetaWern/81584f6f70027d5b2bcefc8610c0173b

Next, we add the route for processing the webhook. 'We've already added it earlier in the Stream Chat app dashboard as webhooks, so we have to use the same name. To determine if it did, indeed, come from 'Stream's server, we use the verifyWebhook() function. This accepts the request body as the first argument and the value of the x-signature header as the second. Only if 'it's valid and 'it's triggered because a new message was sent (message.new), do we continue processing the request:

https://gist.github.com/anchetaWern/00c1da6ab0c483451ff256b8ed5402b0

Now that we know that the request is valid, we can extract the message from the request body and construct the request to be sent to the Translator Text API. The API expects a JSON string containing an array of objects with the Text property. Where the Text is the text that you want to process. This will be passed as the request body. The options for the request can be given as query parameters:

  • api-version - the version of the API we want to work with. This is a required parameter. Currently, the latest version is 3.0.
  • to – the language to translate the text to. In this case, we'll assume that the users all speak and write in English, so we set it to en. If you want to handle multiple languages, you can make a GET request to the /languages endpoint of the API to get a list of the supported languages.
  • profanityAction - the action to perform if the API finds profanity in the supplied text. In this case, we want to mask it so that asterisk will show instead. For that, we use Marked. NoAction and Deleted are the two other values you can supply.
  • profanityMarker - the marker that will be used for profanities. This can be either Asterisk or Tag. We'll use Asterisk so we 'don't have to make any additional processing to the text. This will turn all profanities to triple asterisk (***).

The response that we get back is an array. The number of items in this array corresponds to the number of objects you passed in your request body. In this case, we're only working with one text so it's available on the first index. There can also be many possible translations, but we'll extract the first one since it's the most accurate:

https://gist.github.com/anchetaWern/2c37c6dda87e11f314a81d897d0f6792

Next, we check if there are profanities in the text. Since we already know that they're replaced as a triple asterisk, we can check if that value is within the text that was returned. If it is, then we request Stream Chat's API to update the message with the version where the profanities are already masked with asterisks:

https://gist.github.com/anchetaWern/5f2fe535dcfc968b641e3abb2619c9cf

The last thing that we need to do is check for the user's warn_count. This is a custom value that we're adding to a user if their message has profanities in it. If their warn_count is below 3, we simply increment that number. But by the time it becomes 3, we ban the user from ever sending a message to any room:

https://gist.github.com/anchetaWern/6ec1f2b65ad54b5b034dc68ed5ca2a7b

Optionally, you can also add a route for unbanning a user. This would allow the banned user to send messages again:

https://gist.github.com/anchetaWern/741f2b7bbe80ac815f67834238aef9d8
Updating the React Native App
Now 'we're ready to update the app itself. As mentioned earlier, all of the underlying chat features have already been implemented. All we have to do is perform the following:

  • Warning the user if profanities are detected in their message.
  • Alert the user if their message can no longer be sent because 'they're banned.
  • In the modal for showing the members of the room, show a "banned" status next to users who got banned.

The first thing that we need to do is to return the custom user field warn_count. This way, we can show a warning when a user receives an updated message (Because an updated message only means one thing: someone in the room has used colorful language):

https://gist.github.com/anchetaWern/0fc2c086bc51927782432d17fbf296d4

Next, we need to listen to the event when a message is updated. This gets triggered when the message is updated on the server. Just like in the server, each message contains the user object. Because we update the getMessage() function to return the custom field warn_count, 'it's now also available, so we can check for it and issue the corresponding warning:

https://gist.github.com/anchetaWern/37bf0c1091ef2d5098e57e75c2b29546

Next, we update the function for sending a message so that it shows an alert when an error response comes back:

https://gist.github.com/anchetaWern/288f78d03877cfb0315deefc51c02fa0

Lastly, we add the "banned" label to users whose warn_count is greater than two:

https://gist.github.com/anchetaWern/1da8057f9529b6cb72c3a3ab1b0c2431
Running the App
At this point, we're now ready to run the app. The first thing that you need to do is run the server and expose it to the internet:

https://gist.github.com/anchetaWern/31bef8d455e224d8f9187b0d60326b47

Next, update the src/screens/Login.js and src/screens/Chat.js file with the ngrok HTTPS URL.

At this point, we haven't created the channel yet so the app won't work. To set this up, we have to temporarily comment out the code for adding a member into the channel, and the code for subscribing to it:

https://gist.github.com/anchetaWern/b58cae926b410b6cc1950314ac656c00

Once that's complete, you can run the app. Enable live reload and remote debugging so you can see the console.log output. Then, run one of the following commands:

https://gist.github.com/anchetaWern/d04607ebb71ec63b961383654950617f

Log in with the user that we will use as the channel creator.

Once the user is logged in, and the code for setting the user is executed (indicated by "user is set!" console.log output) we know that Stream Chat now recognizes the user.

In the server terminal, you should see the user ID outputted in the console. Copy that and access http://localhost:5000/create-channel?user_id=THE_USER_ID in your browser. That will create the channel for us. If it responds with "ok", you can go ahead and remove the comments on the code that we commented out earlier!

At this point, you can now test the app. If you use a new username for logging in, the username will automatically be added as a member of the room, and you can send messages with that user. If you 'don't have another device to test on, hit the following URL to test the receiving of messages: http://localhost:5000/send-message?user_id=THE_USER_ID.

You can test if the profanity filtering and moderation are working by typing any profanity you can think of. It should ban you after the third offense, and you will no longer be able to send messages. If you want to unban a user, you can access http://localhost:5000/unban?user_id=THE_USER_ID to unban them.

Final Thoughts

In this tutorial, you learned how to implement automatic profanity moderation in a chat app created with React Native and Stream Chat. Specifically, you learned how to use the Translator Text API from Microsoft Cognitive Services to detect profanity in text and obscure them from the user's view. You also learned how to ban users from sending a message if they continue engaging in their bad behavior after being warned.

You can view the full code used in this tutorial on this GitHub repo.

Tutorials

Chat