AI Text (LLM)

The LLM engine reads each message in context, so it catches harmful intent that keyword lists miss.

• Describe in plain language what you want the system to detect, and the engine will handle the rest.
• Share context about your application so the LLM can better understand the context in which users are interacting.
• Moderation considers both the message and its context, focusing on user intent rather than just keywords.
• Detects and blocks attempts to bypass moderation by splitting harmful content across multiple messages.

Language Support

There is no fixed language list. The engine reads the message in whatever language it is written in, and your rules and app context can stay in English. Accuracy is highest in widely used languages. For less common languages, dialects, and heavy slang, test with real samples of your content in Evaluations before you rely on automatic actions. If your users mainly write in specific languages, say so in the app context.

How it works

The LLM moderation engine classifies content based on your configuration and recent conversation history. To set it up, you'll provide two key inputs:

App Context: A short description of your application and how users interact. This helps the LLM understand your platform's environment (e.g., casual chat, professional tone, teen audience).

LLM Rules: Clear harm categories written in plain language. For example, you might define what counts as SCAM, HATE SPEECH, or SEXUAL HARASSMENT in your community.

When a new message is sent, the engine considers:

• Your defined app context
• Your LLM rules
• The last few messages in the conversation

It then classifies the message against your rules and applies the configured action (flag, shadow, remove, bounce, or keep).

Context from custom data

A comment often only makes sense next to what it replies to. "Hope he breaks the other one too" means one thing under an article about a record-breaking season and another under a video of a player's broken leg. To give the LLM that background, list the moderation_payload.custom keys it may read on the policy's llm_config:

The context is background only:

• Only the texts are moderated. Results, labels and flagged text always refer to them, never to the context.
• The LLM treats the context as data. It cannot change your rules, labels, severities or strictness, and instructions inside it are ignored.
• Only the listed keys are sent, and only the ones present on the item. Other keys in custom are not shown to the LLM. When none of the listed keys are present, the description is dropped too and the check runs as if no context were configured.
• The listed data is sent as JSON, capped at 4000 bytes. Longer data is cut off. The full custom is still stored on the review queue item and sent on webhooks unchanged. The Check limit of 10KB for custom still applies.
• It works for any entity_type. Chat messages and Feeds activities and comments pass their own custom fields as moderation_payload.custom, so for them the context has to be set on the message or activity itself.

Context can lower a match's severity without changing its label, for example when an article shows a comment is trash talk rather than a threat. For a lower severity to change the outcome, the label needs an action per severity, for example high to remove and low to flag. Set this with LLM severity rules. Without them, the label's single action applies at every severity.

Upserting the policy replaces the whole llm_config, so an upsert without these fields clears them.

Configure the policy with the keys and how to read them:

await client.moderation.upsertConfig({
  key: "news-comments",
  llm_config: {
    enabled: true,
    app_context:
      "Comment sections under sports articles and videos. Trash talk about teams and performances is normal and allowed; attacks on people are not.",
    context_custom_keys: ["content", "thread"],
    custom_keys_context_description:
      "content is the article or video the comment is posted under. thread.topLevel is the comment being replied to; thread.earlierReplies are earlier replies, oldest first.",
    rules: [
      {
        label: "HARASSMENT",
        description:
          "Insulting, demeaning or bullying another user, player, coach or referee as a person",
        action: "flag",
      },
      {
        label: "VIOLENCE",
        description:
          "Threats, wishing injury or death on someone, or encouraging people to show up at someone's home",
        action: "remove",
      },
    ],
  },
});

Then send the item with its surroundings in custom:

await client.moderation.check({
  entity_type: "comment",
  entity_id: "comment-123",
  entity_creator_id: "user-42",
  config_key: "news-comments",
  moderation_payload: {
    texts: [
      "bet, see everyone at his place after the game, I'll bring friends",
    ],
    custom: {
      schemaVersion: 1,
      content: {
        type: "video",
        title: "Controversial late call by referee decides the game",
      },
      thread: {
        topLevel: {
          id: "comment-100",
          user: { id: "user-7" },
          text: "That ref lives at 22 Oak Ridge Drive, someone should pay him a visit",
        },
        earlierReplies: [
          {
            id: "comment-110",
            user: { id: "user-9" },
            text: "He deserves whatever he gets",
          },
        ],
      },
    },
  },
});

On its own the comment reads as harmless. With the thread as context the LLM returns remove with the VIOLENCE label, and the flagged text is only the comment. schemaVersion is not listed in context_custom_keys, so the LLM never sees it.

The content and thread keys above use the comment context format, so the same custom also shows the article and thread to moderators in the review queue. The two features are independent: the review queue reads custom by its shape, the LLM reads only the keys you list.

Configuration

await client.moderation.upsertConfig({
  key: "chat:messaging",
  llm_config: {
    app_context:
      "A social chat app for adults. Casual tone is fine. Threats and scams are not.",
    rules: [
      {
        label: "SCAM",
        description: "Fraud, phishing, or requests for payment credentials.",
        action: "remove",
      },
      {
        label: "HATE_SPEECH",
        description: "Hatred, discrimination, or violence against a group.",
        action: "flag",
      },
    ],
    severity_descriptions: {
      low: "Rude but not targeting a person.",
      high: "A direct attack or a credible threat.",
    },
  },
});

LLM rule actions are keep, flag, shadow, remove, bounce, bounce_flag, and bounce_remove. severity_rules on a rule map low, medium, high, and critical to an action. queryModerationRules returns default_llm_labels with the starter descriptions.

You can set up the same fields in the dashboard.

1. In the dashboard, navigate to Moderation → Policies.
2. Select the policy you want to update.
3. Navigate to AI (LLM) Text.
4. Add your app context and harm labels, then define prompts for each label.

Context keys and their description are set through the API (llm_config.context_custom_keys and custom_keys_context_description).

Starter labels are SCAM, SEXUAL_HARASSMENT, HATE_SPEECH, PII, and PLATFORM_BYPASS. You can edit or remove any of them, or add your own.