# Text Normalization Before any engine runs — word and regex [filters](https://getstream.io/moderation/docs/go-golang/configuration/filters/), domain lists, AI text, LLM rules, or the review queue — Stream decodes the text to the form a browser would render, then strips invisible characters. Engines therefore match the **decoded** string, not the raw payload. This is always on for Chat messages, Feeds activities and comments, and the [Check](https://getstream.io/moderation/docs/go-golang/content-moderation/check/) API. The stored message body is left unchanged; only the copy sent to moderation is rewritten. ## Why this exists A common bypass is to entity-encode a blocked word or URL so a plain rule never sees it, while the client still renders the real text: ```text visit spam now see https://evil.example/path ``` Those used to miss a word list containing `spam` and a regex like `(?i)https?://`. They now decode to `visit spam now` and `see https://evil.example/path`, and the same plain rules match. ## What decoding covers HTML character references are resolved, including the forms browsers accept without a trailing semicolon. | Form | Example input | After decode | | --------------------- | --------------------------- | --------------------- | | Decimal | `http` | `http` | | Hex | `http` | `http` | | Hex, uppercase prefix | `h` | `h` | | Decimal, no semicolon | `http` | `http` | | Hex, no semicolon | `http` | `http` | | Named | `https://example.com` | `https://example.com` | | Named, no semicolon | `https&colon//example.com` | `https://example.com` | | Standard named | `a < b & c` | `a < b & c` | | Nested amp-encoding | `&#104;` | `h` | | Triple-encoded amp | `&amp;#104;` | `h` | Nested encoding is unwound for several passes (`&#104;` → `h` → `h`), which covers the usual double- and triple-encoded chains. Named entities are matched case-insensitively when the lowercase HTML name exists (`&COLON;` → `:`, `/` → `/`). A name that is itself a distinct HTML entity keeps that code point (`∷` is `∷`, not `:`). Unknown names such as `&zzzxnotanentity;` are left as-is. Ampersands that are not part of an entity are left alone (`A & B`). ## Zero-width and invisible characters After decode, these code points are **removed** (not replaced with a space). Encoded forms such as `​` or `​` are decoded first, then stripped, so `h​t​t​p` and a literal `h` + U+200B + `ttp` both become `http`. | Category | Code points | | ----------------------------------------------- | --------------------------------------------------------------------------------------- | | Zero-width | U+200B (ZWSP), U+200C (ZWNJ), U+200D (ZWJ), U+2060 (word joiner), U+FEFF (BOM / ZWNBSP) | | Soft hyphen / joiners | U+00AD, U+034F (CGJ), U+180E (Mongolian vowel separator) | | Bidirectional marks | U+200E, U+200F, U+061C | | Bidirectional embeddings / overrides / isolates | U+202A–U+202E, U+2066–U+2069 | | Invisible operators | U+2061–U+2064 | | Line / paragraph separators | U+2028, U+2029 | | Interlinear annotation | U+FFF9–U+FFFB | ## What this means for your rules You can write filters against the **rendered** text. - A word list containing `spam` matches `visit spam now`. - A regex `(?i)https?://` matches an entity-encoded URL. - A [domain](https://getstream.io/moderation/docs/go-golang/configuration/filters/#domain-matching) list matches the host after decode (and after the usual host canonicalization: lowercase, IDN / Punycode, fullwidth dots). If you added encoding-aware regexes to catch `&#\d+;` / `&#` chains, those are no longer required for this bypass. Prefer a plain URL regex or a domain list. Keeping the encoding regex is only useful if you still want to flag the _presence_ of entity syntax itself. > > **Info:** Review-queue items store the **normalized** text the engines evaluated, so dashboard review and webhooks show `https://evil.example/path`, not the encoded source. The Chat / Feeds message the user sent is stored as they typed it. > ## What is not covered This pre-pass does **not** decode or fold: | Bypass | Example | What to do | | ----------------------------------------- | ----------------------------- | ----------------------------------------------------------------------------------------------------------- | | Percent-encoding | `%68%74%74%70://evil.example` | Still needs a dedicated regex if you care about this form | | Unicode NFKC / fullwidth | `ｈｔｔｐ://evil.example` | Not folded here | | Homoglyphs (Cyrillic / Greek look-alikes) | `ѕсam` vs `scam` | Not folded by this pre-pass | | Leetspeak | `h4ck3r` vs `hacker` | Enable leet detection on that [word list](https://getstream.io/moderation/docs/go-golang/configuration/filters/#word-matching) | | Base64 or other encodings | `aHR0cDovL2V2aWw=` | Not decoded | Word lists still match whole words (case-insensitive). Adjacent punctuation is ignored; `house` does not match `lighthouse`. ## Examples ```text Input: see https://evil.example/path Output: see https://evil.example/path Input: http://evil.example Output: http://evil.example Input: <>& missing semicolons Output: <>& missing semicolons Input: h​t​tp://evil.example Output: http://evil.example Input: visit &#115;&#112;&#97;&#109; now Output: visit spam now ``` --- For the most recent version of this documentation, visit [https://getstream.io/moderation/docs/go-golang/guides/text-normalization/](https://getstream.io/moderation/docs/go-golang/guides/text-normalization/).