Text Normalization
Before any engine runs — word and regex filters, domain lists, AI text, LLM rules, or the review queue — Stream decodes the text to the form a browser would render, then strips invisible characters. Engines therefore match the decoded string, not the raw payload.
This is always on for Chat messages, Feeds activities and comments, and the Check API. The stored message body is left unchanged; only the copy sent to moderation is rewritten.
Why this exists
A common bypass is to entity-encode a blocked word or URL so a plain rule never sees it, while the client still renders the real text:
visit spam now
see https://evil.example/pathThose used to miss a word list containing spam and a regex like (?i)https?://. They now decode to visit spam now and see https://evil.example/path, and the same plain rules match.
What decoding covers
HTML character references are resolved, including the forms browsers accept without a trailing semicolon.
| Form | Example input | After decode |
|---|---|---|
| Decimal | http |
http |
| Hex | http |
http |
| Hex, uppercase prefix | h |
h |
| Decimal, no semicolon | http |
http |
| Hex, no semicolon | http |
http |
| Named | https://example.com |
https://example.com |
| Named, no semicolon | https&colon//example.com |
https://example.com |
| Standard named | a < b & c |
a < b & c |
| Nested amp-encoding | &#104; |
h |
| Triple-encoded amp | &amp;#104; |
h |
Nested encoding is unwound for several passes (&#104; → h → h), which covers the usual double- and triple-encoded chains.
Named entities are matched case-insensitively when the lowercase HTML name exists (&COLON; → :, / → /). A name that is itself a distinct HTML entity keeps that code point (∷ is ∷, not :). Unknown names such as &zzzxnotanentity; are left as-is.
Ampersands that are not part of an entity are left alone (A & B).
Zero-width and invisible characters
After decode, these code points are removed (not replaced with a space). Encoded forms such as ​ or ​ are decoded first, then stripped, so h​t​t​p and a literal h + U+200B + ttp both become http.
| Category | Code points |
|---|---|
| Zero-width | U+200B (ZWSP), U+200C (ZWNJ), U+200D (ZWJ), U+2060 (word joiner), U+FEFF (BOM / ZWNBSP) |
| Soft hyphen / joiners | U+00AD, U+034F (CGJ), U+180E (Mongolian vowel separator) |
| Bidirectional marks | U+200E, U+200F, U+061C |
| Bidirectional embeddings / overrides / isolates | U+202A–U+202E, U+2066–U+2069 |
| Invisible operators | U+2061–U+2064 |
| Line / paragraph separators | U+2028, U+2029 |
| Interlinear annotation | U+FFF9–U+FFFB |
What this means for your rules
You can write filters against the rendered text.
- A word list containing
spammatchesvisit spam now. - A regex
(?i)https?://matches an entity-encoded URL. - A domain list matches the host after decode (and after the usual host canonicalization: lowercase, IDN / Punycode, fullwidth dots).
If you added encoding-aware regexes to catch &#\d+; / &# chains, those are no longer required for this bypass. Prefer a plain URL regex or a domain list. Keeping the encoding regex is only useful if you still want to flag the presence of entity syntax itself.
Review-queue items store the normalized text the engines evaluated, so dashboard review and webhooks show https://evil.example/path, not the encoded source. The Chat / Feeds message the user sent is stored as they typed it.
What is not covered
This pre-pass does not decode or fold:
| Bypass | Example | What to do |
|---|---|---|
| Percent-encoding | %68%74%74%70://evil.example |
Still needs a dedicated regex if you care about this form |
| Unicode NFKC / fullwidth | ｈｔｔｐ://evil.example |
Not folded here |
| Homoglyphs (Cyrillic / Greek look-alikes) | ѕсam vs scam |
Not folded by this pre-pass |
| Leetspeak | h4ck3r vs hacker |
Enable leet detection on that word list |
| Base64 or other encodings | aHR0cDovL2V2aWw= |
Not decoded |
Word lists still match whole words (case-insensitive). Adjacent punctuation is ignored; house does not match lighthouse.
Examples
Input: see https://evil.example/path
Output: see https://evil.example/path
Input: http://evil.example
Output: http://evil.example
Input: <>& missing semicolons
Output: <>& missing semicolons
Input: h​t​tp://evil.example
Output: http://evil.example
Input: visit &#115;&#112;&#97;&#109; now
Output: visit spam now