Covers July-September 2026.

Harm Trends Report: Q3 2026

Every quarter, Stream reviews anonymized moderation activity across 27 apps on 22 platforms and nearly 4 million harm detections, lines it up with the public research T&S teams follow, and lays out what trust and safety teams should change in response. This edition covers July through September 2026.

01

AI-Generated Sexual Imagery Already Passed Last Year's Total

Public data

Volume: The IWF assessed 6,310 AI-generated images as child sexual abuse material in the first half of 2026, 40% more than its 2025 total of 4,512 (IWF via Free Press Journal).

Who is targeted: Girls appeared in 98% of those images where age and gender were recorded.

Manipulated real photos: Children made 420 reports to the IWF's Report Remove service in 2026 about images they believe were faked to look explicit, already above the 397 made in all of 2025.

How people find the tools: ISD tracked 5.7 million social media referrals to nudify sites from December 2025 to March 2026. YouTube sent nearly 30% and X about 20% (ISD, Analytics Insight).

Across Stream

+402%

On the same 17 apps, sexual content and nudity detections on images and video rose from 87,097 in Q2 to 437,440 in Q3 (+402%).

Sexual content and nudity detections on images and video+402%
Q287,097Q3437,440
What it means

Image moderation that only covers public posts misses where this content moves. Profile photos, avatars, and media sent in DMs need the same checks.

02

Financial Sextortion Runs Through Ordinary DMs

Public data

Volume: NCMEC received more than 50,000 reports of financially motivated sextortion in 2025, an average of 137 per day, up from more than 36,000 in 2024 (NCMEC).

Who is targeted: In a Thorn survey of 1,200 people aged 13-20, 1 in 5 said they had experienced sextortion, and 1 in 6 victims were 12 or younger the first time (Thorn, June 2025).

Where it happens: 81% of victims said the threats happened only online (Thorn).

Pattern: The conversation starts on a normal-looking surface (a game, a dating match, a community chat), moves to private messages, and then to a payment demand.

Across Stream

+266%

Stream's labels don't isolate sextortion, so we track the signals around it. On the same 17 apps, child-safety detections (underage users, sexualization of minors) rose 63%, from 4,478 to 7,281, and rose on 10 of the 11 apps that use them. Attempts to move a conversation off-platform rose 266%, from 34,470 to 126,254.

Attempts to move a conversation off-platform+266%
Q234,470Q3126,254
Child-safety detections+63%
Q24,478Q37,281
What it means

Single-message keyword filters miss this. Detection has to read the conversation, including requests to move to another app.

03

Scams Start in Feeds and Chat, Then Move Off-Platform

Public data

Volume: Americans reported losing $15.9 billion to fraud in 2025 across about 3 million reports (FTC, March 25, 2026).

Social media's share: Americans reported losing $2.1 billion to scams that started on social media in 2025, about eight times the 2020 figure (FTC, PPC Land).

Biggest categories: Investment scams accounted for $1.1 billion. Romance scams accounted for $298 million, and nearly 60% of romance scam losses began on social media.

Most reported: Shopping scams. More than 40% of victims said it began with an ad for a product they ordered.

Across Stream

+219%

On the same 17 apps, scam, fraud, and off-platform detections rose 219%, from 53,858 to 171,622, and increased on 11 of 15 apps. Flags labeled as scams alone rose 118%. Marketplaces and live shopping produced 59% of all Q3 scam and off-platform detections (134,566 of 227,928).

Scam, fraud, and off-platform detections+219%
Q253,858Q3171,622
What it means

Marketplaces, dating apps, and trading communities are now the first touchpoint for fraud. Blocking off-platform payment and contact requests closes the most common exit.

04

Automation Now Finds Most Violations Before Users Report Them

Public data

Volume: In Reddit's July-December 2025 transparency report, 86.4% of post and comment reports that reached admin review were surfaced by automated systems, and 99% for chat (Reddit, r/RedditSafety, April 2026).

Spam share: Spam made up 54% of Reddit admin removals.

Hate and harassment: Automated enforcement actions against hate and harassment rose 200%.

Volunteer moderators: Moderators removed more than 81 million posts and comments, 68.6% of them with automated tools.

Across Stream

+85%

AI moderation took 5.1 million actions in Q3 (flag, remove, or shadow block), up 85% from Q2. Moderators reviewed 9.5% of items that reached a review queue, up from 3.8% in Q2. Livestream moderation flagged 31,300 live video and audio sessions in Q3, its first quarter with activity in this group.

Moderators reviewed (% of items that reached a review queue)
Q23.8%Q39.5%
What it means

The review queue should hold the gray-area cases. If moderators spend their day clearing spam, the automation thresholds need adjusting.

05

Regulators Are Fining Platforms for Missing Basic Controls

Public data

Volume: The EU fined Temu €200 million under the Digital Services Act on May 28, 2026, its largest DSA penalty so far, after a €120 million fine against X in December 2025 (Slaughter and May).

Fine: On May 26, 2026, Ofcom fined Youngtek Solutions £600,000: £500,000 for not using highly effective age assurance and £100,000 for missing the deadline on a statutory information request (GBlock).

Open investigation: Ofcom is investigating X over reports that an AI chatbot was used to create and share sexual deepfakes, including of children (Burges Salmon).

2026 priorities: Ofcom's stated focus areas include child sexual abuse and grooming, effective age checks, removal of terror and illegal hate content, and safer experiences for women and girls.

Across Stream

+42%

On the same 17 apps, underage-user detections rose 42%, from 3,433 in Q2 to 4,891 in Q3, and child-safety detections rose on 10 of the 11 apps that use them. Platforms are flagging more users who look under age before a regulator asks.

Underage-user detections+42%
Q23,433Q34,891
What it means

The fines so far are for basics: age checks and answering the regulator on time. Teams need records of what was flagged, what was done, and when, ready before anyone asks.

What to Change This Quarter

ChangeWhereAddresses
Run image and video checks on profile photos, avatars, and DM media, not only public postsWhereImage/video moderationAddressesAI sexual imagery, nudify output
Detect grooming and sextortion across the conversation, not per messageWhere1:1 and group chatAddressesFinancial sextortion
Flag or block requests to move to another app or pay off-platformWhereChat, comments, listingsAddressesScams, sextortion
Auto-action high-confidence spam and scams; send gray-area cases to reviewWhereRules and thresholdsAddressesQueue overload
Keep an audit log of every flag, action, reviewer, and timestampWhereModeration dashboardAddressesRegulatory requests

FAQ

Every quarter. Each edition covers the previous three months and compares them with the quarter before.

Thanks, you're subscribed