How We Serve 268,000+ Chat Events a Second
On Sunday, September 13th, two of our biggest customers' chat peaked two minutes apart. One crossed 35,000 concurrent WebSocket connections at 4:18pm ET, while the other crossed 45,000 at 4:20pm.
The NFL's late-afternoon window had just started. Both customers run prediction markets where people bet on the outcome of live events, and each chat channel maps to one market. Moreover, both platforms were (and still are) growing rapidly. They collectively added 7.5x more monthly active users over the course of just five months.
Supporting that scale at Stream has taken a few years of unglamorous decisions, a lot of Sunday afternoons watching dashboards, and more coffee than my team would like to admit.
Below, I'll walk through how we serve that load, including what it's actually made of, how a chat shard is built to absorb it, and where the design is supposed to fail. (I'm also sharing the two things we still got wrong.)
What "high concurrency" means here
Let's start with the comparison that reframed the whole problem for us.
At that Sunday peak we took in 16.7 message sends a second. You could serve that off a laptop.
In the same second we pushed out 39,336 message.new events. That's 2,355 deliveries for every message sent.
A chat message gets written once and read thousands of times. The number of reads depends on how many people are sitting in the channel, and a televised event increases that number directly.
The obvious reaction is that it's one row and a broadcast, so who cares. Every client that should see the message needs something to decide it should see it, build a payload for it, write to its connection, and cope when that connection is slow or dead. A database read can serve a hundred queries off one cached page. A delivery shares nothing with any other delivery, because each one ends at a different socket. So the cost is messages × audience, and audience is set by the customer's product, not by us.
Sharding the database wouldn't have helped. It saw 17 writes. More API capacity wouldn't have either. That tier finished in milliseconds. The multiplier is entirely in the tier holding the sockets.
Messages aren't even the bulk of it. At that peak we were delivering 268,681 events a second in total, and message.new was about 15% of them. Typing indicators on their own beat messages two to one. Channel updates, member joins, and presence made up most of the rest. One of the two customers is stranger still: nine-tenths of its fanout is custom events defined by its own application. It's using chat channels as a real-time bus, and in event terms it's barely a chat app.
That means a capacity plan counted in messages per second is out by a factor of six before it starts.
And people show up far more than they talk. We handled 753 channel joins a second at that peak, 45x the message rate, and every arrival is itself an event sent to everyone already there. They don't all pile into one room, though. Channels are one per market, dozens run at once, and the biggest game channel took only about a sixth of the joins.
This is a breadth problem. Next, I'll walk through how a chat shard is built to handle that scale.
The shape of a shard
Chat runs as independent shards. A shard is a relational database, an in-memory store for cache and coordination, a stateless HTTP tier, and a stateful WebSocket tier. Every customer app lives on exactly one, and the big ones get a shard to themselves.
That isolation is the first thing that makes a Sunday like this survivable. Their spike is their spike. It can't reach another tenant on a different shard, even when every company in the same industry is watching the same broadcast and spikes in the same minute.
╔═ one shard ════════════════════════════════════════════════╗
║ ┏━━━━━━━━━━━━┓ persist ┏━━━━━━━━━━━━━━━━┓ ║
║ ┃ HTTP API ┃─────────────▶┃ relational ┃ ║
║ ┃ (stateless)┃ ┃ database ┃ ║
║ ┗━━━━━┳━━━━━━┛ ┗━━━━━━━━━━━━━━━━┛ ║
║ │ publish ║
║ ▼ ║
║ ┏━━━━━━━━━━━━━━━━━━━━━━┓ partitioned by ║
║ ┃ event bus ┃ hash(app ^ channel ^ users) ║
║ ┗━━━━━━━━━━┳━━━━━━━━━━━┛ ║
║ ┌───────┼───────┬───────────┐ ║
║ ▼ ▼ ▼ ▼ ║
║ ┏━━━━━━┓┏━━━━━━┓┏━━━━━━┓ ┏━━━━━━┓ ║
║ ┃ WS ┃┃ WS ┃┃ WS ┃ ┃ WS ┃ ten of them, ║
║ ┃ node ┃┃ node ┃┃ node ┃ ┃ node ┃ ~3,500 conns each ║
║ ┗━━━━━━┛┗━━━━━━┛┗━━━━━━┛ ┗━━━━━━┛ ║
╚════════════════════════════════════════════════════════════╝The write path is deliberately short. It validates, saves, acknowledges, and publishes one event. The API tier drops the event on the bus and returns. It never looks up channel membership or counts watchers. That's how 17 writes and a quarter million deliveries happen in the same second without the write path doing any extra work.
The bus decides ordering by how it partitions. Events are hashed on the app, the channel and the users involved, so everything for a given channel lands on the same partition. Per-channel ordering falls out of that for free, without locks, sequence service, or global coordination. And because the hash spreads channels across partitions, the bus scales out by adding partitions rather than by getting cleverer.
Everything interesting after that happens in the WebSocket tier.
Inside a WebSocket node
An HTTP request is a transaction. It arrives, you serve it, and it's gone. A WebSocket connection is a tenant. It sits in memory for hours, it costs a file descriptor and a buffer whether or not anyone ever types, and it has to be somewhere specific, which means the tier holding it is stateful and can't be load-balanced away.
At these events the crowds are mostly lurkers. 35,000 people connected, but only 17 of them sent a message in any given second.
╔═ one WebSocket node ═══════════════════════════════════╗
║ event in ║
║ ▼ ║
║ ┏━━━━━━━━━━━━━━━━━━┓ "does anyone here care?" ║
║ ┃ cuckoo filters ┃ one for channels, one for users║
║ ┗━━━━━━━┳━━━━━━━━━━┛ ── no ──▶ drop, cost ≈ 0 ║
║ ▼ yes ║
║ ┏━━━━━━━━━━━━━━━━━━┓ channel → clients ║
║ ┃ subscription idx ┃ user → clients ║
║ ┗━━━━━━━┳━━━━━━━━━━┛ (in memory, per node) ║
║ ▼ per subscribed client ║
║ ┏━━━━━━━━━━━━━━━━━━┓ critical │ ordered │ mergeable ║
║ ┃ per-client queues┃ sized in TIME, not in events ║
║ ┗━━━━━━━┳━━━━━━━━━━┛ merge keys · score eviction ║
║ ▼ ║
║ socket write ║
╚════════════════════════════════════════════════════════╝Four ideas carry the load.
1. Answer "nobody here cares" for almost nothing
Every node sees events for channels it may have no subscribers for. Checking properly means a map lookup per event per node, and at a quarter of a million events a second that adds up to real work spent discovering you had nothing to do.
So before touching the subscription index, a node asks a Cuckoo filter one question: does anyone connected here care about this channel? There are two of them, one for channel subscriptions and one for user subscriptions, and both answer in constant time from a fraction of the memory a real index would need.
The filter's answer is approximate, but only in one direction. It can say yes when the truth is no, never no when the truth is yes. A wrong yes costs one wasted lookup. A wrong no would silently drop a message, which isn't acceptable. Most events die at this probe at almost no cost.
2. Decide up front what each event is worth
Every built-in event is registered into one of three groups. There's no default, and an unregistered event doesn't ship:
- Critical: health checks, bans, access revocations. Never dropped for being stale.
- Ordered: messages, typing indicators, reactions. Order preserved.
- Mergeable: presence, user and channel updates. Carries a merge key.
Mergeable is the fun one. Say a user flips online, offline, online again inside two seconds while a client's queue is backed up. Sending all three is worse than sending one; the client paints two states it's about to throw away. The merge key here is the user, so each new event overwrites the last one still sitting in the queue, and the client gets a single event carrying the current truth. The queue absorbs the burst by shrinking instead of growing.
That only works when the newest value completely replaces the older one. Presence, yes. A message, obviously not. You can't merge two messages. Getting that call right per event type is most of the design work, and it's why there's no default group. An unclassified event would be handled wrong at exactly the moment everything is under load.
There is one exception. Custom events are defined by the application, so we can't know them in advance, and they fall back to Ordered with a modest rate limit. Remember the customer where nine-tenths of the fanout is custom events? All of it goes down that generic path.
3. Size the queue in time
This is the part I'd steal for another system.
The obvious way to size a per-client buffer is to pick a number of events. A thousand feels fine. Ten thousand feels safer. An unexpected wordload breaks either one.
Instead each queue is sized as how long a client is allowed to fall behind, multiplied by the rate we expect for that class of event. The buffer is about a second and a bit. The rates differ by group: critical events get the most headroom, mergeable the least, because mergeable ones collapse anyway. Multiply the two, clamp the result into a sane range, and that's your capacity.
The nice property is that the question becomes answerable. "How many events should a client buffer?" has no principled answer. "How long should a client be allowed to lag before we start dropping?" does.
4. When it overflows, lose the right thing
Even with all of the above, a client on hotel Wi-Fi during a touchdown will fall behind.
Every event carries a priority. When a client's queue is full, the lowest score is dropped, including the event that just arrived, if it's worth less than everything already in there. A struggling client sheds presence churn and typing noise and keeps its messages. It degrades into a worse chat experience rather than a broken one, and it does that without anyone paging anyone.
That, more than any throughput number, is what makes a quarter of a million events a second survivable.
Where the design says it breaks
Before looking at production, I'll share what each stage scales with.
| Stage | Scales with |
|---|---|
| Persist the write | messages |
| Publish to the bus | messages |
| Filter probe | events seen per node |
| Subscription lookup | events seen per node |
| Queue write | events × audience |
| Socket write | events × audience |
| Connection memory | connections |
events × audience is the only term carrying the multiplier, and all four ideas above target it directly:
- skip with the filter
- classify so cheap things stay cheap
- bound the buffer in time
- drop by score
connections scales with nothing else at all, which is why the WebSocket tier gets sized on connection count and never on request rate. A capacity plan written in requests per second will under-provision the one tier actually holding the load.
Notice what's missing from both rows: the database. That's the part we got wrong.
The two things we still missed
We prepared the wrong bottleneck. Six weeks out we audited every shard with a sports customer on it and found a constraint: storage I/O on the database tier.
Both hot shard databases sat on a machine size whose attached storage tops out at 40,000 IOPS. That ceiling belongs to the machine, not the disk. In a cloud, the disk sits on the network, reached over a link whose speed comes from the instance size. There are two ceilings: 1) what's provisioned on the volume, and 2) what the machine can pull over that link. The lower ceiling applies, and provisioning past it does nothing. One of our shards was already provisioned at exactly that ceiling, so buying more IOPS wouldn't have changed anything.
We increased the machine size instead, and dropped a redundant 136 GB index at the same time. Sunday arrived, and the shard we'd pushed hardest on had its quietest day of the week. It peaked on Monday, driven by a live-commerce tenant with nothing to do with football, and the customer the audit was built around wasn't in its top six apps by volume that day.
On the Wednesday night opener, one market's message-send p99 sat around 300ms; the other's sat at 1,918ms. These were the same platform and the same kind of shard, running the same game. The traces showed 1.6 seconds of a 1.7-second request inside the moderation check. That customer was pointed at an LLM-based moderation service. The faster one ran a local blocklist, at close to zero cost.
We left it alone during the game. Swapping a live customer's moderation provider mid-broadcast, with no fast way back, wasn't a trade we were willing to make. We changed it Friday. p99 dropped to 493ms and stayed there, steady to within two milliseconds, through Sunday's tripling of connections.
The 90 minute gap
The architecture can't fix this part on its own. The API tier scales reactively. CPU over a threshold for three minutes adds instances, then a cooldown before it scales again. From a normal baseline, our slowest shard takes an hour and a half to reach its maximum instance count. Game-day traffic arrives in about five minutes.
We wrote a ticket to pre-scale before kickoff, then cancelled it. Putting a new scheduled-scaling mechanism into production for the first time during the biggest week of the year adds risk rather than removing it. Instead, we covered the gap with people, including a published rota, named engineers on named windows, four fixed checks per slot, and a rule to post an update every check, even when nothing's wrong.
It earned its place twice that Sunday. The API tier's maximum instance count was five when the day started. We raised it to 10, then to 16, after confirming there was room left in the database connection pool. Raising the ceiling without checking that first would have just moved the pressure somewhere worse. Autoscaling worked correctly the entire time; it had a ceiling set for a workload that no longer existed, and nothing alerts on a maximum that's set too low. You find out when you hit it, during the event.
What happened
Zero 5xx on the tracked endpoints at the Sunday peak, for both customers. Channel creation and channel queries came in under a quarter of a second. Message send at 493 ms, better than the week before under three times the connections. There were no incidents. Month over month, one of them added 107% more monthly actives and nearly 5x the peak concurrent connections.
Autoscaling would have needed 90 minutes to do what a person did in 90 seconds. What actually held was the rest of the design. The write path stays short, the filters drop most traffic before it costs anything, the queues are bounded in time, and anything that overflows drops the events lowest in priority first. 17 writes a second becomes a quarter million deliveries without the system falling over.
The rest of it was the unglamorous half. A storage ceiling turned out to be attached to a machine size. A third-party call was sitting inside our own request, adding a second and a half nobody had measured. An instance limit somebody set years ago was still there, quietly wrong for a workload that no longer existed.
None of it was in the model going in (which is probably worth another cup of coffee before next season).