Javascript

Server-side

LAST EDIT Oct 01 2024

Stream's Firehose allows you to subscribe to all feed updates and have them shipped to your own infrastructure via different transports. These transports make it easy to extend Stream.

Realtime transports are useful for:

  • push notifications using your own infrastructure

  • send emails based user behavior

  • do offline processing

  • data warehousing

SQS

Copied!

New SQS events are created when new activities are added to a feed. Stream can send all the updates on your feed group to an Amazon SQS queue you own.

The real-time updates are:

  • Published on an SQS queue you own

  • Stored in the same queue

  • Sent in JSON format and Base64 encoded

A list with the following properties for each item in the list:

Name

Type

Description

feed

string

Name of the feed this update was published on

deleted

list

All activities deleted by this update

deleted_foreign_ids

list

A pair of foreign_id and time of the deleted activities

new

list

All activities created by this update

published_at

date

Time of the update in ISO format

type

string

Type of the event (available if feed property is empty)

generic

object

Generic payload according to the type of the event. For example, reaction as in API response if event type is reaction.new (available if type exists)

Configuration

Copied!

To configure a SQS queue, go to the dashboard and select the feed group for which you want to configure the SQS realtime transport. If you enable "Real-time Notifications" in the form, you can open the advanced realtime configuration via the "Realtime Configuration" link. Now enable the SQS transport and configure your queues URL, API Key, and API Secret.

If you click the "Test SQS" button we will send a test request to your SQS queue and show any errors with the current configuration if found.

SQS Permissions

Copied!

Stream needs the right permissions on your SQS queue to be able to send realtime updates to it. If updates are not showing up in your queue add the following permission policy to the queue:

Here's an example list of messages read from your SQS queue:

SQS Best practices and Assumptions

Copied!
  • Ensure to have the queue size as 256 KB

Stream uses batching for updates and tries to include as many messages as possible to reduce network delays. That's why the maximum queue size (256 KB) is assumed. If your queue size is smaller than this, your messages might be dropped.

  • Setup a deadletter queue for your main queue

SQS has a queue type that can be a redrive policy for a queue, which skips normal validation and can accept everything. This queue could be useful for debugging because it could hold messages that would be dropped otherwise.

Webhooks

Copied!

New webhook events are created when new activities are added to a feed. Stream can send these events on your feed group to a webhook you manage. Currently, we send updates for the following actions:

  • Add and delete activities

  • Add and delete files and images

  • Add, update, and delete collections

  • Add, update, and delete users

  • Add, update, and delete reactions

Real-time updates are:

  • Sent with a POST request

  • Data is serialized to JSON

  • Single request can contain multiple updates (up to 100 activities per push)

You are responsible for the security of the webhook endpoint. We suggest that you use HTTPS in combination with HTTP simple authentication. (e.g. https://username:password@domain.tld/path/to/webhook/)

Configuration and Verification

Copied!

Webhook URLs can be configured directly on the dashboard. Navigate to the app overview page and select the feed group for which you want to configure webhooks. Once "Realtime Notifications" are enabled you can open a dropdown by clicking on "Realtime Configuration". The "Test Webhook" button will perform ownership verification (see below) and send a test POST request to the endpoint. Click save if your configuration is complete and new feed updates will be sent to the endpoint in realtime.

To ensure that you are the owner of the endpoint you configure, we perform a verification step on saving a new endpoint. There are two verification methods available: verification via DNS and verification via HTTP. You need to pass one of these two verifications in order to set up the webhook.

Verification Via DNS

Copied!

Configure a TXT record _getstream for the domain of the webhook with value getstream=your_app_api_key. For example if your webhook endpoint is https://example.com/feeds/ you will need to create a TXT record _getstream.example.com with value getstream=your_app_api_key.

Verification Via HTTP Request

Copied!

To validate using a GET request, make sure the webhook returns your most recent API Key on a GET request. Make sure that the response body is in plain text (no markup, whitespaces, ...).

Example Data

Copied!

Please find below an example of the data your webhook will receive:

Activities are NOT included for follow, follow many and unfollow events.

App Level Real-time

Copied!

In addition above feed level real-time, you can configure exactly same way in dashboard feed overview page for reactions, collections, users and files. These events aren't attached to any feeds so in payload, feed will be empty and there will be a type property. According to type property, generic property will contain different payloads such as added reaction, created user, deleted collection, etc.