Server-side

Last Edit: May 05 2020

Stream's Firehose allows you to subscribe to all feed updates and have them shipped to your own infrastructure via different transports. These transports make easy to extend Stream.

Realtime transports are useful:

  • push notifications using your own infrastructure

  • send emails based user behavior

  • do offline processing

  • data warehousing

SQS

Stream can send all the updates on your feed group to an Amazon SQS queue you own.

The real-time updates are:

  • Published on an SQS queue you own

  • Stored in the same queue

  • Sent in JSON format and Base64 encoded

A list with the following properties for each item in the list:

Name Type Description
feed string Name of the feed this update was published on
deleted list All activities deleted by this update
deleted_foreign_ids list A pair of foreign_id and time of the deleted activities
new list All activities created by this update
published_at date Time of the update in ISO format

Configuration

To configure a SQS queue, go to the dashboard and select the feed group for which you want to configure the SQS realtime transport. If you enable "Realtime Notifications" in the form, you can open the advanced realtime configuration via the "Realtime Configuration" link. Now enable the SQS transport and configure your queues URL, API Key, and API Secret.

If you click the "Test SQS" button we will send a test request to your SQS queue and show any errors with the current configuration if found.

SQS Permissions

Stream needs the right permissions on your SQS queue to be able to send realtime updates to it. If updates are not showing up in your queue add the following permission policy to the queue:


{
    "Version": "2012-10-17",
    "Statement": [
      {
        "Sid": "Stmt1459523779000",
        "Effect": "Allow",
        "Action": [
          "sqs:GetQueueUrl",
          "sqs:SendMessage",
         "sqs:SendMessageBatch",
         "sqs:GetQueueAttributes"
       ],
       "Resource": [
         "arn:aws:sqs:region:acc_id:queue_name"
       ]
     }
   ]
}
                    

Here's an example list of messages read from your SQS queue:


[
  {
    deleted: [],
    deleted_foreign_ids: [],
    new: [
      {
        actor: '1',
        verb: 'tweet',
        object: '1',
        target: null,
        time: '2014-12-15T17:20:37.258',
        foreign_id: null,
        id: 'af781804-847e-11e4-8080-80012fb97b9e',
        tweet: 'Hello world'
      }
    ],
    published_at: '2014-12-15T17:20:37.263518+00:00',
    feed: 'user:2',
    app_id: '123'
  }
],
[
    {
        deleted: ['38f81366-847f-11e4-9c94-0cc47a024be0'],
        deleted_foreign_ids: [['your_foreign_id', '2014-12-15T17:20:37.263518+00:00']],
        new: [],
        published_at: '2014-12-15T17:20:37.263518+00:00',
        feed: 'timeline:1',
        app_id: '123'
    }
];
                    

SQS Best practices and Assumptions

  • Ensure to have the queue size as 256 KB

Stream uses batching for updates and tries to include as many messages as possible to reduce network delays. That's why the maximum queue size (256 KB) is assumed. If your queue size is smaller than this, your messages might be dropped.

  • Setup a deadletter queue for your main queue

SQS has a queue type that can be a redrive policy for a queue, which skips normal validation and can accept everything. This queue could be useful for debugging because it could hold messages that would be dropped otherwise.

Webhooks

Stream can send all the updates on your feed group to a webhook you manage.

Real-time updates are:

  • Sent with a POST request

  • Data is serialized to JSON

  • Single request can contain multiple updates (up to 100 activities per push)

You are responsible for the security of the webhook endpoint. We suggest that you use HTTPS in combination with HTTP simple authentication. (e.g. https://username:password@domain.tld/path/to/webhook/)

Configuration and Verification

Webhook URLs can be configured directly on the dashboard. Navigate to the app overview page and select the feed group for which you want to configure webhooks. Once "Realtime Notifications" are enabled you can open a dropdown by clicking on "Realtime Configuration". The "Test Webhook" button will perform ownership verification (see below) and send a test POST request to the endpoint. Click save if your configuration is complete and new feed updates will be sent to the endpoint in realtime.

To ensure that you are the owner of the endpoint you configure, we perform a verification step on saving a new endpoint. There are two verification methods available: verification via DNS and verification via HTTP. You need to pass one of these two verifications in order to set up the webhook.

Verification Via DNS

Configure a TXT record _getstream for the domain of the webhook with value getstream=your_app_api_key. For example if your webhook endpoint is https://example.com/feeds/ you will need to create a TXT record _getstream.example.com with value getstream=your_app_api_key.

Verification Via HTTP Request

To validate using a GET request, make sure the webhook returns your most recent API Key on a GET request. Make sure that the response body is in plain text (no markup, whitespaces, ...).

Example Data

Please find below an example of the data your webhook will receive:


[
	{
		deleted: [],
      deleted_foreign_ids: [],
		new: [
			{
				actor: '1',
				verb: 'tweet',
				object: '1',
				target: null,
				time: '2014-12-15T17:20:37.258',
				foreign_id: null,
				id: 'af781804-847e-11e4-8080-80012fb97b9e',
				tweet: 'Hello world',
			},
		],
		published_at: '2014-12-15T17:20:37.263518+00:00',
		feed: 'user:2',
		app_id: '123',
	},
	{
		deleted: ['38f81366-847f-11e4-9c94-0cc47a024be0'],
     deleted_foreign_ids: [['your_foreign_id', '2014-12-15T17:20:37.263518+00:00']],
		new: [],
		published_at: '2014-12-15T17:20:37.263518+00:00',
		feed: 'timeline:1',
		app_id: '123',
	},
];