Server-side

Stream’s Firehose allows you to subscribe to all feed updates and have them shipped to your own infrastructure via different transports. These transports make it easy to extend Stream.

Realtime transports are useful for:

• push notifications using your own infrastructure

• send emails based user behavior

• do offline processing

• data warehousing

SQS

New SQS events are created when new activities are added to a feed. Stream can send all the updates on your feed group to an Amazon SQS queue you own.

The real-time updates are:

• Published on an SQS queue you own

• Stored in the same queue

• Sent in JSON format and Base64 encoded

A list with the following properties for each item in the list:

Configuration

To configure a SQS queue, go to the dashboard and select the feed group for which you want to configure the SQS realtime transport. If you enable “Real-time Notifications” in the form, you can open the advanced realtime configuration via the “Realtime Configuration” link. Now enable the SQS transport and configure your queues URL, API Key, and API Secret.

If you click the “Test SQS” button we will send a test request to your SQS queue and show any errors with the current configuration if found.

SQS Permissions

Stream needs the right permissions on your SQS queue to be able to send realtime updates to it. If updates are not showing up in your queue add the following permission policy to the queue:

{
  "Version": "2012-10-17",
  "Statement": [
   {
    "Sid": "Stmt1459523779000",
    "Effect": "Allow",
    "Action": [
     "sqs:GetQueueUrl",
     "sqs:SendMessage",
     "sqs:SendMessageBatch",
     "sqs:GetQueueAttributes"
    ],
    "Resource": [
     "arn:aws:sqs:region:acc_id:queue_name"
    ]
   }
  ]
}

Here’s an example list of messages read from your SQS queue:

[
 {
  deleted: [],
  deleted_foreign_ids: [],
  new: [
   {
    actor: '1',
    verb: 'tweet',
    object: '1',
    target: null,
    time: '2014-12-15T17:20:37.258',
    foreign_id: null,
    id: 'af781804-847e-11e4-8080-80012fb97b9e',
    tweet: 'Hello world'
   }
  ],
  published_at: '2014-12-15T17:20:37.263518+00:00',
  feed: 'user:2',
  app_id: '123'
 }
],
[
  {
    deleted: ['38f81366-847f-11e4-9c94-0cc47a024be0'],
    deleted_foreign_ids: [['your_foreign_id', '2014-12-15T17:20:37.263518+00:00']],
    new: [],
    published_at: '2014-12-15T17:20:37.263518+00:00',
    feed: 'timeline:1',
    app_id: '123'
  }
];

SQS Best practices and Assumptions

• Ensure to have the queue size as 256 KB

Stream uses batching for updates and tries to include as many messages as possible to reduce network delays. That’s why the maximum queue size (256 KB) is assumed. If your queue size is smaller than this, your messages might be dropped.

• Setup a deadletter queue for your main queue

SQS has a queue type that can be a redrive policy for a queue, which skips normal validation and can accept everything. This queue could be useful for debugging because it could hold messages that would be dropped otherwise.

Webhooks

New webhook events are created when new activities are added to a feed. Stream can send these events on your feed group to a webhook you manage. Currently, we send updates for the following actions:

• Add and delete activities

• Add and delete files and images

• Add, update, and delete collections

• Add, update, and delete users

• Add, update, and delete reactions

Real-time updates are:

• Sent with a POST request

• Data is serialized to JSON

• Single request can contain multiple updates (up to 100 activities per push)

You are responsible for the security of the webhook endpoint. We suggest that you use HTTPS in combination with HTTP simple authentication. (e.g. https://username:password@domain.tld/path/to/webhook/ )

Configuration and Verification

Webhook URLs can be configured directly on the dashboard. Navigate to the app overview page and select the feed group for which you want to configure webhooks. Once “Realtime Notifications” are enabled you can open a dropdown by clicking on “Realtime Configuration”. The “Test Webhook” button will perform ownership verification (see below) and send a test POST request to the endpoint. Click save if your configuration is complete and new feed updates will be sent to the endpoint in realtime.

To ensure that you are the owner of the endpoint you configure, we perform a verification step on saving a new endpoint. There are two verification methods available: verification via DNS and verification via HTTP. You need to pass one of these two verifications in order to set up the webhook.

Verification Via DNS

Configure a TXT record _getstream for the domain of the webhook with value getstream=your_app_api_key. For example if your webhook endpoint is https://example.com/feeds/ you will need to create a TXT record _getstream.example.com with value getstream=your_app_api_key.

Verification Via HTTP Request

To validate using a GET request, make sure the webhook returns your most recent API Key on a GET request. Make sure that the response body is in plain text (no markup, whitespaces, …).

Example Data

Please find below an example of the data your webhook will receive:

[
	{
   deleted: [],
   deleted_foreign_ids: [],
	  new: [
			{
				actor: '1',
				verb: 'tweet',
				object: '1',
				target: null,
				time: '2014-12-15T17:20:37.258',
				foreign_id: null,
				id: 'af781804-847e-11e4-8080-80012fb97b9e',
				tweet: 'Hello world',
			},
		],
		published_at: '2014-12-15T17:20:37.263518+00:00',
		feed: 'user:2',
		app_id: '123',
	},
	{
   deleted: ['38f81366-847f-11e4-9c94-0cc47a024be0'],
   deleted_foreign_ids: [['your_foreign_id', '2014-12-15T17:20:37.263518+00:00']],
	  new: [],
		published_at: '2014-12-15T17:20:37.263518+00:00',
		feed: 'timeline:1',
		app_id: '123',
	},
];

App Level Real-time

In addition above feed level real-time, you can configure exactly same way in dashboard feed overview page for reactions, collections, users and files. These events aren’t attached to any feeds so in payload, feed will be empty and there will be a type property. According to type property, generic property will contain different payloads such as added reaction, created user, deleted collection, etc.