Firehose

Last Edit: Feb 02 2020

Stream's Firehose allows you to subscribe to all feed updates and have them shipped to your own infrastructure via different transports. This is useful if you want to push notifications using your own infrastructure, do offline processing, or data warehousing.

SQS

Stream can send all the updates on your feed group to an Amazon SQS queue you own. The Firehose makes it easy to extend Stream. It's often used to send push notifications or emails based on feed activity.

The real-time updates are:

  • Published on an SQS queue you own

  • Stored in the same queue

  • Sent in JSON format and Base64 encoded

A list with the following properties for each item in the list:

Name

Type

Description

feed

string

Name of the feed this update was published on

deleted

list

All activities deleted by this update

deleted_foreign_ids

list

A pair of foreign_id and time of the deleted activities

new

list

All activities created by this update

published_at

date

Time of the update in ISO format

Configuration

To configure an SQS queue go the dashboard and select the feed group for which you want to configure the SQS realtime transport. If you enable "Realtime Notifications" in the form you can open the advanced realtime configuration via the "Realtime Configuration" link. Now enable the SQS transport and configure your queues URL, API Key, and API Secret.

If you click the "Test SQS" button we will send a test request to your SQS queue and show any errors with the current configuration if found.

SQS Permissions

Stream needs the right permissions on your SQS queue to be able to send realtime updates to it. If updates are not showing up in your queue add the following permission policy to the queue:


{
    "Version": "2012-10-17",
        "Statement": [
            {
                "Sid": "Stmt1459523779000",
                "Effect": "Allow",
                "Action": [
                    "sqs:GetQueueUrl",
                    "sqs:SendMessage",
                    "sqs:SendMessageBatch",
                    "sqs:GetQueueAttributes"
                ],
                "Resource": [
                    "arn:aws:sqs:region:acc_id:queue_name"
                ]
            }
        ]
}
                    

Here's an example list of messages read from your SQS queue:


[
  {
    deleted: [],
    deleted_foreign_ids: [],
    new: [
      {
        actor: '1',
        verb: 'tweet',
        object: '1',
        target: null,
        time: '2014-12-15T17:20:37.258',
        foreign_id: null,
        id: 'af781804-847e-11e4-8080-80012fb97b9e',
        tweet: 'Hello world'
      }
    ],
    published_at: '2014-12-15T17:20:37.263518+00:00',
    feed: 'user:2',
    app_id: '123'
  }
],
[
    {
        deleted: ['38f81366-847f-11e4-9c94-0cc47a024be0'],
        deleted_foreign_ids: [['your_foreign_id', '2014-12-15T17:20:37.263518+00:00']],
        new: [],
        published_at: '2014-12-15T17:20:37.263518+00:00',
        feed: 'timeline:1',
        app_id: '123'
    }
];
                    

Webhooks

Stream can send all the updates on your feed group to a webhook you manage. The Firehose makes it easy to extend Stream. It's often used to send push notifications or emails based on feed activity.

Real-time updates are:

  • Sent with a POST request

  • Data is serialized to JSON

  • Single request can contain multiple updates (up to 100 activities per push)

You are responsible for the security of the webhook endpoint. We suggest that you use HTTPS in combination with HTTP simple authentication. (e.g. https://username:password@domain.tld/path/to/webhook/)

Configuration and Verification

Webhook URLs can be configured directly on the dashboard. Navigate to the app overview page and select the feed group for which you want to configure webhooks. Once "Realtime Notifications" are enabled you can open a dropdown by clicking on "Realtime Configuration". The "Test Webhook" button will perform ownership verification (see below) and send a test POST request to the endpoint. Click save if your configuration is complete and new feed updates will be sent to the endpoint in realtime.

To ensure that you are the owner of the endpoint you configure, we perform a verification step on saving a new endpoint. There are two verification methods available: verification via DNS and verification via HTTP. You need to pass one of these two verifications in order to set up the webhook.

Verification Via DNS

Configure a TXT record _getstream for the domain of the webhook with value getstream=your_app_api_key. For example if your webhook endpoint is https://example.com/feeds/ you will need to create a TXT record _getstream.example.com with value getstream=your_app_api_key.

Verification Via HTTP Request

To validate using a GET request, make sure the webhook returns your most recent API Key on a GET request. Make sure that the response body is in plain text (no markup, whitespaces, ...).

Example Data

Please find below an example of the data your webhook will receive:


[
	{
		deleted: [],
      deleted_foreign_ids: [],
		new: [
			{
				actor: '1',
				verb: 'tweet',
				object: '1',
				target: null,
				time: '2014-12-15T17:20:37.258',
				foreign_id: null,
				id: 'af781804-847e-11e4-8080-80012fb97b9e',
				tweet: 'Hello world',
			},
		],
		published_at: '2014-12-15T17:20:37.263518+00:00',
		feed: 'user:2',
		app_id: '123',
	},
	{
		deleted: ['38f81366-847f-11e4-9c94-0cc47a024be0'],
     deleted_foreign_ids: [['your_foreign_id', '2014-12-15T17:20:37.263518+00:00']],
		new: [],
		published_at: '2014-12-15T17:20:37.263518+00:00',
		feed: 'timeline:1',
		app_id: '123',
	},
];