Using the Stream Real-Time Firehose with AWS SQS, Lambda, and SNS

Stream enables you to listen to fee changes in near real-time using SQS, webhooks or websockets. In this tutorial, we will discuss how to use AWS SQS & Lambda to respond to feed updates. While websockets are the preferred method to listen to changes, SQS notifications have a special spot in the feed infrastructure. Stream really shines in its ability to provide real-time because you can interact with the results that you receive. For example, you can listen to for changes to the feed and respond with, for example, an email to a user enticing them to pull the trigger on a purchase, send an SMS message based on an action taken within your application, among hundreds of other scenarios. The goal of this post is for you to walk away with a strong understanding of AWS SQS, Lambda, and of course, Stream.

Note: To ensure that you can follow along from start to finish, please create an account or make sure that you have the necessary permissions to use the services mentioned above – we won’t be discussing permissions within AWS.

Setting up SQS

Setting up Stream for Use with SQS

Before we get started, we’ll need to create a user for Stream. I prefer to create a separate user so that I can change permissions; however, you’re free to use an existing key and secret if you’d like (just skip the first part of this section if that is the case).

Create a new IAM User & Role

Hook It up to Stream

Now, let’s go ahead and drop our API Key and Secret into the Stream dashboard so that Stream backend knows where to route outbound messages. Steps:

1. Go to the Stream dashboard
2. Create a flat feed with the feed name of “user” (be sure to enable notifications)
3. Click on the “user” feed
4. Enable SQS notifications by clicking the button labeled “Active”
5. Fill out the SQS URL, AWS Key, and AWS Secret
6. Click the save button in the top right-hand corner

Configuring Lambda

AWS Lambda lets will allow us to run code without provisioning or managing servers. This service is going to come in handy for decoding the base64 encoded string that is sent to SQS as the payload. Once the message is decoded, you can do anything – send an SMS, email, etc. Let’s kick things off by setting up Lambda. First things first, you’ll want to head over to the Lambda section of AWS (this can easily be found with the search bar). If you’re a new user, go through the short-2 minute tutorial. If you’ve done this before, jump right in. The first step will be to create a new Lambda function as shown in the screenshot below. To copy my configuration, you’ll want to do the following:

• Select the “Author from scratch” box
• Name your function “STREAM”
• Update the Node.js runtime to Node.js v8.10 (outdated, but it is the latest version that is available on Lambda)
• Specify your “Role” name as “STREAM”
• Include “SQS Poller” permissions

1. Under the “Designer” section, click “SQS” in the scrollable section
2. In the “Configure triggers” section, search for your SQS queue (in our case, it will be named “STREAM”)
3. Keep the batch size at 10
4. Check “Enable trigger”
5. Finally, click on “Add”

Sending SQS Messages to Lambda

Testing Lambda Functionality 🐑

Piping the Payload to AWS SNS

1. Click “Create topic”
2. Set your “Topic name” and “Display name” to STREAM and then click “Create topic”
3. Copy your “Topic ARN” as you’ll need that in a bit

Copy and Paste the Following Commands

Note: For those who are not familiar with command line, the command above creates a new file  (index.js) and installs all of the node modules that we’ll need. Follow the prompts in the terminal to finish up.

Modify Your package.json File

Open package.json and change scripts section to this:

Open .env and Modify Your Keys

Inside of the directory you generated, there should be a .env file and paste your AWS credentials, role ARN (which you’ve copied before) and fill rest of the parameters to match your configuration in AWS. I recommend that you specify the following settings:

Update Your index.js File

Update your index.js file with the following code:

Note: We’re still logging out the contents that Stream sends as an example. And most importantly, deploy your code!

Fix the SQS Connection

Associate Lambda with SNS Topic

Remember the ARN from the topic we created? Now is the time to use it. Open your .env file and create a new environment variable called AWS_SNS_TOPIC_ARN and set the value to your generated SNS Topic ARN.

Create Your SNS Subscription

Upload the Final Code

The second to last part of this SNS setup is uploading some updated code to your Lambda function. Simply update your index.js file with the code below and use the yarn deploy command to push it to AWS.

All Done!

Congratulations! You’ve set up an end to end test using Stream, AWS SQS, Lambda, and SNS to send an SMS message when an event happens to a feed (even if you are the one pressing the “Test SQS” button 😉). I’ve hosted the repo on GitHub, so feel free to clone it if you run into issues: https://github.com/astrotars/stream-sqs-lambda-sns I’m also available to answer any questions in the comments below. Happy coding! 👋