Winds – An in Depth Tutorial on Making Your First Contribution to Open-Source Software

The team here at Stream enjoys building open-source sample applications to showcase the functionality of our API. Our perspective has always been that it’s better to demonstrate the capabilities of our offerings in a fully functional platform. In this case, leveraging Stream and other great services allowed us to build a podcast and RSS reader, Winds, in months rather than years. In addition, as an open-source project, Winds keeps getting better thanks to contributions from its growing user base (now over 14,000 users and ~5,500 stars!). In this post, we’ll give you a rundown on how Winds - Stream’s most popular open-source sample application – is built. If you’re not familiar with Winds, you can read more about it here. We’ll start with a detailed walkthrough on adding a feature that requires us to touch multiple aspects of the application’s front and backend. By the end of this post, you’ll be ready to add your own features to Winds and contribute to the open-source community! Whether you’re a new coder or a veteran, we are confident you’ll learn something new. 😀 Please note, this tutorial assumes the following:

1. You’re running macOS or understand how to install the various required dependencies on your OS of choice. 🎁
2. You understand JavaScript 🤔
3. You have a basic understanding of React (it’s okay if you don’t, but it helps) 💻
4. You have an understanding of git (we won’t be diving deep, but general knowledge is required). 🔦
5. You’re super stoked to learn how to code against the Winds codebase! 💥

Let’s get started!

System Dependencies 🧙‍

As you may know, system-wide dependencies are required for every application. To ensure that we stay on track, let's only cover installations for macOS.

1. Homebrew

For those of you who are new to coding, Homebrew is an amazing tool for handling installations of system dependencies. In a single command, you can install a coding language of your choice, or use Homebrew’s Cask functionality to install full-blown applications on your machine. If you don’t have Homebrew installed, you can install it with the following command:

Once you’ve got Homebrew all squared away, we can move on to the next step...

2. Node.js

Node.js is heavily used throughout this project – mostly for the API and test suite. With that said, let’s make sure you’re running the latest version of node. At the time of writing this, Node.js is at v10.7.0 (and changing often). If you have Node.js installed, you can check your node version with the following command:

Note: We’ll assume that you are running the latest version of node. If you don’t have Node.js installed, you can do so with one of the following ways:

a) Homebrew

OR

b) NVM (Recommended)

3. MongoDB

MongoDB is our primary datastore for user data, RSS, Podcasts, and much more. We use MongoDB Atlas, a hosted version of MongoDB built and maintained by MongoDB.

Note: The command to start MongoDB is brew services start MongoDB.

4. Redis

Redis is important as it serves as our job queue for processing RSS and Podcast feeds. We also use Redis for some basic caching on items that are not updated (such as interests).

Note: The command to start Redis is simply redis-server. A full list of commands can be found here.

4. Yarn

Yarn is a replacement for npm (node package manager). We recommend yarn over npm as we have found it to be more reliable and an overall better package manager for Node.js dependencies.

Global Yarn Dependencies 🌎

There’s one Node.js dependency that we need to be local, and for that, we’ll use Yarn. The dependency is PM2, a process manager that we’ll talk about in a bit. For now, run the following command to install PM2:

Clone the Repo 💾

Setting Up Third-Party Services 👨‍👨‍👧‍👦

Winds relies on a couple of third-party resources to run. All external services will have API Keys/Secrets and other values that you will need to save for later in the post – I recommend using the Notes app in macOS. In total, it’ll take about 15-20 minutes for you to complete. Note: All services required to run Winds are free up to a certain level (generally production numbers), so no need to worry about fees for now. None of the services we recommend using will require a credit card.

1. Mercury Web Parser (~2 minutes)

2. Stream (~5 minutes)

1. podcast (flat)
2. rss (flat)
3. user (flat)
4. timeline (flat)
5. user_episode (flat)
6. user_article (flat)

3. Algolia (~10 minutes)

4. Sentry (~2 minutes)

Cloning the Repo 📀

To get started with next steps, you’ll need to clone the repository from GitHub. You can use the following command to do so:

Great! Now that you’ve cloned the repo, let’s go ahead and install the required dependencies with yarn.

Winds API

Winds App

The Build

Before we move on, I’d like to take a minute to discuss the front- and back-end structure of the site. With any application, it’s important to understand the architecture and thought process behind it.

Winds Frontend

The front end portion of Winds is pretty straightforward. We used Create React App (CRA) to bootstrap the application and then start the development process. The frontend code can be found here: https://github.com/GetStream/Winds/tree/master/app

Winds Backend

The backend API is slightly more complicated than the frontend. Aside from being powered by Node.js, the backend handles nearly all of the business logic – communicating with third-party services, orchestrating workers for parsing RSS, Podcasts, and Open Graph data, etc. The backend can be viewed here: https://github.com/GetStream/Winds/tree/master/api.

ES6 Standards

Almost all of the code that we use is written in ES6. This allows us to keep our footprint small while maintaining readable code.

API Routes

API Controllers

API Models

API Workers

The workers perform very special tasks that would otherwise be blocking processes. For example, we use dedicated tasks for processing RSS feeds, Podcast feeds, Open Graph Images, and more. Without having dedicated processes for these tasks, our API would quickly come to a halt and users would not receive a response message in a timely manner – the API would likely timeout. Our workers use Bull Queue, a queueing infrastructure for Redis. Basically, our API inserts a call to Redis using the Bull Node.js library, then our workers pick up the job and process it asynchronously. For example, here’s the code from the Podcast.js Controller that adds a podcast after a user adds it to the system (notice how we add a high priority of 1):

From there, the following things happen:

1. The conductor picks up on the task that needs to be processed
2. The file podcast.js is notified it has a job to do (process the incoming job)
3. The database is filled with populated episodes
4. The User is notified that new podcasts are available

CLI Commands

API Tests

Winds ENV 🌪️

There are two places that require a .env (dotenv) file for running the application: /app/.env as well as /api/tests (assuming you are going to be writing tests). You’ll need to create a .env file inside of /app in order for the application to work. Here’s a boilerplate .env file to help you get started:

Note: There are inline notes to help guide you through the setup of your .env file

Running PM2 🏃

PM2 is a process manager and we use it extensively for Winds. It’s an extremely powerful tool and we’re big fans of the project, as well as the maintainers. They are quick to respond if a bug arises, and most importantly, it works very well for what we need to do. Node.js is single threaded by design. This has its ups and downs – it’s extremely fast, but bound to a single I/O operation at a given time. Under the hood, PM2 uses the Node.js cluster module so that the scaled application’s child processes can automatically share server ports. The cluster mode allows networked Node.js applications to be scaled across all CPUs available, without any code modifications. This greatly increases the performance and reliability of the application at hand, depending on the number of CPUs available. I’d recommend learning the commands for PM2 if you are going to be developing on Winds, or if you plan on using PM2 for your own application. In all honesty, the best feature is the watch command that is built-in – it automatically watches for changes and reloads the app when necessary. Here are a few commands that I use daily:

• pm2 start process_dev.json (Starts the processes via commands set in the process_dev.json file)
• pm2 list (Lists all running processes)
• pm2 restart all (Restarts all running processes managed by pm2)
• pm2 log (Tails the logs that the various processes are spitting out)

Note: Use ctrl+c to get out of the tailing logs

Let’s Get Started 👯

You’ve made it this far. Congratulations! All of the dependencies are installed, repo is cloned, your .env is set up… we’re ready to go!

Create a New Branch

Inside of your working directory, create a new branch called “feature”. Here’s the code for it if you need:

Start MongoDB

Start Redis

Start the Winds API & Workers

Let’s Start the Winds UI

Adding a New Feature 🔔

Blank State

1. The Featured on Winds requires that a MongoDB database flag is set to true. For example, it must say “featured: true” on an RSS feed or a Podcast feed.
2. The Discover recommendation feature is powered by our machine learning. Machine learning takes time, as it learns from your interactions with content. The more that you interact with your content, the better.

Note: For this tutorial, it’s not necessary to get these two components up and running. But if you want to experiment, I’d suggest jumping into MongoDB Compass and setting an RSS feed’s “featured” key to the boolean value of true.

Starting to Code

• Facebook Like Buttons
• Bookmark Support
• Dark mode to support macOS Mojave
• Likes (our API already provides support for them)
• General CSS cleanup
• General JavaScript cleanup
• Test coverage for the API and Workers

Note: We’re also hiring a Full-time (contract) developer to join the team to work on Winds. Think you have what it takes? Ping me here with your LinkedIn profile and any examples of coding work that you’ve worked on as the sole contributor.

Final Thoughts 🤔

Winds is the most popular open-source application of its type – and we couldn’t be more excited. Free desktop applications are available for macOS, Linux, and Windows, and a web version is available as well. The application features several pieces of functionality, notably feeds and personalized content recommendations, all of which are powered by Stream, the leader in API based news feeds, activity streams, and personalization as a service. Thank you for sticking around and learning a bit about Winds! We hope to see some PRs from you in the near future! Happy Coding!