•Updated: Mar 4, 2020
•Published: Jul 19, 2018
First Things First
Visualizing & Optimizing API Calls
The first step to triaging an issue is understanding the underlying problem. This can be a daunting task if you’re not using the correct tools to help you visualize what’s going on in your codebase. For Winds, we utilize a combination of tools to gain better insight into what’s happening in our production environment, allowing us to pinpoint latency issues and bugs. Below is a quick rundown on which tools we use and why we love them.
Originally written by Etsy, StatsD is a set of tools that can be used to send, collect, and aggregate custom metrics from your application. The name refers to both the protocol used in the original daemon, as well as a collection of software and services that implement the protocol. A StatsD system requires three components: a client, a server, and a backend. The client is a library that is invoked within our Winds application code to send metrics. These metrics are collected by the StatsD server. The server aggregates these metrics and then sends the aggregated data to a backend at regular intervals. The backends then perform various tasks with our data — for example, Grafana (shown in the Grafana section) is used to view real-time graphs of Winds workers (RSS, Podcasts, Open Graph, etc.) and other important infrastructure metrics. For gathering data, we use a node library called node-statsd. Using the library, we’ve created a helper file to keep our code clean:And we can call the util like this (everywhere a statsd comment is made): It’s important to note that you can use the following metric types with StatsD:
- Counters – they represent a value over time or number of occurrences of a certain event (you can either set or increment/decrement). StatsD will calculate mean/average values, percentiles, etc.)
- Gauges - similar to counters, but instead of value over time it represents one value (latest data point available)
- Timers - like counters but for measuring the time that an operation took instead of how many times it occurred
Optimizing Our MongoDB Database
In Winds, we had nearly all of the proper indexes, however, we failed to recognize one major requirement/constraint in MongoDB. The total size of an index entry must be less than 1024 bytes. That’s not so hard to ensure... right? Well, the thing is, we index URLs to speed up the lookup process, and some of the URLs that enter the Winds API are more than 1024 bytes. This caused all sorts of unexpected errors. After much research, we found a solution – hashed indexes. Hashed indexes maintain entries with hashes of the values of the indexed field. This meant that we could do a lookup on the same URL as before without worrying about its length AND still maintain integrity.
MongoDB automatically computes the hashes when resolving queries using hashed indexes. Applications do not need to compute hashes.
Similar to other databases, MongoDB supports compound indexes, where a single index structure holds references to multiple fields.
Note: You may not create compound indexes that have hashed index type. You will receive an error if you attempt to create a compound index that includes a hashed index field.
Note: The above shows an example of adding a compound index, which is similar to adding a single index. The only difference is that with a single index, you only have one key with a value of 1 or -1.
The value of the field in the index describes the kind of index for that field. For example, a value of 1 specifies an index that orders items in ascending order. A value of -1 specifies an index that orders items in descending order.
One feature that MongoDB provides is the createIndex() operation. What this does is loops through every document in your database and “ensures” that it has the proper index. Many people fail to realize that this is a very tasking activity on the database, taking up precious CPU and memory, slowing down your database. If you make updates to your indexes, you’ll need to run the createIndex() operation; however, we recommend doing this during off-peak hours when you’re database isn’t being hit with tons of requests/second. To enable this feature in Mongoose, have a look at the documentation found here.
Note: If you’d like to read up on the createIndex() operation, MongoDB has extensive documentation on the topic
Being Smart with Queries
Reducing queries to the database is the number one performance enhancement you can make. With New Relic, we found that a user signup route was extremely slow. A single signup API call turned into ~60 requests to the Stream API along with a couple calls to the database to fetch interests. After looking into the issue, we realized that we could do two things to reduce the total number of requests:
- Cache interests in Redis (coming up next), instead of making N number of API calls to the database for every user signup
- Batch API calls to third-party services such as Stream
Another common mistake is doing a simple query N times, instead of single query that reads all required records. When reading the feed from Stream we simply use the MongoDB $in operator to retrieve all of the articles with the id values stored in Stream.
The Explain Query
The $explain operator is extremely handy. As the name may suggest, it returns a document that describes the process and indexes used to return the query. This provides useful insight when attempting to optimize a query (e.g. what indexes to add to speed up the query). This documentation on the $explain operator specifies how you can implement the query.
- Use Redis for:
- Storing popular and frequently queried data to keep database queries down (it’s far more efficient to query memory than have a database do a lookup against disk).
- Caching your data for the amount of time for which it will be valid (e.g. 60 minutes in our case for Winds) and using auto-expiration (this is by default).
- Associating your cache key with your application version number (e.g. interests:v2.0.0). This will force a re-cache when you deploy an update to your application, avoiding potential bugs and cache mismatches.
- Stringifying your data to ensure that it can be stored in Redis (it’s key value only and both must be a string.
Bull is a fantastic queuing system that sits on top of Redis. We use Bull for our worker infrastructure in Winds, and have a couple of queues that we use to process (scrape) data:
- Open Graph
Most importantly, we chose Bull for the following reasons:
- Minimal CPU usage due to a polling-free design
- Robust design based on Redis
- Delayed jobs
- Rate limiter for jobs
- Multiple job types per queue
- Threaded (sandboxed) processing functions
- Automatic recovery from process crashes
We use a queuing mechanism for feeds and open-graph scraping because it separates the processes out so they don’t conflict without API performance. This is generally best practice as you don’t want your API hanging while you’re parsing 10,000 or more feeds every so often. When building an API or app server, it’s important to step back and question whether or not a task is going to get in the way of your response times. Response times should be at most 250ms and even that number is slow. When in doubt, throw it into a queue and process it on a separate thread.
Note: Here’s a quick screenshot of our queue. This shows the number of active RSS, Podcast, and Open Graph jobs in the queue, along with their status. Bull provides all of this data via an API, allowing us to have a better insight into what’s going on behind the scenes. There are also other third-party UIs that you can use for monitoring such as Taskforce and Arena.
Use Dedicated Solutions
Similar to how Bull offloads processing burdens, it can be insanely valuable to use dedicated solutions to speed up your application. For example, we use our own service, Stream, to ensure that users are following the necessary feeds and receiving updates when they are put into the system. Building this from scratch would take months, if not years; however, by using a service, we were able to implement in hours. Another third-party service that we love is Algolia. Algolia powers our search functionality in Winds, making lookups lightning fast (less than 50ms lookup times). While we could build search ourselves with a dedicated endpoint and a fancy MongoDB query, there’s no way that we would be able to scale the search functionality and maintain the same speed Algolia provides, and it would eat up precious API resources while it a user is performing a search. Both these solutions far outperform what you can build in-house using a general purpose database.
We hope you enjoyed this list of basic tips for improving your Node app’s performance. Most real-world performance problems are caused by simple things. This post listed a few common solutions:
- APM tools such as New Relic, StatsD, and Grafana
- Index usage & Query optimization
- Redis & Caching
- Async Tasks
- Using specialized data stores for search and feeds
Looking to learn how MongoDB stacks up against Stream? Have a look at the breakdown here: https://getstream.io/activity-feeds/mongodb/ If you’re building an application that requires newsfeeds or activity feeds, have a look at Stream. We have a 5-minute tutorial that will walk you through the various use-cases that we cover. The API tutorial can be found here. Should you have any questions, please drop them in the comments below. And if you’re interested in following me on Twitter, my handle is @nickparsons. Happy coding!