Go 1.11 Rocket Tutorial

This tutorial combines two of my favorite things, the Go programming language and images of SpaceX rocket launches.

With Go rapidly picking up adoption in the developer community, its becoming one of the leading languages for building backend systems. Go’s performance is similar to Java and C++, yet it’s almost as easy to write as Python. Since its inception in 2009 by Google, Go has been adopted by Netflix, Stripe, Twitch, Digital Ocean and many others. Here at Stream we use Go & RocksDB to power news feeds for over 300 million end users.

In this tutorial, you’ll build a highly concurrent web server which reads SpaceX images from the Unsplash API and applies seam carving on those images. The pace is high and assumes you already know how to program. You’ll learn the basics of Go and we’ll also cover more advanced topics such as Goroutines and Channels.

Part 1 - Setup and Seam Carving

Step 1 - Install Go 1.11

Homebrew on macOS is the easiest way to install Go:

bash

brew install go

Alternatively head over to the official Go download page and follow the install instructions for your system.

Be sure that you have installed Go 1.11 or later before continuing this guide. Once installed, check your version with

bash

go version

Step 2 - A Simple Request Handler

Go v1.11 simplified the initial setup of a Go project. Older versions of Go required the GOPATH environment variable. With v1.11 you can follow the three steps below to get your server up and running:

A. Create a directory

Create an empty directory called rockets:

bash

mkdir rockets
cd rockets

B. Go mod init

Run the following command to start a new Go module:

bash

go mod init github.com/GetStream/rockets-go-tutorial

This will create a file called go.mod in your directory. This file will contain your dependencies. It is similar to package.json in Node, or requirements.txt in Python. (Go doesn’t have a package repository

C. main.go

Create a file called main.go in the rockets directory with the following content. (The full path is rockets/main.go):

package main
import (
    "bytes"
    "fmt"
    "io"
    "log"
    "net/http"
    "github.com/esimov/caire"
    "github.com/pkg/errors"
)
const (
    IMAGE_URL string = "https://bit.ly/2QGPDkr"
)
func ContentAwareResize(url string) ([]byte, error) {
    fmt.Printf("Download starting for url %s\n", url)

You can run it like this:

bash

go run main.go

After a moment of installing dependencies, you’ll be able to visit http://localhost:3000/occupymars

Click the “Resize using Seam Carving” link to start the resizing. The computation is quite heavy and may take a bit, so give it a few seconds.

Voila, one resized rocket picture:

Step 3 - Go’s Syntax

The first thing you’ll notice in the above example is that Go is very similar to other programming languages. Let’s quickly review the code in main.go:

A. Go is a statically typed language

func ContentAwareResize(url string) ([]byte, error) {
...
}

The function takes a string as input and returns a slice of bytes and an error.

B. Go infers types

shrinkFactor := 30

Is short for:

var shrinkFactor int = 30

Here’s the syntax for the most commonly used types in Go:

message := "hello world"
friends := []string{"john", "amy"}
friends = append(friends, "jack")
populationByCity := map[string]int{"Boulder": 108090, "Amsterdam": 821752}
populationByCity["Palo Alto"] = 67024

LearnXinY is a great resource to quickly check the Go syntax.

C. Types are also inferred when calling functions

resized, err := ContentAwareResize(url)

Note how the function returns two elements which we use to initialize these new variables: resized and err.

D. Go & Unused variables

One of the quirks of Go is that unlike most other languages it will throw a syntax error if you leave a variable or import unused. If you add the following code after the shrinkFactor on line 29:

oldvariable := 20

And run main.go you’ll get the following error:
./main.go:30:12: oldvariable declared and not used
Most editors will help you clean up unused code, so in practice this is not that annoying. If you want to ignore one of the variables returned by a function you can do it like this:

_, err := http.Get(url)

The _ simply means discard the value.

E. Interfaces

On line 36 the p.Process function converts the image in response.Body and stores it in the converted variable. What’s interesting is the function definition of p.Process:

func (p *Processor) Process(r io.Reader, w io.Writer) error {
    …
}

This function takes an io.Reader as the first argument and an io.Writer as the second. io.Reader and io.Writer are interfaces. Any type that implements the methods required by these interfaces can be passed to the function.

Note: One interesting fact about interfaces in Go is that types implement them implicitly. There is no explicit declaration of intent, no "implements" keyword. A type implements an interface by applying its methods.

(Fun read: This blogpost about Streaming IO in Go gives more details about the io.Reader and io.Writer interfaces)

F. Imports

import (
    "bytes"
    "fmt"
    "io"
    "log"
    "net/http"
    "github.com/esimov/caire"
    "github.com/pkg/errors"
)

The import syntax is easy to understand. Let’s refactor the ContentAwareResize function into a separate package to clean up our code.

Step 4 - Refactoring to a Package

A. Create a directory with the name “seam”:

bash

mkdir seam

Your full path will look like rockets/seam.
Note: If you’re wondering why the folder is called seam it’s because this type of content aware image resizing is also typically called seam carving.

B. In the seam directory, create a file called “seam.go” with the following content:

package seam
import (
    "bytes"
    "fmt"
    "net/http"
    "github.com/esimov/caire"
    "github.com/pkg/errors"
)
func ContentAwareResize(url string) ([]byte, error) {
    fmt.Printf("Download starting for url %s \n", url)
    response, err := http.Get(url)
    if err != nil {
        return nil, errors.Wrap(err, "Failed to read the image")
    }
    defer response.Body.Close()

C. Open main.go and make the following changes:

Remove the import for "github.com/esimov/caire"
Remove the import for "github.com/pkg/errors"
Add the import for "github.com/GetStream/rockets-go-tutorial/seam"
Update the function call in main.go from ContentAwareResize to seam.ContentAwareResize (Line 50)
Remove the ContentAwareResize function from main.go (Line 18-43)

To clarify, before the changes the function call looked like this:

resized, err := ContentAwareResize(url)

After the changes you’re calling the method defined in the seam module like this:

resized, err := seam.ContentAwareResize(url)

D. See if it worked:

Start the server and see if it still works!

bash

go run main.go

If everything went according to plan you should still be able to access at http://localhost:3000/occupymars
Note: Packages give you a nice way to structure your code. Capitalized functions are public and lowerCase functions are private.
If something went wrong with these steps you can also clone the Github repo for a working version:

Part 2 - Unsplash API, Channels and Concurrency

Step 5 - APIs, Structs and JSON

You’re already well on your way to learning Go. Easy right? Not bad for a language that’s 20-40 times faster than Python!
For this second example we’ll learn how to use read data from an API. We’ll use the excellent Unsplash photography API to search for more rocket pictures.

A. Create a folder called unsplash:

bash

mkdir unsplash

The full path is rockets/unsplash.

B. Inside the unsplash directory create a file called unsplash.go with the following content:

package unsplash
import (
    "encoding/json"
    "fmt"
    "net/http"
    "github.com/pkg/errors"
)
type APIResponse struct {
    Total      int

C. Update rockets/main.go to match this content:

https://raw.githubusercontent.com/GetStream/rockets-go-tutorial/master/main.go

D. Create a file called rockets/spacex.html:

The content should match this:

jinja

<html>
<body>
<ol>
    {% for result in response.Results|slice:":8" %}
    <li>
        <img src="{{ result.URLs.small }}" />
        {% if result.Resized %}
            <img src="data:image/png;base64, {{ result.Resized }} " />
        {% endif %}
    </li>
    {% endfor %}
</ol>
</body>
</html>

E. Restart your Go server:

bash

go run main.go

Assuming nothing went wrong you can now open up this URL
http://localhost:3000/spacex
You should see the latest images tagged with SpaceX on Unsplash:

Wicked!

Step 6 - Understand Those API Calls

Let’s open up unsplash.go and have a look at how the code works:

A. Making the request:

Go’s builtin HTTP library is pretty functional, here’s the GET request to the Unsplash API

resp, err := http.Get(url)

B. Structs and methods:

Go has objects but it’s different from traditional object oriented languages because it doesn’t support inheritance. Go relies on composition instead of inheritance to help you structure your code. Structs in Go are the closest you’ll come to classes in more object oriented languages. Let’s see how we’re using Structs to talk to the Unsplash API in unsplash.go:

type APIClient struct {
    // note how the lowercase accessToken is private
    accessToken string
}
func NewAPIClient(token string) APIClient {
    return APIClient{token}
}
func (c *APIClient) Search(query string) (*APIResponse, error) {
    ...
    return &response, nil
}

The NewAPIClient creates a new APIClient struct. The Search method enables you to write code like:

client := NewAPIClient("accessToken")
response, err := client.Search(query)

This blogpost explains the concept of composition nicely.

C. Parsing JSON:

Since Go is a statically typed language you’ll want to parse the JSON into a Struct. If you’re coming from a dynamic language this will be a little bit confusing at first. You’ll get the hang of it quickly. The Unsplash API returns the following JSON:

json

{
  "total": 133,
  "total_pages": 7,
  "results": [
    {
      "id": "eOLpJytrbsQ",
      "created_at": "2014-11-18T14:35:36-05:00",
      "urls": {
        "raw": "https://images.unsplash.com/photo-1416339306562-f3d12fefd36f",
        "full": "https://hd.unsplash.com/photo-1416339306562-f3d12fefd36f"
      },
    }
  ]
}

There are more fields in the JSON but I simplified it a bit for the example. Next we’ll want to parse the JSON into the following structs:

type APIResponse struct {
    Total      int

The struct definition specifies the mapping between the field name in the JSON and the struct:

TotalPages int

This means that when decoding JSON it will take the value from the total_page and set it to the TotalPages property.
To decode the JSON we use the following code:

response := APIResponse{}
err = json.NewDecoder(resp.Body).Decode(&response)
if err != nil {
    return nil, errors.Wrap(err, "failed to parse JSON")
}

We create a new APIResponse struct and parse a pointer to this object in the Decode function.
The next section will discuss pointers in more detail.

D. The defer statement

One of the fairly unique concepts of Go is the defer statement. In unsplash.go line 41 you see the following defer statement:

defer resp.Body.Close()

Deferred function calls are pushed onto a stack. When a function returns, its deferred calls are executed in last-in-first-out order. So after the Search function finishes the defer statement will trigger and close the request body.

Step 7 - Pointers

Throughout this tutorial you probably spotted the and & symbols. These two are used in Go to work with pointer variables. You can think of pointer variables as references to values. Every type in Go can have its pointer counterpart; you can easily tell if a variable is a pointer or not since its type will start with the symbol.

var a int // a is an integer variable
var b *int // b is a pointer to an integer

The & and also work as operators: & returns a pointer to its value and returns the value a pointer refers to.

A. Basic Pointer operations:

If you haven’t used pointers before this concept can be a little bit confusing at first. This little snippet clarifies how pointer operations work:

a := "Go and Rockets!"
// the & operator gives you the pointer for a variable, Go infers the type of pointer (*string)
pointer := &a
fmt.Println(pointer)
// 0x1040c128
// the * operator gives you access to the value
fmt.Println(*pointer)
// Go and Rockets!

You can run the above example in the Go Playground

B. Pointers & Functions:

The example below shows how functions can accept both pointers and regular types:

type Post struct {
    Upvotes int
    Title   string
}
func IncrementInPlace(p *Post) {
    p.Upvotes++
}
func Increment(p Post) Post {
    p.Upvotes++
    return p
}
func main() {
    a := Post{0, "Go Rocket Tutorial"}
    IncrementInPlace(&a)
    fmt.Println(a)
    b := Increment(a)
    fmt.Println(b)
}

You can run the above example in the Go Playground
If you’re new to the concept of Pointers you might also enjoy the “Understand Go pointers in less than 800 words” blog post.

C. Pointer Method Receivers:

One thing that trips up a lot of new Go developers is the way you attach methods to structs:

func (c *APIClient) Search(query string) (*APIResponse, error) {
    ...
}

In this example we’re attaching the Search function to the *APIClient type. This allows us to invoke it using this syntax:

client := NewAPIClient("accessToken")
response, err := client.Search(query)

So far so good. However if you use APIClient instead of *APIClient you’ll run into two unintended consequences:

Go will create a copy of your struct while calling the function. Changes you make will only affect your copy
Memory usage goes up because you’re copying your struct

Note that this isn’t always a bad thing. In some cases it’s actually faster to not use pointers. Here’s an example of where it goes wrong though:

package main
import (
    "fmt"
)
type Post struct {
    Upvotes      int
    Title string
}
func (p Post) BrokenIncrement() {
    p.Upvotes++
    fmt.Printf("p.Upvotes in the function is %d, memory location for p %p\n", p.Upvotes, &p)
}
func main() {
    p := Post{0, "Go Rocket Tuturial"}
    p.BrokenIncrement()
    fmt.Printf("p.Upvotes in main is still %d, memory location for p %p", p.Upvotes, &p)
}

You can run it on the Go playground to see why the above code doesn’t work:
TL/DR use pointer receivers unless you have a good reason not to. The “Don't Get Bitten by Pointer vs Non-Pointer Method Receivers in Golang” covers this issue in more detail.

Step 8 - Errors

Errors in Go are just variables that you return. Conceptually this approach is very easy to reason about when working with concurrent programming. It can be a bit hard to get the right workflow though. So before you get to read about channels & concurrency (the next topic) we’ll bore you with some mandatory error handling best practices:

A. Errors are returned as the last argument of a function:

Don’t deviate from this standard, if you’re writing a function that can fail be sure to allow for an error object as the last return value.

func Search(query string) (*APIResponse, error) {
…
}

B. Stack traces & Errors.Wrap:

The default errors don’t include a stack trace. This is why pretty much every Go app uses the pkg/errors module to wrap errors.

_, err := ioutil.ReadAll(r)
if err != nil {
        return errors.Wrap(err, "read failed")
}

C. Error causes:

The pkg/errors module also makes it easy to detect the cause of the error.

switch err := errors.Cause(err).(type) {
case *MyError:
        // handle specifically
default:
        // unknown error
}

D. Let errors bubble up:

Most of the time you want to centralize your error handling in one place of your app. You’ll for instance want to return a 500 from your server and log the error in Sentry.
That’s why most functions at the lower levels just return an error in case something breaks. You’ll see this pattern often:

err := doSomething()
if err != nil {
        return errors.Wrap(err, "read failed")
}

Step 9 - Channels & Goroutines

Channels are probably one of the sexiest features of Go. In this part of the tutorial we’ll take a deep dive on how you can leverage Goroutines and Channels for asynchronous programming in Go.
It's good to remember that channels are a feature for power users. By default Go uses a separate Goroutine for every incoming request. So typically you already get the benefit of a highly concurrent server without ever thinking about asynchronous programming. As the creator of Node said: “I think Node is not the best system to build a massive server web. I would use Go for that...”
Channels and Goroutines give you powerful tools to implement concurrency. For this exercise we’ll start 4 worker goroutines to download and resize 8 pictures from the Unsplash API.
Head over to http://localhost:3000/spacex_seams and watch the log output. You’ll see that 4 workers are downloading and resizing the images concurrently. Depending on the speed of the machine this will take anywhere from 1-20 seconds.

Let’s have a look at main.go to see how this works.

A. Channel Creation:

As a first step we create a channel for tasks and a channel for results:

resultChannel := make(chan TaskResult)
taskChannel := make(chan Task)
imagesToResize := 8

B. Starting the workers:

This loop starts the workers

// start 4 workers
for w := 1; w <= 4; w++ {
    go worker(w, taskChannel, resultChannel)
}

The “go” statement run the specified function in a separate Goroutine. A Goroutine is a very lightweight alternative to threads. Since a Goroutine only takes 2kb of overhead you can run millions of them even on commodity hardware.
The Goroutines execute the following code:

func worker(id int, taskChannel <-chan Task, resultChannel chan<- TaskResult) {
    for j := range taskChannel {
        fmt.Println("worker", id, "started  job", j.Position)
        resized, err := seam.ContentAwareResize(j.URL)
        resultChannel <- TaskResult{j.Position, resized, err}
    }
}

Note how the worker uses the range keyword to iterate over the taskChannel till it’s closed. When the worker receives a new task on the taskChannel it downloads the image, resizes it and writes the result to the resultChannel.

C. Write to the task channel:

As a next step we write to the taskChannel and close the channel when we’re done.

// write to the taskChannel and close it when we're done
go func() {
    for i, r := range response.Results[:imagesToResize] {
        taskChannel <- Task{i, r.URLs["small"]}
    }
    close(taskChannel)
}()

Writing or reading from a channel is a blocking operation. To prevent a situation in which we’re waiting forever we run this code in a separate Goroutine. Note the go func() {…}() syntax to execute the code in a Goroutine.

D. Retrieving the results:

This loop retrieves the results from the resultChannel and base64 encodes it for easy embedding in the resulting page.

// start listening for results in a separate goroutine
for a := 1; a <= imagesToResize; a++ {
    taskResult := <-resultChannel
    if taskResult.Err != nil {
        log.Printf("Image %d failed to resize", taskResult.Position)
    } else {
        sEnc := b64.StdEncoding.EncodeToString(taskResult.Resized)
        response.Results[taskResult.Position].Resized = sEnc
    }
}

Channels & Goroutines

And that’s it. Channels are a pretty low level concept. Go also provides Mutexes and Wait groups to help with concurrent programming. When your first learn about Channels it’s easy to become excited and use them to solve every problem. Some issues like preventing multiple writes on a shared object are sometimes easier solved using a simple Mutex. This wiki provides some guidelines on when to use a sync.Mutex or a Channel.
The cool part about Go’s concurrency model is that most of the time you don’t even need to think about it. You just write your synchronous code and Go handles the switching between Goroutines without you, the programmer, doing any work.
Channels, GoRoutines, Wait Groups and Mutexes give you full control over how you’re writing asynchronous code when you need it. If you want to learn more about channels check out this Golang channels tutorial.

Step 10 - Libraries and Tools

Go is an easy language to learn. What tends to take more time is figuring out the tooling and the ecosystem. Here are a few library and tool recommendations to help you get started.
Tool: Go doc
Go doc is an awesome little tool for generating reference docs for your Go APIs.
https://godoc.org/golang.org/x/tools/cmd/godoc
Tool: go fmt & go imports
Go fmt and go imports two tools help you automatically clean up your Go files.
Tool: PPROF
Go ships with awesome profiling capabilities and beautiful flamegraphs about your code’s performance. This tutorial is a solid starting point: https://jvns.ca/blog/2017/09/24/profiling-go-with-pprof/

Mux
Mux is an awesome little library for URL routing in Go:
CLI
The aptly named CLI is a great library for building command line interfaces in Go.
Zap
Zap, Uber’s logging library, is a better alternative to Logrus and the default Go logging system.
Go.Rice
Go Rice enables you to easily embed static files such as templates into your Go binary.
Squirrel
Squirrel makes it easy to work with SQL in Go.
One of the reasons why we picked Go is the mature ecosystem of libraries. Most of the time you’ll find high quality open source libraries to help you with your projects.

Concluding the Go Tutorial

At Stream we use Go & RocksDB to power the news feeds for over 300 million end users. What’s unique about Go is the balance it strikes. It’s ridiculously fast and still easy and productive to work with.

It’s been 18 months since Stream switched from Python to Go. There is always some uncertainty when making such a large change. Fortunately moving to Go worked out very well. We’re able to power a complex and large infrastructure with a relatively small development team. Go improves with every release. Every time a new version is released we rerun our benchmarks shave a few milliseconds of our response times. The tooling also gets better with every release.

Go is a fun language and I hope you enjoyed this tutorial. We covered the basics and even some more advanced topics such as Goroutines and Channels. Let me know if anything in this tutorial isn’t clear (@tschellenbach).

If you’ve never used Stream before, try out this Tour of the API. You can try it out with Javascript in the browser, or of course with the official Stream Go client.