Stream logo
Start Coding Free

Go 10 Week Backend Engineering Onboarding

Engineering
Go
13 min read
Tommaso B.
Tommaso B.
Published August 5, 2024

Welcome to Stream. If you’re reading the public version of this consider checking out our careers page. We’re currently hiring for Go roles of entry to principal/director levels.

Stream is an API for building chat, live video and activity feeds. We power thousands of apps and reach over a billion end users. Apps like Strava, Nextdoor, IBM, Adobe and Patreon rely on our tech.

As we scaled the team, the importance of excellent onboarding became quite clear. It always felt a little strange to me that most sales teams have excellent onboarding and engineering teams typically don’t.

This 10 week Go onboarding program covers a few different topics:

  • Go fundamentals & performance
  • Databases, scaling & Redis
  • Common patterns for scalability
  • Testing best practices
  • Review, measurement, errors and the whole code lifecycle
  • Raft & WebRTC

During this onboarding we work in a group and pick up a real project, the best way to learn is to combine the materials with actual work. Welcome, and let’s get started.

Onboarding Table of Contents

Week 1: Go Basics
Week 2: Testing with Go
Week 3: SQL & Database Performance
Week 4: OpenAI & JSon
Week 5: Redis
Week 6: Go Advanced Syntax
Week 7: Tracing & Performance
Week 8: Pebble & Raft Consensus
Week 9: WebRTC
Week 10: Exam Time - Complete the Following Tasks
Recap & Reminders
Appendix 1: Tech Lead Onboarding Tips
Appendix 2: Resources & Other Topics

💡 Some parts of this document refer to internal documentation requiring authentication

Week 1: Go Basics

You’ll be surprised how easy it is to pickup the basics of Go. Go as a language aims to have less features, and instead focuses on simplicity, performance and compilation speed. The power of Go comes from having performance that’s close to C++/Rust, with developer productivity that’s close to Python/Ruby.

Editor & Setup

Internal Docs

  • Our internal docs are available here:
  • https://stream-api-docs.vercel.app/
  • They also explain setting up Copilot and a custom Claude for easier development.

Learning the Basics of Go

Differences From Other Languages

If you’re coming from a different language most concepts in Go will feel familiar. Some things are a little different though so take a moment to understand those:

Other Gotchas: https://divan.dev/posts/avoid_gotchas/

Week 2: Testing with Go

Learning the syntax of Go is relatively easy. The most common reasons why we see backend engineers struggle are around testing, understanding databases or understanding the requirements/ business needs. That’s why week 2 focuses on testing.

It’s Easy to Get Testing Wrong

  • Not writing tests. If you do this, it eventually becomes very hard to fix issues in your codebase.
  • Mock everything. Some Mocking is essential, however for some devs that turns into an obsession with mocks, and if you do that you end up testing nothing.
  • Test everything. You can overdo it. If you add tests everywhere it can become hard to refactor your code and update tests.
  • Not having integration tests. This is related to mocking too much. I’ve seen large test suites that test lots of things without actually guaranteeing that the software does what it’s intended to do. You always need some integration tests

Resources to Learn Testing

Mini Practice Session

  • Create a little CLI command that searches google (no API just open the URL) for a given keyword and returns a boolean if “getstream.io” is in the response body
  • Abstract the google searching with a SearchEngine interface
  • Create a DummySearchEngine Mock with testify
  • Write a test that verifies you’re correctly detecting if getstream.io is present

Week 3: SQL & Database Performance

Now that we’ve covered the basics of Go & testing it’s time to talk about databases. In particular you want to understand DB performance in more detail since the traffic on Stream’s API is quite high. Join the #learn_db slack channel.

Database Basics

Indexes & Performance

Architecture Tips

  • APIs tend to have more reads than writes
  • Denormalizing data can often help with performance. IE you can have a “reaction count” field on a message instead of having the DB fetch all messages
  • A database doesn’t scale as well as a caching layer. So if you have a lot of repeated reads on something you often want to have a cache in front of the database
  • A common mistake is doing N queries. Say that you have a list of blogposts and you do 1 query per blogpost to fetch the Author. In that you’re doing N queries. You should avoid this and fetch the data in 1 or 2 queries
  • The second most common mistake with databases is not getting the indexes right. Be sure to check index usage with EXPLAIN
  • https://www.cybertec-postgresql.com/en/how-to-interpret-postgresql-explain-analyze-output/?utm_source=dbplatz

Other Tips

Week 4: OpenAPI & JSON

HTTP

Make sure to first play with some basic HTTP/REST/JSON stuff. Our APIs do a lot of custom stuff but behind the scenes we still the standard library (net/http).

Things to Study to Get Ready

Here’s one that covers many common things related to HTTP client/servers. https://www.digitalocean.com/community/tutorials/how-to-make-an-http-server-in-go

  • Building a simple HTTP server
  • HTTP handler functions and middleware
  • Build simple client/server program that handles input coming from body, query params and path params

How to Test an HTTP Server and Its Controllers

Golang comes with default testing facilities to test request/response objects. We abstract most of this stuff in our API but it still good to have a good understanding on what happens behind the scenes. Here’s a good article that explains the basics around testing HTTP req/resp

https://quii.gitbook.io/learn-go-with-tests/build-an-application/http-server

  • Testing HTTP servers and handlers
  • Input validation

Note: In most cases HTTP controllers and HTTP servers do not do much more than exposing some complex business logic. At Stream most of the logic is not inside the controller but inside the state package. Controllers mostly handle authentication and permission on top of that.

Testing HTTP controllers really often requires mocking/stubbing, make sure to spend a bit of time reading about Go’s interfaces and mocking.

go
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
type GranadeLauncher interface {
    Reload() error
    Aim(x, y, z float64) error
    Shoot() (bool, error)
}

type M75 struct {
    // here the real implementation throws real granades, very expensive to test
}

type MyMock struct {
    GranadeLauncher
}

func (MyMock) Reload() {}

Tip: sometimes you want to create a quick implementation of an interface but do not want to get all methods implemented. You can embed the interface type into your struct. Doing so gives you a real-implementation. NOTE: calling Aim or Shoot on this interface will raise a nil pointer exception 😉

JSON & Golang

Compared to Node.js or Python, Golang exposes a lower-level API when it comes to HTTP. Make sure to understand the basics related to reading the body for a request and a response.

Golang supports JSON out-of-the-box, you can encode/decode (marshall/unmarshall) types from/to JSON/Go very easily.

Make sure to study how JSON works in Golang:

  • Field tags (annotations)
  • Encoding and Decoding payloads
  • Zero-values + pointer types + JSON encode/decode

Request Validation

  • Query parameters
  • Path
  • Body

OpenAPI

  • Quick introduction
  • Go + OpenAPI for public APIs

Goland Editor Tips

Week 5: Redis

https://redis.io/

https://github.com/redis/go-redis

Use Cases

  • Caching
  • Rate limits
  • Counters

This interactive tutorial is a nice start: https://try.redis.io/

Here’s a super basic tutorial: https://betterstack.com/community/guides/scaling-go/redis-caching-golang/#final-thoughts

Basics

  • String
  • Hash
  • Sorted Set
  • PubSub

Client Side Caching:

Pipelines:

LUA Scripting:

Understand Locks & Semaphores: https://pkg.go.dev/golang.org/x/sync/semaphore and proceed with how Redis enables you build distributes versions of this. See section 6 of this book

https://redis.com/ebook/redis-in-action/

Understand Rate Limiting: https://medium.com/@SaiRahulAkarapu/rate-limiting-algorithms-using-redis-eb4427b47e33

Assignment

  • Create a sorted set of followers
  • Store the sorted set in Redis
  • Keep both redis & local memory in sync with the client side caching approach discussed above

Cache stampede: https://zahere.com/cache-stampede-a-problem-the-industry-fights-every-day

Week 6: Go Advanced Syntax

Go Routines

Context

Functions & Constructors

Sync package

Maps, Slices and Interfaces

Structs: Embedding structs https://gobyexample.com/struct-embedding

Generics: https://go.dev/doc/tutorial/generics

Design Patterns in Go

Week 7: Tracing & Performance

Tracing & Errors

The first step of fixing performance is understanding what’s slow. Often that means instrumenting your code with timers using Prometheus and Open telemetry.

Some components are automatically traced and measured for you, for instance:

  • API controllers latency and errors
  • Redis calls with their latency (open telemetry)
  • SQL queries (open telemetry)

Benchmarks:

Monitoring Errors & Performance

At Stream we use a few different tools to monitor errors and performance:

  • ELK stack for logs
  • Prometheus
  • Grafana
  • Sentry for tracking programming errors
  • Jaeger to inspect traces locally
  • Grafana Tempo (production)
  • Cloudwatch/SNS for AWS metrics and alerts

Common Performance Issues

Go is extremely fast and likely won’t be the bottleneck for performance issues. The most common causes of performance issues are:

  • Doing too many queries
  • Queries not using the right index
  • No caching in place where you need caching
  • Deadlocks either in the DB or using mutexes in Go
  • Code complexity causing you to do things you don’t need to be doing

That being said, you will occasionally run into Go performance challenges. Especially on the video project where buffering can be intense.

Go Performance Optimization

Production Issues

  • Searching logs
  • Using Grafana
  • Using tracing
  • Using ELK stack to understand the logs
  • Reproduce an issue
  • Root cause
  • Post mortem

Week 8: Pebble & Raft Consensus

Raft and RockDB/Pebble are things you don’t often need to use. In 99% of cases simple CockroachDB & Redis will be a better fit. It can be interesting for systems where you have very high throughput though.

Pebble & Raft

Week 9: WebRTC

You can’t fully learn WebRTC in a week, but going through the webrtccourse is possible in one week and you’ll learn the basics.

Week 10: Exam Time - Complete the Following Tasks in One Week

OG Tag URL Preview Service (1 day)

  • Goal: Basic testing and monitoring
  • Create an API that gives you OG tags as json for a URL
  • Consider redis caching for popular urls
  • Consider failure scenarios and circuit breakers
  • Write tests
  • Think about metrics
  • This has to be easy for you. If it takes you a week you’re not up to speed yet

Latest Apple Stock Price Lookup (1 day)

  • Goal: understand client side caching and network latency
  • Start 3 servers that use redis client side caching to store the latest apple stock price
  • Measure response time (should be 0-1ms to show the stock price)
  • Think about how something like this could be used for realtime

Create a Realtime Text Editor API (1 day)

  • Goal: understand finite state machines and redis pubsub
  • The API should receive text edits (not the final text, just the edits) from multiple users
  • You replicate these edits via redis pub sub
  • The API returns the
    • Resulting text
    • The deltas via pubsub
  • On the other client you should be able to reconstruct the text from the deltas

Create a Commenting API (1 day)

  • Goal: database usage
  • Create a threaded commenting API powered by cockroachdb
  • Comments should have upvotes, downvotes, likes and a reply count
  • Efficient API endpoints to show
    • Comments by date
    • Comments by reply count
    • Comments by upvotes
  • Consider both database efficiency and Redis efficiency

Grading: Send a Github Repo to Tommaso

GradeDescription
5complete 2 out of 4
6complete 3 out of 4
7-10complete 4 out of 4, more points for a solid understanding per exercise

Recap & Reminders

The most common reasons for a backend engineer to get stuck are:

  • Not writing tests. And the resulting complexity that you end up with if you don’t write tests. Very easy to get stuck
  • Not understanding the database and Redis
  • Not understanding the business/product full stack requirements. (if you’ve ever worked on SDKs you have a major advantage in this area)
  • Making things more complex than they need to be

Keep that in mind and be sure to avoid those pitfalls

A note on working in a team:

  • As a guideline we recommend everyone post an update to their slack channel and lattice by Friday. You want to let your manager know how things are going and keep them updated. If you don’t do this the manager will end up continuously asking you how it’s going. If your manager needs to reach out to you and ask how its going its a sign you’re not reporting up effectively
  • It’s not just about your manager, also keep your team informed about what you’re working on and how things are going
  • Use channels instead of DMs on slack

Appendix 1 - Tech Lead Onboarding Tips

Appendix 2 - Resources & Other Topics

Devops Basics

  • Cloudinit
  • Cloud Formation
  • Puppet

Other Storage Tech

  • Fulltext search with ElasticSearch
  • RabbitMQ or similar message queues

Scaling

We’ve started on this topic, but didn’t cover everything yet.

Postgresql Performance

  1. https://gitlab.com/postgres-ai/postgresql-consulting/postgres-howtos
  2. https://github.com/NikolayS/postgres_dba/tree/master/sql
  3. https://www.highgo.ca/blog/
  4. https://youtu.be/gtXNWdW42SQ
  5. https://youtu.be/fHlIJg4x13g
  6. https://twitter.com/jer_s
  7. https://www.youtube.com/watch?v=ysG3x2QOu0c&list=PLhr1KZpdzukeAZuX0pNZ1lLDkaZWBOhQT

Postgresql Misc.

Recommended posts
Using GStreamer with Go for Advanced Streaming SolutionsIs Go The Right Choice for Building a Backend?Go 10 Week Backend Engineering OnboardingStream Is Deprecating Virtual Go – It’s Time to Move to Go ModulesHow a Go Program Compiles down to Machine CodeGo 1.11 Rocket TutorialFixing the billion dollar mistake in Go by borrowing from RustGo Client Now Available