Create a scalable clone of Twitter with Django and Stream

In this tutorial, we will make a Twitter clone using Django and Stream, a hosted API for newsfeed development. By the end, you'll see how easy is to power your newsfeeds with Stream. For brevity, we leave out some basic Django-specific code and recommend referring to the Github project for the complete runnable source code. At the end of this tutorial, we'll have a Django app (with a profile feed), a timeline feed, and support for following users, hashtags and mentions. For this tutorial, we're assuming you're familiar with Django. If you're new to Django the official tutorial explains it very well.

Bootstrap the Django application

We will use Python 3.6 and Django 2.0, which is the latest major release at the time of writing. Make sure you have a working Django project before you continue to the next part of the tutorial.

Create the Django app

Let's start by creating a new Django app called stream_twitter:

bash

1
python manage.py startapp stream_twitter

Install stream_django

The stream_django project provides the GetStream integration for Django. It is built on top of thelow-levell stream_python API client.

bash

1
pip install stream-django

To enable stream_django you need to add it to your INSTALLED_APPS:

ini

1
2
3
4
5
6
7
8
9
10
INSTALLED_APPS = (
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
    'stream_twitter',
    'stream_django'
)

Stream setup

First of all, we need to create a Stream account. You can signup with Github, which is free for usage below 3 million feed updates per month. Once you've signed up, get your API 'key' and 'secret' from the dashboard and add them to Django's settings:

ini

1
2
STREAM_API_KEY = 'my_api_key'
STREAM_API_SECRET = 'my_api_secret'

The models

In this application we will have three different models: users, tweets and follows. To keep it as simple as possible, we will use Django's contrib.auth user model. Have a look below at the initial version of the Tweet and Follow models.

python

1
2
3
4
5
6
7
8
9
10
11
from django.db import models
class Tweet(models.Model):
    user = models.ForeignKey('auth.User', on_delete=models.CASCADE)
    text = models.CharField(max_length=160)
    created_at = models.DateTimeField(auto_now_add=True)
class Follow(models.Model):
    user = models.ForeignKey('auth.User', related_name='friends', on_delete=models.CASCADE)
    target = models.ForeignKey('auth.User', related_name='followers', on_delete=models.CASCADE)
    created_at = models.DateTimeField(auto_now_add=True)
    class Meta:
        unique_together = ('user', 'target')

Now, let's create the schema migrations and apply them:

bash

1
2
python manage.py makemigrations stream_twitter
python manage.py migrate

Let's also setup the view to add tweets:

python

1
2
3
4
5
6
7
8
from django.views.generic.edit import CreateView
from stream_twitter.models import Tweet
class TweetView(CreateView):
    model = Tweet
    fields = ['text']
    def form_valid(self, form):
        form.instance.user = self.request.user
        return super(Tweet, self).form_valid(form)

And, of course, add it to urls.py:

python

1
2
3
4
5
6
7
8
9
from django.contrib import admin
from django.contrib.auth.decorators import login_required
from django.urls import path
from stream_twitter import views
urlpatterns = [
    path('admin/', admin.site.urls),
    path('timeline/', login_required(views.TimelineView.as_view()), name='timeline'),
    re_path(r'^user/(?P.+)/', views.UserView.as_view(), name='user_feed')
]

Now that we have the view setup for creating tweets, we can move on to setting up the newsfeed.

Model integration

We want the tweets to be stored in the author's feed. This is when we start using the stream_django integration. We can configure the Tweet model so that it will syncronise automatically with feeds. To do this, we need to make Tweet a subclass of 'stream_django.activity.Activity': We'll do that by modifying the model we defined earlier in stream_twitter.models:

python

1
2
3
from stream_django import activity
class Tweet(activity.Activity, models.Model):
    ...

From now on, new tweets will be added to the user feed of the author *and* to the feeds of all their followers. The same applies to deleting a tweet. So, let's give it a try using Django's shell:

python

1
2
3
4
5
6
7
8
python manage.py shell

from stream_twitter.models import Tweet
from django.contrib.auth.models import User
user, _created = User.objects.get_or_create(username='tester')
Tweet.objects.create(
    user=user,
    text='Go Cows!')

We've now created our first Tweet and, in turn, added an Activity to a Feed via the Stream API. By default, *stream-django* creates and adds the Activity to a feed named after the 'actor' property. This can be customized by overriding the _attr functions inherited from the stream_django.activity.Activity mixin on the Django Model. Now, this is the first time we talk about Activities and Feeds so let's take a moment to define what an activity is. An activity is an object that contains information about an action that is performed by someone involving an object. When you write data to Stream's feeds, you send this data in the form of activities. The simplest activity is made by these three fields: actor, object and verb. For example: Tommaso tweets 'Go cows!' Stream's APIs allow you to store additional fields in your feeds, as you can see from the documentation here. We can verify the Activity was added by using the Data Browser in Stream's Dashboard. In this example you can determine the feed name by inspecting the activity_actor property:

python

1
2
>>> t.activity_actor
'auth.User:1'

The Stream Data Explorer:

Get started! Activate your free Stream account today and start prototyping with feeds.

User feed

So now that every tweet gets stored in the author's feed, we'll add a view that reads them.

python

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
from django.contrib.auth.models import User
from django.shortcuts import render, get_object_or_404
from django.views.generic import DetailView
from stream_django.enrich import Enrich
from stream_django.feed_manager import feed_manager
enricher = Enrich()
class UserView(DetailView):
    model = User
    template_name = 'stream_twitter/user.html'
    def get_object(self):
        return self.get_queryset().get(username=self.kwargs['username'])
    def get_context_data(self, object):
        user = self.object
        feeds = feed_manager.get_user_feed(user.id)
        activities = feeds.get()['results']
        activities = enricher.enrich_activities(activities)
        return {
            'activities': activities,
            'user': user,
            'login_user': self.request.user
        }

There are two new things that I should explain: the *feed manager* and the *enricher*. As the name suggests, the feed manager takes care of managing the different feeds involved in your app. In this case, we ask the feed manager to give us the user feed for the current user. We learned before that data is stored as activities within feeds. This is what a tweet looks like when we read it from Stream:

json

1
2
3
4
5
6
[{
    'actor': 'auth.User:1',
    'object': 'stream_twitter.Tweet:1',
    'verb': 'tweet',
    ... other fields ...
}]

As you can see, 'object' field does not contain the tweet itself but a reference to that instead(the same applies to the 'actor' field). The enricher replaces these references with model instances.

Templating feeds

*django_stream* comes with a template tag that helps you to show the content from feeds in your templates. This can get quite complex as you add different kinds of activities to your feeds. Here is a very minimal tweets.html template:

django

1
2
3
4
{% load activity_tags %}
{% for activity in activities %}
    {% render_activity activity %}
{% endfor %}

The first time you run this, Django will complain that 'activity/tweet.html' is missing. That's because the render_activity templatetag inspects the activity object and loads the template based on the verb. Because the verb in this case is 'tweet', it will look for tweet.html in activity path. The template tag accepts extra options to make your templates as re-usable as possible, see here for the template tag documentation.

Feed Follow

As a next step, well add the ability to follow users to the application. To do this we create a view that creates Follow objects.

python

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
from django.views.generic.edit import CreateView, DeleteView
from django.urls import reverse_lazy
from stream_twitter.forms import FollowForm
from stream_twitter.models import Follow
class FollowView(CreateView):
    form_class = FollowForm
    model = Follow
    success_url = reverse_lazy('timeline_feed')
    def form_valid(self, form):
        form.instance.user = self.request.user
        return super(FollowView, self).form_valid(form)
class UnfollowView(DeleteView):
    model = Follow
    success_url = reverse_lazy('timeline_feed')
    def get_object(self):
        target_id = self.kwargs['target_id']
        return self.get_queryset().get(target__id=target_id)

Now we can use Django's signals to perform follow/unfollow requests on Stream's APIs.

python

1
2
3
4
5
6
7
def unfollow_feed(sender, instance, **kwargs):
    feed_manager.unfollow_user(instance.user_id, instance.target_id)
def follow_feed(sender, instance, created, **kwargs):
    if created:
        feed_manager.follow_user(instance.user_id, instance.target_id)
    post_save.connect(follow_feed, sender=Follow)
    post_delete.connect(unfollow_feed, sender=Follow)

Timeline view (AKA Flat Feed)

The hardest part for a scalable Twitter clone is displaying a feed that shows the tweets from people you follow. This is commonly called the timeline view or newsfeed. The code below shows the timeline.

python

1
2
3
4
5
6
7
8
9
10
from django.views.generic import TemplateView
class TimelineView(TemplateView):
    template_name = 'stream_twitter/timeline.html'
def get_context_data(self):
    context = super(TimelineView, self).get_context_data()
    feeds = feed_manager.get_news_feeds(self.request.user.id)
    activities = feeds.get('timeline').get()['results']
    enriched_activities = enricher.enrich_activities(activities)
    context['activities'] = enriched_activities
    return context

This code looks very similar to the code of profile_feed. The main difference is we use feed manager's get_news_feeds. By default, Stream apps and stream_django come with two newsfeeds predefined: flat and aggregated feeds. When you use feed_manager.get_news_feeds, you get a dictionary with flat and aggregated feeds. Since we are not going to use aggregated feeds, we can adjust Django settings:

ini

1
STREAM_NEWS_FEEDS = dict(flat='flat')

Adding activities

Let's modify the TimelineView to include a form that will accept new tweets:

python

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
from django.views.generic.edit import CreateView
class TimelineView(CreateView):
    fields= ['text']
    model = Tweet
    success_url = reverse_lazy('timeline_feed')
    template_name = 'stream_twitter/timeline.html'
    def form_valid(self, form):
        form.instance.user = self.request.user
        return super(TimelineView, self).form_valid(form)
    def get_context_data(self, form=None):
        context = super(TimelineView, self).get_context_data()
        feeds = feed_manager.get_news_feeds(self.request.user.id)
        activities = feeds.get('timeline').get()['results']
        enriched_activities = enricher.enrich_activities(activities)
        context['activities'] = enriched_activities
        context['login_user'] = self.request.user
        context['hashtags'] = Hashtag.objects.order_by('-occurrences')
        return context

Hashtags feeds

We want Twitter style hashtags to work as well - which is surprisingly easy. First, let's open Stream's dashboard and create the 'hashtag' feed type. Note: by default Stream will setup user, timeline, timeline_aggregated and notification feeds. If you more feeds, you need to configure them in the dashboard.

python

1
2
3
4
from django.template.defaultfilters import slugify
class Tweet(activity.Activity, models.Model):
    def parse_hashtags(self):
        return [slugify(i) for i in self.text.split() if i.startswith("#")]

Now that we have parsed the hashtags, we could loop over them and publish the same activity to every hashtag feed. Fortunately, there's a shortcut. Stream allows you to send a copy of an activity to many feeds with a single request. To do this, we only need to implement the activity_notify method to the Twitter model we created previously:

python

1
2
3
4
5
6
7
8
from stream_django.feed_manager import feed_manager
class Tweet(activity.Activity, models.Model):
    @property
    def activity_notify(self):
        targets = []
        for hashtag in self.parse_hashtags():
            targets.append(feed_manager.get_feed('hashtag', hashtag))
        return targets

From now on, activities will be stored to hashtags feeds as well. For instance, the feed 'hashtag:Django' will contain all tweets with '#Django'. Again, the view code looks really similar to the other views.

python

1
2
3
4
5
6
7
8
9
10
11
12
13
from django.views.generic import TemplateView
from stream_django.enrich import Enrich
from stream_django.feed_manager import feed_manager
class HashtagView(TemplateView):
    template_name = 'stream_twitter/hashtag.html'
    def get_context_data(self, hashtag):
        context = super(TemplateView, self).get_context_data()
        hashtag = hashtag.lower()
        feed = feed_manager.get_feed('user', f'hash_{hashtag}')
        activities = feed.get(limit=25)['results']
        context['hashtag'] = hashtag
        context['activities'] = enricher.enrich_activities(activities)
        return context

Mentions

Now that we found out about the activity_notify property, it only takes a bunch of extra lines of code to add user mentions.

python

1
2
3
4
5
6
7
8
9
10
11
12
class Tweet(activity.Activity, models.Model):
    def parse_mentions(self):
        mentions = [slugify(i) for i in self.text.split() if i.startswith("@")]
        return User.objects.filter(username__in=mentions)
    @property
    def activity_notify(self):
        targets = []
    for hashtag in self.parse_hashtags():
        targets.append(feed_manager.get_feed('hashtag', hashtag))
    for user in self.parse_mentions():
        targets.append(feed_manager.get_news_feeds(user.id)['flat'])
    return targets

Wrapping up

Congratulations, you've reached the end of this tutorial! This article showed you how easy it is to build scalable newsfeeds with Django and GetStream.io. It took us just 100 LoC and (I hope) less than one hour to get this far. You can find the code from this tutorial and the fully functional application on GitHub. I hope you found this interesting and useful and I'd be glad to answer all of your questions. If you're new to Django or Stream, I highly recommend the official Django tutorial and the Stream 'getting started' tutorial.