Importing Data

Last Edit: Feb 02 2020

If you've just started using Stream, you might want to import data from your old infrastructure. Instead of using the APIs and creating your own import scripts, you can make use of our import feature. If you are on a paid plan you can import data into your app via the dashboard.

The Process

The steps for importing data into your app are as follows:

  1. Generate the import file for your data (full file reference below)

  2. Upload the import file via the dashboard. The import file is limited to a maximum size of 300MB. For larger imports please contact us

  3. The file will be validated according to the rules described in this doc

  4. If validation passes, a member of our team will approve and run the import

  5. Once the import is completed (usually a few minutes), you will get a confirmation email

Before uploading the import file make sure that every feed group in your import file is configured in your app.
Importing data on a live app may cause high response times for requests during the import process.

Import File

The import file must be in the JSON Lines text format (newline-delimited json). That means that each line must be a valid json. More info here.

Instruction Reference

Each line consists of an import instruction and must have the following elements:

Name

Type

Description

instruction

string

Instruction name. Must be a valid instruction type.

feedId

string

The feed to import data to, required by add_activities and follow

data

array

The data for that particular instruction, see below.

Instruction Types

An import file can contain different types of instructions. This is the list of supported instruction types:

  • add_activities

  • follow

  • reactions

  • users

  • collections

follow

An array of feed ids. As you can only follow flat feeds, make sure that all the feeds are flat feeds.


{
    "instruction": "follow",
    "feedId": "user:bob",
    "data": [
        "user:alice",
        "user:tim"
    ]
}
                    

add_activities

An array of activities. The rules described in the adding activities doc apply for each activity. In addition, each activity must have the foreign_id and time fields present. The time field must have the RFC3339 format.


{
    "instruction": "add_activities",
    "feedId": "user:1",
    "data": [
        {
            "actor": "user:1",
            "verb": "like",
            "object": "cat:1",
            "foreign_id": "like:2",
            "time": "2017-02-01T16:03:39+00:00"
        },
        {
            "actor": "user:1",
            "verb": "share",
            "object": "picture:2",
            "foreign_id": "share:3",
            "time": "2017-02-01T19:03:39+00:00"
        }
    ]
}
                    

collections

An array of collection entries in which each line can have up to 10,000 entries. Entries must have the following fields not empty: collectionid and data.


{
    "instruction": "collections",
    "data": [
        {
            "id": "qotsa",
            "collection": "bands",
            "site": "http://www.qotsa.com/"<a href="https://getstream.io/docs/js?preview=1#feeds-import-data" target="_self">https://getstream.io/docs/js?preview=1#feeds-import-data</a>
        }
    ]
}
                    

users

An array of users in which each line can have up to 10,000 entries. Entries must have the id field.


{
    "instruction": "users",
    "data": [
        {
            "id": "415",
            "username": "josh-homme",
            "band": "SO:bands:qotsa"
        },
        {
            "id": "42",
            "username": "bob",
            "band": "SO:bands:bob-dylan"
        }
    ]
}
                    

reactions

An array of reactions in which each line can have up to 10,000 entries. Entries must include: kinduser_idactivity_idtime. If the related activity is part of the same import, activity_foreign_id and activity_time replace activity_id. Check the REST docs for the complete list of allowed fields.


{
    "instruction": "reactions",
    "data": [
        {
            "kind": "like",
            "time": "2017-02-01T16:03:39+00:00",
            "user_id": "tim",
            "activity_id": "af781804-847e-11e4-8080-80012fb97b9e"
        }
    ]
}
{
    "instruction": "reactions",
    "data": [
        {
            "kind": "comment",
            "data": {
                "text": "how many imports must a man"
            },
            "time": "2017-02-01T19:03:39+00:00",
            "user_id": "bob",
            "activity_time": "2017-02-01T16:03:38+00:00",
            "activity_foreign_id": "post:123"
        }
    ]
}