Importing Data

If you’ve just started using Stream, you might want to import data from your old infrastructure. Instead of using the APIs and creating your own import scripts, you can make use of our import feature. If you are on a paid plan you can import data into your app via the dashboard.

The Process

The steps for importing data into your app are as follows:

Generate the import file for your data (full file reference below)
Upload the import file via the dashboard. The import file is limited to a maximum size of 300MB. For larger imports please contact us
The file will be validated according to the rules described in this doc
If validation passes, a member of our team will approve and run the import
Once the import is completed (usually a few minutes), you will get a confirmation email

Before uploading the import file make sure that every feed group in your import file is configured in your app.

Importing data on a live app may cause high response times for requests during the import process.

Import File

The import file must be in the JSON Lines text format (newline-delimited json). That means that each line must be a valid json. More info here.

Below example snippets are written in multiple lines for clarity but they are wrong formats for JSON Lines because each instruction must occupy a single line. If you’re copying these examples directly, get rid of line breaks. Otherwise, import will fail in analyze step.

Instruction Reference

Each line consists of an import instruction and must have the following elements:

Name	Type	Description
instruction	string	Instruction name. Must be a valid instruction type.
feedId	string	The feed to import data to, required by add_activities and follow
data	array	The data for that particular instruction, see below.

Instruction Types

An import file can contain different types of instructions. This is the list of supported instruction types:

add_activities
follow
reactions
users
collections

follow

An array of feed ids. As you can only follow flat feeds, make sure that all the feeds are flat feeds.

{
  "instruction": "follow",
  "feedId": "user:bob",
  "data": ["user:alice", "user:tim"]
}

add_activities

An array of activities at max 3600 in a line. The rules described in the adding activities doc apply for each activity. In addition, each activity must have the foreign_id and time fields present. The time field must have the RFC3339 format.

{
  "instruction": "add_activities",
  "feedId": "user:1",
  "data": [
    {
      "actor": "user:1",
      "verb": "like",
      "object": "cat:1",
      "foreign_id": "like:2",
      "time": "2017-02-01T16:03:39+00:00"
    },
    {
      "actor": "user:1",
      "verb": "share",
      "object": "picture:2",
      "foreign_id": "share:3",
      "time": "2017-02-01T19:03:39+00:00"
    }
  ]
}

collections

An array of collection entries in which each line can have up to 10,000 entries. Entries must have the following fields not empty: collection, id and data.

{
  "instruction": "collections",
  "data": [
    {
      "id": "qotsa",
      "collection": "bands",
      "site": "http://www.qotsa.com/"https://getstream.io/docs/js?preview=1#feeds-import-data
    }
  ]
}

users

An array of users in which each line can have up to 10,000 entries. Entries must have the id field.

{
  "instruction": "users",
  "data": [
    {
      "id": "415",
      "username": "josh-homme",
      "band": "SO:bands:qotsa"
    },
    {
      "id": "42",
      "username": "bob",
      "band": "SO:bands:bob-dylan"
    }
  ]
}

reactions

An array of reactions in which each line can have up to 10,000 entries. Entries must include: id, kind, user_id, activity_id, time. If the related activity is part of the same import, activity_foreign_id and activity_time replace activity_id. Check the REST docs for the complete list of allowed fields.

{
  "instruction": "reactions",
  "data": [
    {
      "id": "8c1740ed-51f1-456f-b911-6d7c7aa183dd",
      "kind": "like",
      "time": "2017-02-01T16:03:39+00:00",
      "user_id": "tim",
      "activity_id": "af781804-847e-11e4-8080-80012fb97b9e"
    }
  ]
}
{
  "instruction": "reactions",
  "data": [
    {
      "id": "8c1740ed-51f1-456f-b911-6d7c7aa183dd",
      "kind": "comment",
      "data": {
        "text": "how many imports must a man"
      },
      "time": "2017-02-01T19:03:39+00:00",
      "user_id": "bob",
      "activity_time": "2017-02-01T16:03:38+00:00",
      "activity_foreign_id": "post:123"
    }
  ]
}

Batch Methods

Rate Limits