Importing Data

If you’ve just started using Stream, you might want to import data from your infrastructure or previous chat provider. User, Channel, and Message data can all be imported through the Stream Dashboard.

Importing From Dashboard

From your dashboard select the app you wish to import chat data for. Then click on ‘App Settings’ in the top-right corner of your screen, and click ‘Import Chat Data’

You are only allowed to import data for apps in development mode.

You will be presented with a modal asking for a JSON file. Please ensure you follow our import format below to format to avoid errors.

The max size of a file we are able to handle in an upload is 300MB please get in touch with support if you need to upload a larger file size.

Once you have uploaded your file, scroll down on the chat overview page to see a ‘Data Import’ section. Here you can view some details about the file upload:

The person who uploaded the file
The date the file was uploaded
The filename that was uploaded
The current status of the upload

If your import fails validation on our end you can upload a new file by clicking on the ‘New Import’ button. To view more details of your import click on it within the list shown under ‘Data Imports’. This will show a modal where you can see the size of the import and a preview of the JSON that was uploaded.

What To Expect

Once the file validation begins, the status of the job will update to analyzing.

If unsuccessful, the job moves to analyze_failed status including a json with the results to help correct errors. Once corrected, upload a new file, which will create a new job.

If successfully validated, the job moves to waiting_confirmation status. We will approve the import on our side, and the import process will startcustomer supportto complete the rest of the import process. A completed status means the data has been imported and will be visible on the dashboard explorer, or through API query.

A file which is error free can be imported quickly, however, it is recommended to allow up to 7 days for this process as it can be difficult to predict errors, especially with large amounts of data.

By default all imported messages will be marked as read. If for a channel member the last_read timestamp is provided in the import data, for that member the unread count will be determined based on the amount of messages that have been created after the last_read timestamp.

Your import file will be validated for references. That’s why all related records must be provided in the same file. For example, a channel can’t be imported without its creator user not in the file. This automated validation phase is needed to prevent mistakes.

Any reference to data that already exists in an application (channel, user) will be treated as a full update in the import. The only data that will remain are channel members.

For distinct channels, don’t try to emulate our hashing algorithm or setting an id. If you provide member ids, it will be auto generated deterministically.

For time columns, support format is same with the API, RFC 3339: 1985-04-12T23:20:50.52Z https://www.rfc-editor.org/rfc/rfc3339#section-5.8

Import File Format

The import file must be structured as a JSON array of objects. Each object is an item to add to your application (eg. a user, a message, etc.).

Here’s an example of an import file:

[
 {
   "type": "channel",
   "item": {
     "id": "6e693c74-262d-4d3d-8846-686364c571c8",
     "type": "livestream",
     "created_by": "aad24491-c286-4169-bbfc-9280de419cb6",
     "name": "Rock'n Roll Circus"
   }
 },
 {
   "type": "member",
   "item": {
     "channel_type": "livestream",
     "channel_id": "6e693c74-262d-4d3d-8846-686364c571c8",
     "user_id": "aad24491-c286-4169-bbfc-9280de419cb6",
     "is_moderator": true,
     "created_at": "2017-02-01T02:00:00Z"
   }
 },
 {
   "type": "message",
   "item": {
     "id": "977e691a-c091-4e3b-8f70-ba8944a3e500",
     "channel_type": "livestream",
     "channel_id": "6e693c74-262d-4d3d-8846-686364c571c8",
     "user": "aad24491-c286-4169-bbfc-9280de419cb6",
     "text": "Such a great song, check out my solo at 2:25",
     "type": "regular",
     "created_at": "2017-02-01T02:00:00Z",
     "attachments": [
       {
         "type": "video",
         "url": "https://www.youtube.com/watch?v=flgUbBtjl9c"
         "image_url": "https://my.domain.com/awesome/image.jpg"
       }
     ]
   }
 },
 {
   "type": "reaction",
   "item": {
     "message_id": "977e691a-c091-4e3b-8f70-ba8944a3e500",
     "type": "love",
     "user_id": "aad24491-c286-4169-bbfc-9280de419cb6",
     "created_at": "2019-03-02T15:00:00Z"
   }
 },
 {
  "type": "user",
  "item": {
   "id": "aad24491-c286-4169-bbfc-9280de419cb6",
   "name": "Jesse",
   "image": "http://getstream.com",
   "created_at": "2017-01-01T01:00:00Z",
   "role": "moderator",
   "invisible": true,
   "description": "Taj Mahal guitar player at some point"
  }
 }
]

Import Entries Format

Name	Type	Description
type	string	the type of data for this entry. Allowed values are: user, channel, message, reaction, member
item	object	the data for this entry, see below for the format of each type

Item Types

An import file can contain entries with any of these types.

Note that you can add custom fields to users, channels, members, messages, attachments and reactions. The limit is 5KB of custom field data per object.

User Type

Any user referenced in the import file needs to be included in its entirety, even if it already exists. The user type fields are shown below:

name	type	description	default	optional
id	string	the unique id for the user	-	✓
created_at	string	user creation time in RFC3339 format	-	✓
deleted_at	string	user deletion time in RFC3339 format	-	✓
updated_at	string	user update time in RFC3339 format	-	✓
role	string	user role	-	✓
invisible	boolean	user visibility	-	✓
teams	list of string	list of teams the user is part of	-	✓
*	string/list/object	add as many custom fields as needed	-	✓

{
  "type": "user",
  "item": {
    "id": "aad24491-c286-4169-bbfc-9280de419cb6",
    "name": "Jesse",
    "image": "http://getstream.com",
    "created_at": "2017-01-01T01:00:00Z",
    "role": "moderator",
    "invisible": true,
    "teams": ["admins"],
    "description": "Taj Mahal guitar player at some point"
  }
}

Channel Type

The channel type fields are shown below:

name	type	description	default	optional
id	string	the unique id for the channel, not required if you provide member_ids
type	string	the type of the channel. ie livestream, messaging etc.
created_by	string	the id of the user that created the message
frozen	boolean	you can’t add messages to a frozen channel	false	✓
member_ids	list	can be used to generate the channel lD based on member IDs and create distinct channels		✓
disabled	boolean	if the channel is disabled	false	✓
created_at	string	channel creation time in RFC3339 format	-	✓
updated_at	string	channel update time in RFC3339 format	-	✓
*	string/list/object	add as many custom fields as needed	-	✓

{
  "type": "channel",
  "item": {
    "id": "6e693c74-262d-4d3d-8846-686364c571c8",
    "type": "livestream",
    "created_by": "aad24491-c286-4169-bbfc-9280de419cb6",
    "name": "Rock'n Roll Circus"
  }
}

Distinct Channels

Distinct channels are channels that are defined by their specific members, not by id. In this case the Channel will be defined as:

{
  "type": "channel",
  "item": {
    "member_ids": ["userA", "userB"],
    "type": "livestream",
    "created_by": "aad24491-c286-4169-bbfc-9280de419cb6",
    "name": "Rock'n Roll Circus"
  }
}

Any object referencing this channel (including Member objects), will have to reference this channel similarly:

{
  "type": "message",
  "item": {
    "id": "977e691a-c091-4e3b-8f70-ba8944a3e500",
    "channel_type": "livestream",
    "channel_member_ids": ["userA", "userB"],
    "user": "userA",
    "text": "Such a great song, check out my solo at 2:25",
    "type": "regular",
    "created_at": "2017-02-01T02:00:00Z"
  }
}

Member Type

Channel members store the mapping between users and channels. The fields are shown below:

name	type	description	default	optional
channel_type	string	the type of channel
channel_id	string	the id of the channel
user_id	string	the id of the user
channel_role	string	the role of the member in channel (channel_member, channel_moderator or any custom role)	channel_member	✓
created_at	string	the time the member was created in RFC3339 format	the import time	✓
last_read	string	the time when the member last read the channel in RFC3339 format		✓

{
  "type": "member",
  "item": {
    "channel_type": "livestream",
    "channel_id": "6e693c74-262d-4d3d-8846-686364c571c8",
    "user_id": "aad24491-c286-4169-bbfc-9280de419cb6",
    "channel_role": "channel_member",
    "created_at": "2017-02-01T02:00:00Z"
  }
}

Message Type

name	type	description	default	optional
id	string	the id of the message	A random UUIDv4
channel_type	string	the type of channel for this message
channel_id	string	the id of the channel for this message
type	string	the type of the message, regular, deleted, system or blocked	regular	✓
user	string	the id of the user that posted the message
attachments	list of attachments	list of message attachments, see the attachment section below	[]	✓
parent_id	string	the id of the parent message (if the message is a reply)	null	✓
created_at	string	message creation time in RFC3339 format
updated_at	string	last time the message was updated	created_at	✓
deleted_at	string	message deletion time in RFC3339 format	null	✓
show_in_channel	bool	reply messages are only shown inside the message thread unless show_in_channel is set to true, in that case the message is shown in the channel as well	false	✓
*	string/list/object	Add as many custom fields as needed		✓

{
 "type": "message",
 "item": {
  "id": "977e691a-c091-4e3b-8f70-ba8944a3e500",
  "channel_type": "livestream",
  "channel_id": "6e693c74-262d-4d3d-8846-686364c571c8",
  "user":"aad24491-c286-4169-bbfc-9280de419cb6",
  "text": "Such a great song, check out my solo at 2:25",
  "type": "regular",
  "created_at": "2017-02-01T02:00:00Z",
  "attachments": [
   {
    "type": "image",
    "asset_url": "https://my.domain.com/awesome/image.jpg"
    "image_url": "https://my.domain.com/awesome/image.jpg"
     "migrate_resources": true
   }
  ]
 }
}

The only required field of an attachment is “type”. All other fields are optional, and you can add as many custom fields as you’d like. The attachments are a great way to extend Stream’s functionality. If you want to have a custom product attachment, location attachment, checkout, etc., attachments are the way to go. The fields below are automatically picked up and shown by our component libraries.

We support the migration of attachments to our CDN, you should use the the field migrate_resources and set it to true.

Reaction Type

The reaction type has the following fields:

name	type	description	optional
message_id	string	The id of the message
type	string	The type of reaction (up to you to define the types, it’s a string)
user_id	string	The ID of the user
created_at	string	The creation time in RFC3339
*	string/list/object	Add as many custom fields as needed	✓

{
  "type": "reaction",
  "item": {
    "message_id": "977e691a-c091-4e3b-8f70-ba8944a3e500",
    "type": "love",
    "user_id": "aad24491-c286-4169-bbfc-9280de419cb6",
    "created_at": "2019-03-02T15:00:00Z"
  }
}

Error Messages

When problems occur during analyzing, they will show up in the dashboard. A list of errors will be shown in JSON format. Where applicable, the offending item will be included, for example:

{
  "errors": [
    {
      "error": "Validation error: max channelID length exceeded (64)",
      "item_type": "channel",
      "item": {
        "id": "waytoolongwaytoolongwaytoolongwaytoolongwaytoolongwaytoolongwaytoolong",
        "type": "messaging",
        "created_by": "userA-7D3CA510-CB3C-479E-B5FA-69FC2D48410F",
        "created_at": "0001-01-01T00:00:00Z",
        "updated_at": "0001-01-01T00:00:00Z"
      }
    }
  ],
  "stats": {
    "total": {
      "messages": 0,
      "members": 0,
      "reactions": 0,
      "channels": 1,
      "users": 0
    }
  }
}

Error	Description
Validation error: max ” *field *” length exceeded (” *field-length” *)	Maximum length of field exceeded
Validation error: either channel.id or channel.member_ids should be provided, but not both	Either define channel as a regular channel or a distinct channel, but not both
Validation error: channel.id or channel.member_ids required, but not both	At least on of either channel_id member_ids should be provided
Validation error: “field” required	Missing required field
Validation error: createdby user “_username” doesn’t exist (channel “messaging:abc”). please include all users as separate user entries	All users referenced by all objects, for example in channel.created_by_id, should be included in the import file.
Parse error: invalid item type “foobar”	An item was included with an invalid item type, only: user, channel, member, message & reaction are allowed.
Parse error: invalid character ’,’ looking for beginning of value	The import contains malformed json

Importing With CLI Tool

Stream CLI allows you to manage your Stream Chat Imports easily and validate your imports quickly before uploading them for processing. Please, refer to our CLI docs to learn more about import.

Testing Push Notifications

Sendbird Migration