Importing Data

LAST EDIT Jun 30 2022

If you've just started using Stream, you might want to import data from your infrastructure or previous chat provider. User, Channel, and Message data can all be imported through the Stream Dashboard.

Importing From Dashboard

Copied!

From your dashboard click on the ‘select app’ dropdown and select the app you wish to import chat data for. Then from the ‘chat’ dropdown click ‘Overview’. On the Overview screen click ‘Options’ and select ‘Import Chat Data’.

You are only allowed to import data for apps in development mode.

You will be presented with a modal asking for a JSON file. Please ensure you follow our import format below to format to avoid errors.

The max size of a file we are able to handle in an upload is 300MB please get in touch with support if you need to upload a larger file size.

Once you have uploaded your file, scroll down on the chat overview page to see a ‘Data Import’ section. Here you can view some details about the file upload:

  • The person who uploaded the file

  • The date the file was uploaded

  • The filename that was uploaded

  • The current status of the upload

If your import fails validation on our end you can upload a new file by clicking on the ‘New Import’ button. To view more details of your import click on it within the list shown under ‘Data Imports’. This will show a modal where you can see the size of the import and a preview of the JSON that was uploaded.

What To Expect

Copied!

Once the file validation begins, the status of the job will update to analyzing.

If unsuccessful, the job moves to analyze_failed status including a json with the results to help correct errors. Once corrected, upload a new file, which will create a new job.

If successfully validated, the job moves to waiting_confirmation status. This is when you contact customer support to complete the rest of the import process. A completed status means the data has been imported and will be visible on the dashboard explorer, or through API query.

A file which is error free can be imported quickly, however, it is recommended to allow 2-3 days for this process as it can be difficult to predict errors, especially with large amounts of data.

All imported messages will be marked as read. There is no way to track unread counts for imported messages.
Your import file will be validated for references. That's why all related records must be provided in the same file. For example, a channel can't be imported without its creator user not in the file. This automated validation phase is needed to prevent mistakes.
For distinct channels, don't try to emulate our hashing algorithm or setting an id. If you provide member ids, it will be auto generated deterministically.
For time columns, support format is same with the API, RFC 3339: 1985-04-12T23:20:50.52Z https://www.rfc-editor.org/rfc/rfc3339#section-5.8

Import File Format

Copied!

The import file must be structured as a JSON array of objects. Each object is an item to add to your application (eg. a user, a message, etc.).

Here's an example of an import file:

Import Entries Format

Copied!

Name

Type

Description

type

string

the type of data for this entry. Allowed values are: user, channel, message, reaction, member

item

object

the data for this entry, see below for the format of each type

Item Types

Copied!

An import file can contain entries with any of these types.

Note that you can add custom fields to users, channels, members, messages, attachments and reactions. The limit is 5KB of custom field data per object.

User Type

Copied!

Any user referenced in the import file needs to be included in its entirety, even if it already exists. The user type fields are shown below:

nametypedescriptiondefaultoptional
idstringthe unique id for the user-
created_atstringuser creation time in RFC3339 format-
deleted_atstringuser deletion time in RFC3339 format-
updated_atstringuser update time in RFC3339 format-
rolestringuser role-
invisiblebooleanuser visibility-
teamslist of stringlist of teams the user is part of-
*string/list/objectadd as many custom fields as needed-

Channel Type

Copied!

The channel type fields are shown below:

nametypedescriptiondefaultoptional
idstringthe unique id for the channel, not required if you provide member_ids
typestringthe type of the channel. ie livestream, messaging etc.
created_bystringthe id of the user that created the message
frozenbooleanyou can't add messages to a frozen channelfalse
member_idslistcan be used to generate the channel lD based on member IDs and create distinct channels
*string/list/object add as many custom fields as needed-

Distinct Channels

Copied!

Distinct channels are channels that are defined by their specific members, not by id. In this case the Channel will be defined as:

Any object referencing this channel (including Member objects), will have to reference this channel similarly:

Member Type

Copied!

Channel members store the mapping between users and channels. The fields are shown below:

Message Type

Copied!
nametypedescriptiondefaultoptional
idstringthe id of the messageA random UUIDv4
channel_typestringthe type of channel for this message
channel_idstringthe id of the channel for this message
typestringthe type of the message, regular, deleted or systemregular
userstringthe id of the user that posted the message
attachmentslist of attachmentslist of message attachments, see the attachment section below[]
parent_idstringthe id of the parent message (if the message is a reply)null
created_atstring

message creation time in RFC3339 format

updated_atstring

last time the message was updated

created_at
deleted_atstring

message deletion time in RFC3339 format

null
show_in_channelbool

reply messages are only shown inside the message thread unless show_in_channel is set to true, in that case the message is shown in the channel as well

false
*string/list/object

Add as many custom fields as needed

Message Attachments

Copied!

The only required field of an attachment is "type". All other fields are optional, and you can add as many custom fields as you'd like. The attachments are a great way to extend Stream's functionality. If you want to have a custom product attachment, location attachment, checkout, etc., attachments are the way to go. The fields below are automatically picked up and shown by our component libraries.

Reaction Type

Copied!

The reaction type has the following fields:

nametypedescriptiondefaultoptional
message_idstringThe id of the message
typestringThe type of reaction (up to you to define the types, it's a string)
user_idstringThe ID of the user
created_atstringThe creation time in RFC3339
*string/list/objectAdd as many custom fields as needed

Error Messages

Copied!

When problems occur during analyzing, they will show up in the dashboard. A list of errors will be shown in JSON format. Where applicable, the offending item will be included, for example:

Error

Description

Validation error: max "field" length exceeded ("field-length")

Maximum length of field exceeded

Validation error: either channel.id or channel.member_ids should be provided, but not both

Either define channel as a regular channel or a distinct channel, but not both

Validation error: channel.id or channel.member_ids required, but not both

At least on of either channel_id member_ids should be provided

Validation error: "field" required

Missing required field

Validation error: created_by user "username" doesn't exist (channel "messaging:abc"). please include all users as separate user entries

All users referenced by all objects, for example in channel.created_by_id, should be included in the import file.

Parse error: invalid item type "foobar"

An item was included with an invalid item type, only: user, channel, member, message & reaction are allowed.

Parse error: invalid character ',' looking for beginning of value

The import contains malformed json