Importing Data

If you’re switching to Stream from a different provider or you had an in-house chat solution, you are going to want migrate your data.

Here we’ll walk through the options you have to format, validate, and upload your imports. Please follow the formatting guidelines carefully. Some of the formatting may seem redundant, but this is done to maintain the integrity of the data within your app.

All data must be in uncompressed JSON files and can be no larger than 300 MB. A fully validated file will import quickly, but please plan for the whole migration process to take about a week as it can be difficult to predict errors, especially with large data sets.

Importing From Dashboard

From your dashboard select the app you wish to import chat data for. Then click on ‘App Settings’ in the top-right corner of your screen, and click ‘Import Chat Data’

You will be presented with a modal asking for a JSON file. Please ensure you follow the import format described below to avoid errors.

Once you have uploaded your file, scroll down on the chat overview page to see a ‘Data Import’ section. Here you can view some details about the file upload:

• The person who uploaded the file
• The date the file was uploaded
• The filename that was uploaded
• The current status of the upload

If your import fails validation on our end you can upload a new file by clicking on the ‘New Import’ button. To view more details of your import click on it within the list shown under ‘Data Imports’. This will show a modal where you can see the size of the import and a preview of the JSON that was uploaded.

Import Flow

• Validation: Once the file validation begins, the status of the import job will update to analyzing .
• Failure: If validation is unsuccessful, the job moves to analyze_failed status. JSON with the validation results can be viewed to help correct errors. Once corrected, upload a new file, which will create a new job.
• Success: If successfully validated, the job moves to waiting_confirmation status.
• Confirmation: Our support team will manually confirm the job, and the import will start immediately.
• Completion: A completed status means the data has been imported and will be visible on the dashboard explorer, or through API query.

A file which is error free can be imported quickly. However, it is recommended to allow up to 7 days for the entire migration process, as it can be difficult to predict errors, especially with large amounts of data.

Importing With CLI Tool

Advantages of using CLI:

• Validation: The CLI includes a validator tool that will validate your files locally without having to upload it each time. We recommend using this even if you will manage the uploads through the dashboard as it is significantly quicker to identify and correct data issues.
• Importing: The CLI allows you to select between two upload modes: upsert and insert (described in more detail below). Upon successful file validation, the import will begin automatically. The confirmation step is skipped.
• Managing: Allows you to manage multiple uploads at a time.

For more detailed descriptions of all import methods, please refer to our CLI docs.

For CLI installation instructions, refer here.

Upsert vs Insert Mode

Upsert

The Upsert mode will import all the data on the file, including data that already exists on our system.

• If an item exists, the fields provided will be overwritten.
• Some optional fields if ommitted, may be set to their default value.
• Custom data will be merged.

Insert

The Insert mode will skip import items if they already exist.

• It will check for existence of an item by its id. If it exists it will be skipped, even if the fields provided differ from what exists in the database.
• If it does not exist, the whole object will be inserted.
• This mode is only available on the Stream CLI.

Import File Format

Important to Know

As you prepare your JSON files, there are some very important things to consider, otherwise you will run into errors when trying to import your data.

• File Structure: Your file will be structured as JSON array of objects. Each object will be an item of type user, message, channel, member, or reaction that will be added to your application.

• Object References: Every object you reference in the import must also appear as its own object in the same file. For example, if you import a message that references a user_id and a channel_id, your import file must also include separate user and channel objects with those matching IDs so that validation succeeds. This part of the validation process can be tricky especially with large files. It is recommended to use the valdation tool in the Stream CLI to help debug issues quicker.

• Read State: By default all imported messages will be marked as read. If you provide the last_read timestamp on a member item, then that member’s unread count will be determined based on the amount of messages that have been created after the last_read timestamp.

• Distinct Channels: Distinct channels are channels that are created by providing a list of member ids rather than a channel id. Under the hood, our api will use the member ids to generate a unique id for the channel. You can import distinct channels by setting the member_ids field in the channel item instead of the id field.

• Timestamps: For time columns, support format is same with the API, RFC 3339: 1985-04-12T23:20:50.52Z https://www.rfc-editor.org/rfc/rfc3339#section-5.8

Import Entries Format

Here’s an example of an import file:

[
  {
    "type": "channel",
    "item": {
      "id": "6e693c74-262d-4d3d-8846-686364c571c8",
      "type": "livestream",
      "created_by": "aad24491-c286-4169-bbfc-9280de419cb6",
      "name": "Rock'n Roll Circus"
    }
  },
  {
    "type": "member",
    "item": {
      "channel_type": "livestream",
      "channel_id": "6e693c74-262d-4d3d-8846-686364c571c8",
      "user_id": "aad24491-c286-4169-bbfc-9280de419cb6",
      "is_moderator": true,
      "created_at": "2017-02-01T02:00:00Z"
    }
  },
  {
    "type": "message",
    "item": {
      "id": "977e691a-c091-4e3b-8f70-ba8944a3e500",
      "channel_type": "livestream",
      "channel_id": "6e693c74-262d-4d3d-8846-686364c571c8",
      "user": "aad24491-c286-4169-bbfc-9280de419cb6",
      "text": "Learn how to build a chat app with Stream",
      "type": "regular",
      "created_at": "2017-02-01T02:00:00Z",
      "attachments": [
        {
          "type": "video",
          "asset_url": "https://www.youtube.com/watch?v=o-am4BY-dhs",
          "image_url": "https://i.ytimg.com/vi/o-am4BY-dhs/mqdefault.jpg",
          "thumb_url": "https://i.ytimg.com/vi/o-am4BY-dhs/mqdefault.jpg"
        }
      ]
    }
  },
  {
    "type": "reaction",
    "item": {
      "message_id": "977e691a-c091-4e3b-8f70-ba8944a3e500",
      "type": "love",
      "user_id": "aad24491-c286-4169-bbfc-9280de419cb6",
      "created_at": "2019-03-02T15:00:00Z"
    }
  },
  {
    "type": "user",
    "item": {
      "id": "aad24491-c286-4169-bbfc-9280de419cb6",
      "name": "Jesse",
      "image": "http://getstream.com",
      "created_at": "2017-01-01T01:00:00Z",
      "role": "moderator",
      "invisible": true,
      "description": "Taj Mahal guitar player at some point"
    }
  }
]

Item Types

An import file can contain entries with any of these types.

User Type

Any user referenced in the import file needs to be included in its entirety, even if it already exists. The user type fields are shown below:

{
  "type": "user",
  "item": {
    "id": "aad24491-c286-4169-bbfc-9280de419cb6",
    "name": "Jesse",
    "image": "http://getstream.com",
    "created_at": "2017-01-01T01:00:00Z",
    "role": "moderator",
    "invisible": true,
    "teams": ["admins"],
    "description": "Taj Mahal guitar player at some point"
  }
}

Channel Type

The channel type fields are shown below:

{
  "type": "channel",
  "item": {
    "id": "6e693c74-262d-4d3d-8846-686364c571c8",
    "type": "livestream",
    "created_by": "aad24491-c286-4169-bbfc-9280de419cb6",
    "name": "Rock'n Roll Circus"
  }
}

Distinct Channels

Distinct Channels are channels that are defined by their specific members, not by id. In this case the Channel will be defined as:

{
  "type": "channel",
  "item": {
    "member_ids": ["userA", "userB"],
    "type": "livestream",
    "created_by": "aad24491-c286-4169-bbfc-9280de419cb6",
    "name": "Rock'n Roll Circus"
  }
}

Any object referencing this channel (including Member objects, example in Member Type), will have to reference this channel similarly:

{
  "type": "message",
  "item": {
    "id": "977e691a-c091-4e3b-8f70-ba8944a3e500",
    "channel_type": "livestream",
    "channel_member_ids": ["userA", "userB"],
    "user": "userA",
    "text": "Such a great song, check out my solo at 2:25",
    "type": "regular",
    "created_at": "2017-02-01T02:00:00Z"
  }
}

Member Type

Channel members store the mapping between users and channels. The fields are shown below:

{
  "type": "member",
  "item": {
    "channel_type": "livestream",
    "channel_id": "6e693c74-262d-4d3d-8846-686364c571c8",
    "user_id": "aad24491-c286-4169-bbfc-9280de419cb6",
    "channel_role": "channel_member",
    "created_at": "2017-02-01T02:00:00Z"
  }
}

Example for Distinct Channels

{
  "type": "member",
  "item": {
    "channel_type": "livestream",
    "channel_member_ids": ["userA", "userB"],
    "user_id": "aad24491-c286-4169-bbfc-9280de419cb6",
    "channel_role": "channel_member",
    "created_at": "2017-02-01T02:00:00Z"
  }
}

Message Type

{
  "type": "message",
  "item": {
    "id": "977e691a-c091-4e3b-8f70-ba8944a3e500",
    "channel_type": "livestream",
    "channel_id": "6e693c74-262d-4d3d-8846-686364c571c8",
    "user": "aad24491-c286-4169-bbfc-9280de419cb6",
    "text": "Such a great song, check out my solo at 2:25",
    "type": "regular",
    "created_at": "2017-02-01T02:00:00Z",
    "attachments": [
      {
        "type": "audio",
        "asset_url": "https://my.domain.com/awesome/song.mp3"
      }
    ]
  }
}

Message Attachments

The only required field of an attachment is “type”. All other fields are optional, and you can add as many custom fields as you’d like. The attachments are a great way to extend Stream’s functionality. If you want to have a custom product attachment, location attachment, checkout, etc., attachments are the way to go. The fields below are automatically picked up and shown by our component libraries. thumb_url is optional but recommended for images and videos.

{
  "type": "message",
  "item": {
    ...
    "attachments": [
      {
        "type": "image",
        "image_url": "https://my.domain.com/image.jpg",
        "thumb_url": "https://my.domain.com/image-thumb.jpg"
      },
      {
        "type": "video",
        "asset_url": "https://my.domain.com/video.mp4",
        "thumb_url": "https://my.domain.com/video-thumb.jpg"
      },
      {
        "type": "audio",
        "asset_url": "https://my.domain.com/awesome/song.mp3"
      },
      {
        "type": "file",
        "asset_url": "https://my.domain.com/file.zip"
      },
    ]
  }
}

We support the migration of attachments to our CDN, you should use the field migrate_resources and set it to true. Only image_url, thumb_url, asset_url fields will be migrated to our CDN and the original URL will be replaced with the new one. The URLs should be publicly accessible and the files should not be empty. The import will fail if resource migration fails. In the error you can see the URL and message ID for the failed migration.

{
  "type": "message",
  "item": {
    ...
    "attachments": [
      {
        "type": "image",
        "image_url": "https://my.domain.com/image.jpg",
        "migrate_resources": true
      }
    ]
  }
}

Reaction Type

The reaction type has the following fields:

{
  "type": "reaction",
  "item": {
    "message_id": "977e691a-c091-4e3b-8f70-ba8944a3e500",
    "type": "love",
    "user_id": "aad24491-c286-4169-bbfc-9280de419cb6",
    "created_at": "2019-03-02T15:00:00Z"
  }
}

Device Type

Importing devices is the equivalent of registering devices with Stream

This is useful when migrating from another chat provider to Stream because:

• Users already have devices registered for push notifications
• You want to preserve these registrations so users continue receiving notifications immediately after migration
• Without importing devices, users would miss notifications until they open the app again

The device type has the following fields:

{
  "type": "device",
  "item": {
    "id": "123",
    "user_id": "aad24491-c286-4169-bbfc-9280de419cb6",
    "created_at": "2019-01-11T02:00:00Z",
    "push_provider_type": "firebase",
    "push_provider_name": "production-firebase-config"
  }
}

Error Messages

When problems occur during analyzing, they will show up in the dashboard. A list of errors will be shown in JSON format. Where applicable, the offending item will be included, for example:

{
  "errors": [
    {
      "error": "Validation error: max channelID length exceeded (64)",
      "item_type": "channel",
      "item": {
        "id": "waytoolongwaytoolongwaytoolongwaytoolongwaytoolongwaytoolongwaytoolong",
        "type": "messaging",
        "created_by": "userA-7D3CA510-CB3C-479E-B5FA-69FC2D48410F",
        "created_at": "0001-01-01T00:00:00Z",
        "updated_at": "0001-01-01T00:00:00Z"
      }
    }
  ],
  "stats": {
    "total": {
      "messages": 0,
      "members": 0,
      "reactions": 0,
      "channels": 1,
      "users": 0
    }
  }
}

This is not an exhaustive list of possible errors, but these are the most common ones.