Importing Data

Stream offers built-in tooling to help you migrate from your current chat provider while keeping the process smooth.

To import data into Stream, create an import file and upload it either through the dashboard or by using the CLI.

You can refer to the File Format section below for details about the expected format.

To get started, you can download a sample file to familiarize yourself with the expected structure, and then import it using the CLI.

Import with the CLI

1. Install the CLI

Install the Stream CLI:

$ curl -fsSL https://getstream.io/cli.sh | bash

For other installation methods, see the installation guide.

2. Configure Authentication

Log in once per machine, then link your working directory to your app:

$ getstream login
$ getstream init

In CI or scripts, skip login and set STREAM_API_KEY and STREAM_API_SECRET in the environment instead.

3. Upload a file

Once validated, upload your file to start the import. The maximum file size supported for upload is 300 MB. Pass --watch to follow the import task until it finishes:

$ getstream import chat my-data.jsonl --watch

4. Check Import Status

Monitor the status of your import using the task ID printed by the upload:

$ getstream import status 79502357-3f4b-486e-9a78-400a184a1088

To list all imports for your application:

$ getstream import list

For more detailed descriptions of all CLI import commands, see the getstream import reference.

How Imported Data Is Processed

Imports are processed as an upsert: every item in the file is imported — items that don't exist yet are created, and items that already exist are updated with the data from the file.

For items that already exist, the fields marked in the upserted column of each type's field table are overwritten — and this applies to every upserted field, whether or not it is present in your file:

• If the field is present in the file, the existing value is replaced with the value from the file.
• If the field is missing from the file, the existing value is still overwritten — with the field's zero value (empty string, false, empty array, null) or its documented default (for example, role defaults to user and created_at defaults to the import time).
• Custom data is replaced by what the file contains.
• Blocked users (blocked_user_ids) are the exception: they are merged, not replaced. The imported IDs are added to the user's existing blocks (deduplicated). For example, if a user already blocks A and B and the file has them blocking C, they end up blocking A, B, and C.

Import from the Dashboard

File Format

As you prepare your import file, keep the following requirements in mind—otherwise the import may fail during validation.

• File Structure: The file should be generated using the JSON Lines format, where each line in the file should be a valid JSON object. Each object represents one item of type user, device, channel, member, message, reaction, future_channel_ban, poll, or poll_vote that will be imported into your application.

• Item order: Items in the file should be in a specific order, otherwise the validation step will fail. This order makes reference validation more efficient.
  For a Chat import, the items in the file should be defined in the following order:

  • users
  • devices
  • future_channel_ban
  • channels
  • members
  • messages
  • reactions
  • polls
  • poll_votes

• Object References: Every object you reference in the import should either appear as its own object in the file, or the record should already exist in your Stream application.
  For example, if you import a message that references a user_id, your import file may include a separate user with the same ID, or this user should already exist in your application with the same ID.
  This part of the validation process can be tricky, especially with large files. Be cautious with this as it can cause the import to fail.

• Read State: By default, all imported messages will be marked as read. If you provide the last_read timestamp on a member item, then that member’s unread count will be determined based on the amount of messages that have been created after the last_read timestamp.

• Distinct Channels: Distinct channels are channels that are created by providing a list of member IDs rather than a channel ID. Under the hood, our API generates a unique channel ID from the member IDs. To import a distinct channel, include the member_ids field and omit the id field entirely.

• Timestamps: All timestamps must use the same format as the API (RFC 3339), for example: 1985-04-12T23:20:50.52Z.

Object format

As mentioned earlier, each line in the file should be a valid JSON object with the following format:

Here's an example of a valid file:

{"type":"user","data":{"id":"user_001","name":"Jesse","image":"http://getstream.com","created_at":"2017-01-01T01:00:00Z","role":"moderator","invisible":true,"description":"Taj Mahal guitar player at some point"}}
{"type":"device","data":{"id":"device_001","user_id":"user_001","push_provider_type":"firebase","push_provider_name":"firebase"}}
{"type":"channel","data":{"id":"channel_001","type":"messaging","created_by":"user_001","name":"Rock'n Roll Circus"}}
{"type":"member","data":{"channel_type":"messaging","channel_id":"channel_001","user_id":"user_001","is_moderator":true,"created_at":"2017-02-01T02:00:00Z"}}
{"type":"message","data":{"id":"message_001","channel_type":"messaging","channel_id":"channel_001","user":"user_001","text":"Learn how to build a chat app with Stream","type":"regular","created_at":"2017-02-01T02:00:00Z","attachments":[{"type":"video","asset_url":"https://www.youtube.com/watch?v=o-am4BY-dhs","image_url":"https://i.ytimg.com/vi/o-am4BY-dhs/mqdefault.jpg","thumb_url":"https://i.ytimg.com/vi/o-am4BY-dhs/mqdefault.jpg"}]}}
{"type":"reaction","data":{"message_id":"message_001","type":"love","user_id":"user_001","created_at":"2019-03-02T15:00:00Z"}}

Data format

User Type

The user type fields are shown below:

{
  "type": "user",
  "data": {
    "id": "user_001",
    "name": "Jesse",
    "image": "http://getstream.com",
    "created_at": "2017-01-01T01:00:00Z",
    "role": "moderator",
    "invisible": true,
    "teams": ["admins"],
    "teams_role": { "admins": "team_moderator" },
    "description": "Taj Mahal guitar player at some point"
  }
}

{
  "type": "user",
  "data": {
    "id": "user_001",
    "privacy_settings": {
      "delivery_receipts": { "enabled": true },
      "typing_indicators": { "enabled": true },
      "read_receipts": { "enabled": true }
    }
  }
}

{
  "type": "user",
  "data": {
    "id": "user_001",
    "push_preferences": {
      "chat_level": "mentions",
      "disabled_until": "2042-01-01T00:00:01Z"
    }
  }
}

Device Type

Importing devices is the equivalent of registering devices with Stream.

This is useful when migrating from another chat provider to Stream because:

• Users already have devices registered for push notifications
• You want to preserve these registrations so users continue receiving notifications immediately after migration
• Without importing devices, users would miss notifications until they open the app again

The device type fields are shown below:

{
  "type": "device",
  "data": {
    "id": "device_001",
    "user_id": "user_001",
    "created_at": "2019-01-11T02:00:00Z",
    "push_provider_type": "firebase",
    "push_provider_name": "production-firebase-config"
  }
}

Channel Type

The channel type fields are shown below:

// with channel ID
{
  "type": "channel",
  "data": {
    "id": "channel_001",
    "type": "livestream",
    "created_by": "user_001",
    "name": "Rock'n Roll Circus"
  }
}

// with member_ids
{
  "type": "channel",
  "data": {
    "member_ids": ["user_001", "user_002"],
    "type": "livestream",
    "created_by": "user_001",
    "name": "Rock'n Roll Circus"
  }
}

Member Type

Channel members store the mapping between users and channels. The fields are shown below:

{
  "type": "member",
  "data": {
    "channel_id": "channel_001",
    "channel_type": "livestream",
    "user_id": "user_001",
    "channel_role": "channel_member",
    "created_at": "2017-02-01T02:00:00Z"
  }
}

{
  "type": "member",
  "data": {
    "channel_member_ids": ["user_001", "user_002"],
    "channel_type": "livestream",
    "user_id": "user_001",
    "channel_role": "channel_member",
    "created_at": "2017-02-01T02:00:00Z"
  }
}

Message Type

The message type fields are shown below:

{
  "type": "message",
  "data": {
    "id": "message_001",
    "channel_type": "livestream",
    "channel_id": "channel_001",
    "user": "user_001",
    "text": "Such a great song, check out my solo at 2:25",
    "type": "regular",
    "created_at": "2017-02-01T02:00:00Z"
  }
}

Message Attachments

The attachments are a great way to extend Stream's functionality. If you want to have a custom product attachment, location attachment, checkout, etc., attachments are the way to go.
The fields below are automatically picked up and shown by our component libraries.

{"type":"message","data":{...,"attachments":[{"type":"image","image_url":"https://my.domain.com/image.jpg","thumb_url":"https://my.domain.com/image-thumb.jpg"},{"type":"video","asset_url":"https://my.domain.com/video.mp4","thumb_url":"https://my.domain.com/video-thumb.jpg"}]}}

For attachment migration, only image_url, thumb_url and asset_url fields will be migrated to our CDN and the original URL will be replaced with the new one. The files should not be empty. The import will fail if resource migration fails. In the error you can see the URL and message ID for the failed migration.

Reaction Type

The reaction type fields are shown below:

{
  "type": "reaction",
  "data": {
    "message_id": "message_001",
    "type": "love",
    "user_id": "user_001",
    "score": 1,
    "created_at": "2019-03-02T15:00:00Z"
  }
}

Future Channel Ban Type

The future_channel_ban type fields are shown below:

{
  "type": "future_channel_ban",
  "data": {
    "created_by": "user_001",
    "target_id": "user_002",
    "shadow": true,
    "created_at": "2019-03-02T15:00:00Z"
  }
}

Poll Type

Polls are a shared entity: the same poll and poll_vote items are used by both Chat and Feeds. Import the poll first, then its votes and answers as separate poll_vote items. Poll options are nested inline on the poll; their ids are preserved on import so that each poll_vote.option_id lines up.

The poll type fields are shown below:

Each entry in options has the following fields:

{
  "type": "poll",
  "data": {
    "id": "poll_001",
    "name": "Which feature should we build next?",
    "created_by_id": "user_001",
    "voting_visibility": "public",
    "enforce_unique_vote": true,
    "max_votes_allowed": 1,
    "options": [
      { "id": "option_001", "text": "Threads" },
      { "id": "option_002", "text": "Voice messages" },
      { "id": "option_003", "text": "Polls" }
    ],
    "created_at": "2024-01-01T00:00:00Z"
  }
}

Poll Vote Type

A poll_vote represents either a vote (pointing at an option via option_id) or an answer (free-form text via answer_text). The two are mutually exclusive — set is_answer to true and provide answer_text for an answer; otherwise provide option_id for a regular vote. Answers are only accepted when the referenced poll has allow_answers enabled.

The poll_vote type fields are shown below:

{
  "type": "poll_vote",
  "data": {
    "id": "vote_001",
    "poll_id": "poll_001",
    "option_id": "option_001",
    "user_id": "user_002",
    "created_at": "2024-01-01T01:00:00Z"
  }
}

{
  "type": "poll_vote",
  "data": {
    "id": "vote_002",
    "poll_id": "poll_001",
    "is_answer": true,
    "answer_text": "Scheduled messages, please!",
    "user_id": "user_003",
    "created_at": "2024-01-01T01:05:00Z"
  }
}

Validation

We use JSON Schema to define and validate the structure of our data.
The Chat schema files are available here.

These schema files cover approximately 99% of our validation rules. Some validations depend on your specific configuration, such as custom permission policies or custom channel type configurations.

To validate that your data is in the correct format, you can either:

• validate the data on the fly (while the data is generated), or
• validate the data once the file is generated

There are many JSON Schema validators available for different programming languages that you can use to validate your data.

Error Messages

When problems occur during analysis, they will show up in the dashboard. A list of errors will be shown in JSON format. Where applicable, the offending item will be included, for example:

{
  "errors": [
    {
      "error": "Validation error: max channelID length exceeded (64)",
      "item_type": "channel",
      "item": {
        "id": "waytoolongwaytoolongwaytoolongwaytoolongwaytoolongwaytoolongwaytoolong",
        "type": "messaging",
        "created_by": "userA-7D3CA510-CB3C-479E-B5FA-69FC2D48410F",
        "created_at": "0001-01-01T00:00:00Z",
        "updated_at": "0001-01-01T00:00:00Z"
      }
    }
  ],
  "stats": {
    "total": {
      "messages": 0,
      "members": 0,
      "reactions": 0,
      "channels": 1,
      "users": 0
    }
  }
}

This is not an exhaustive list of possible errors, but these are the most common ones.