Exporting Data

Exporting Channels

This method allows you to request an export of one or many channels and download all messages and metadata. Channel exports are created asynchronously, you can use the Task ID returned by the APIs to keep track of the status and to download the final result when it is ready.

The data created by the export is following the same format as the regular APIs and is JSON encoded. By default the entire history is included in the export (excluding truncated messages), but you may optionally request the export to include only the messages in a specific interval using the messages_since and messages_until parameters.

Requesting a channel export

Following code example presents basic usecase of exporting channels. By default truncated messages or soft deleted channels are excluded from export. But you can include them by setting the options include_truncated_messages and include_soft_deleted_channels as true.

//Export one channel
const response = await serverClient.exportChannel({
  type: "livestream",
  id: "white-room", 
  messages_since: "2020-11-10T09:30:00.000Z",
  messages_until: "2020-11-10T11:30:00.000Z",
},{
  include_truncated_messages: true,
  include_soft_deleted_channels: true,
});

const taskID = response.task_id;

//Export multiple channels
const response = await serverClient.exportChannels(
 [
  {
   type: "livestream",
   id: "white-room",
   messages_since: "2020-11-10T09:30:00.000Z",
   messages_until: "2020-11-10T11:30:00.000Z",
  },
  {
   type: "livestream",
   id: "white-room2",
  },
 ],
 {
  include_truncated_messages: true,
  include_soft_deleted_channels: true,
 }
);
const taskID = response.task_id;

response = client.export_channel(
  "livestream",
  "white-room",
  messages_since="2020-11-10T09:30:00.000Z",
  messages_until="2020-11-10T11:30:00.000Z",
  include_truncated_messages=True,
  include_soft_deleted_channels=True,
 )
 

task_id = response["task_id"] 

# Export multiple channels
response = client.export_channels(
  [{"type": "livestream",
   "id": "white-room", },
   {"type": "livestream",
   "id": "white-room2", }]
)

task_id = response["task_id"]

channel_ids = client.query_channels({}, limit:, offset:)['channels'].pluck('channel').pluck('id')
 channel_map = channel_ids.map { |e| [[:type, 'messaging'], [:id, e]].to_h }
 options = { include_truncated_messages: true }
 response = client.export_channels(*channel_map, **options)

$response = $client->exportChannel([
  'type' => 'livestream',
  'id' => 'white-room',
  'messages_since' => '2020-11-10T09:30:00.000Z',
  'messages_until' => '2020-11-10T11:30:00.000Z'
  ],
  ['include_truncated_messages' => true]
)

$taskId = $response['task_id']

since := time.Date(2020, time.November, 10, 9, 30, 0, 0, time.UTC)
until := time.Date(2020, time.November, 10, 11, 30, 0, 0, time.UTC)
opts := &ExportChannelOptions{IncludeTruncatedMessages: &includeTruncated}

expChannels := []*ExportableChannel{
	{Type: "livestream", ID: "white-room", MessagesSince: &since, MessagesUntil: &until},
}

response, err := c.ExportChannels(ctx, expChannels, opts)

taskId := response.TaskID

var response = await channelClient.ExportChannelsAsync(new ExportChannelRequest
{
  Channels = new[] { new ExportChannelItem
  {
    Id = "white-room",
    Type = "livestream",
    MessagesSince = new DateTimeOffset(2020, 11, 10, 9, 30, 0, TimeSpan.Zero),
    MessagesUntil = new DateTimeOffset(2020, 11, 10, 11, 30, 0, TimeSpan.Zero),
  }},
  IncludeTruncatedMessages = true,
});

var taskId = response.TaskId;

Channel.export()
  .channel(
    ChannelExportRequestObject.builder()
      .type("channel-type")
      .id("channel-id")
      .build())
  .request();

Up to 25 channels can be exported with a single export request.

you can use v2 version of the channel export by adding version:"v2" in the request body which will use the new export for channels where each entity of the export would be exported as line separated json.

const response = await serverClient.exportChannel({ 
  type: "livestream", 
  id: "white-room", 
  messages_since: "2020-11-10T09:30:00.000Z", 
  messages_until: "2020-11-10T11:30:00.000Z", 
},{ 
  version: "v2" 
});

response = client.export_channel(
 "livestream",
 "white-room",
 messages_since="2020-11-10T09:30:00.000Z",
 messages_until="2020-11-10T11:30:00.000Z",
 version="v2",
)

options = { version: 'v2' }
resp = client.export_channels({
                type: 'livestream',
                id: 'white-room',
                messages_since: '2020-11-10T09:30:00.000Z',
                messages_until: '2020-11-10T11:30:00.000Z'
                },
                **options)

$response = $client->exportChannel([
  'type' => 'livestream',
  'id' => 'white-room',
  'messages_since' => '2020-11-10T09:30:00.000Z',
  'messages_until' => '2020-11-10T11:30:00.000Z'
  ],
  ['version' => 'v2']
)

since := time.Date(2020, time.November, 10, 9, 30, 0, 0, time.UTC)
until := time.Date(2020, time.November, 10, 11, 30, 0, 0, time.UTC)
opts := &ExportChannelOptions{Version: "v2"}

expChannels := []*ExportableChannel{
	{Type: "livestream", ID: "white-room", MessagesSince: &since, MessagesUntil: &until},
}

response, err := c.ExportChannels(ctx, expChannels, opts)

var response = await channelClient.ExportChannelsAsync(new ExportChannelRequest
{
  Channels = new[] { new ExportChannelItem
  {
    Id = "white-room",
    Type = "livestream",
    MessagesSince = new DateTimeOffset(2020, 11, 10, 9, 30, 0, TimeSpan.Zero),
    MessagesUntil = new DateTimeOffset(2020, 11, 10, 11, 30, 0, TimeSpan.Zero),
  }},
  Version = "v2",
});

Channel.export()
  .channel(
    ChannelExportRequestObject.builder()
      .type("channel-type")
      .id("channel-id")
      .build())
  .version("v2")
  .request();

Retrieving the status of the export

You can check the status of an export request using the task ID returned when the task was created. The result of the task contains the URL to the JSON file.

The URL to the export file has an expiration of 24-hours. The link is generated every time you request the export status. The export will be available for 60 days.

const response = await serverClient.getExportChannelStatus(taskId);

console.log(response.status);    // the status for this task
console.log(response.result);    // the result object, only present if the task is completed
console.log(response.result.url);  // the link to the JSON export
console.log(response.error);    // if not null the description of the task error

response = client.get_export_channel_status(task_id)

print(response["status"]);     # the status for this task
print(response["result"]);     # the result object, only present if the task is completed
print(response["result"]["url"]);  # the link to the JSON export
print(response["error"]);      # if not null the description of the task error

response = client.get_export_channel_status(task_id)

puts response['status']     # the status for this task
puts response['result']     # the result object, only present if the task is completed
puts response['result']['url']  # the link to the JSON export
puts response['error']      # if not null the description of the task error

$response = $client->getExportChannelStatus($taskId)

echo $response['status']     // the status for this task
echo $response['result']     // the result object, only present if the task is completed
echo $response['result']['url']  // the link to the JSON export
echo $response['error']      // if not null the description of the task error

task, err := c.GetExportChannelsTask(ctx, response.TaskID)

fmt.Println(task.Status)     // the status for this task
fmt.Println(task.Result)     // the result object, only present if the task is completed
fmt.Println(task.Result["url"])  // the link to the JSON export
fmt.Println(task.Error)      // if not null the description of the task error

var response = await channelClient.GetExportChannelsStatusAsync(taskId);

Console.WriteLine(response.Status);   // the status for this task
Console.WriteLine(response.Result);   // the result object, only present if the task is completed
Console.WriteLine(response.Result.Url;  // the link to the JSON export
Console.WriteLine(response.Error);    // if not null the description of the task error

var response = Channel.exportStatus(taskId).request();

System.out.println(response.getStatus());     // the status for this task
System.out.println(response.getResult());     // the result object, only present if the task is completed
System.out.println(response.getResult().getUrl()); // the link to the JSON export
System.out.println(response.getError());      // if not null the description of the task error

Timestamps will be returned in this format: “2021-02-17T08:17:49.745857Z” and are in UTC.

Exporting Users

To export user data, Stream Chat exposes an exportUser method. This method can *only * be called server-side due to security concerns, so please keep this in mind when attempting to make the call.

Below is an example of how to execute the call to export user data:

const data = await client.exportUser(userID);

response = client.export_user("user_id");

response = client.export_user('user_id')

$response = $client->exportUser($user["id"]);

user, err := client.ExportUser(ctx, userID, nil)

await userClient.ExportAsync(user.Id);

User.export(userId).request();

The export will return all data about the user, including:

User ID
Messages
Reactions
Custom Data

Users with more than 10,000 messages will throw an error during the export process. Reach out to https://getstream.io/contact/support/ for an export of a user with >10,000 messages

Migrating To Stream From An In-House Solution Or Another Provider

Rate Limits