Closed Captions

The Stream API supports adding real-time closed captions (subtitles for participants) to your calls. This guide shows you how to implement this feature on the client side.

The Stream API supports adding real-time closed captioning (subtitles for participants) to your calls. This guide shows you how to build a closed captioning UI using our React SDK.

Prerequisites

Make sure that the closed captioning feature is enabled in your app’s Stream Dashboard. To enable this feature for a specific call type, go to the call type settings and set the Closed Captions Mode to one of the following:

Available: the feature is available for your call and can be turned on.
Auto-on: the feature is available and will be automatically turned on when the call starts.

Starting and stopping closed captions

If the Closed Caption Mode is set to Auto-on, closed caption will start as soon as the call starts. If it’s set to Available, you can start it manually:

await call.startClosedCaptions(); // start closed captions

In both cases, you can stop closed captions by calling:

await call.stopClosedCaptions(); // stop closed captions

The user must have permission to start or stop closed captioning.

Note that starting and stopping closed captioning affects all call participants. When closed captions are running, all call participants receive them.

If you want to enable or disable closed captioning for each participant individually, consider making it Auto-on for the call, but rendering captions conditionally based on each participant’s client-side preference.

Displaying closed captions

You can access the most recent captions using the call state:

import { CallClosedCaption } from "@stream-io/video-client";

const subscription = call.state.closedCaptions$.subscribe((captions) =>
  updateDisplayedCaptions(captions),
);

const updateDisplayedCaptions = (captions: CallClosedCaption[]) => {
  const innerHTML = captions
    .map((caption) => `<b>${caption.user.name}:</b> ${caption.text}`)
    .join("<br>");
};

subscription.unsubscribe(); // remember to unsubscribe

By default, this state holds two most recent captions. This behavior can be further tweaked.

This is what a sample closed caption may look like:

{
  "text": "Thank you, guys, for listening.",
  "// When did the speaker start speaking": "",
  "start_time": "2024-09-25T12:22:21.310735726Z",
  "// When did the speaker finish saying the caption": "",
  "end_time": "2024-09-25T12:22:24.310735726Z",
  "speaker_id": "zitaszuperagetstreamio",
  "user": {
    "id": "zitaszuperagetstreamio",
    "name": "Zita",
    "role": "host",
    "// other user properties": ""
  }
}

Check if closed captions are running

It’s always possible to check if closed captions are running:

const call = client.call(type, id);
const isCaptioningInProgress = call.state.captioning;

console.log(
  `Closed captions are ${isCaptioningInProgress ? "enabled" : "disabled"}`,
);

// alternatively, you can listen to the captioning state changes
call.state.captioning$.subscribe((isCaptioningInProgress) => {
  console.log(
    `Closed captions are ${isCaptioningInProgress ? "enabled" : "disabled"}`,
  );
});

Advanced usage

If the default behavior is not enough, there are several ways to tweak it.

Override the default Close Caption Mode

It’s possible to override the default Close Caption Mode of the call type when creating a call:

await call.getOrCreate({
  data: {
    settings_override: {
      transcription: {
        mode: "available",
        closed_caption_mode: "available",
      },
    },
  },
});

Tweak visibility settings

By default, we keep a maximum of two captions visible, and each caption is visible for a maximum duration of 2.7 seconds (unless it is pushed out earlier by newer captions). This allows for a nice real-time experience while making sure there’s enough time to read each caption.

The settings can be tweaked further:

call.updateClosedCaptionSettings({
  visibilityDurationMs: 2700, // maximum duration a caption can stay visible
  maxVisibleCaptions: 2, // maximum number of captions visible at one time
});

Be careful when setting both visibilityDurationMs and maxVisibleCaptions to zero: this means that captions will be kept in call state indefinetely.

See it in action

To see it all in action check out our TypeScript sample application on GitHub or in Codesandbox.

Transcriptions

Troubleshooting