Closed Captions

The Stream API supports adding real-time closed captioning (subtitles for participants) to your calls. This guide shows you how to build a closed captioning UI using our Android SDK.

Prerequisites

Make sure that the closed captioning feature is enabled in your app’s Stream Dashboard. To enable this feature for a specific call type, go to the call type settings and set the Closed Captions Mode to one of the following:

Available: the feature is available for your call and can be turned on.
Auto-on: the feature is available and will be automatically turned on when the call starts.
Disabled: the feature is not available for the current call. In this case, you should hide or disable any UI related to closed captions.

Starting and stopping closed captions

If the Closed Caption Mode is set to Auto-on, closed caption will start as soon as the call starts. If it’s set to Available, you can start it manually:

    call.startClosedCaptions() // start closed captions

In both cases, you can stop closed captions by calling:

    call.stopClosedCaptions() // stop closed captions

The user must have permission to start or stop closed captioning.

Note that starting and stopping closed captioning affects all call participants. When closed captions are running, all call participants receive them.

If you want to enable or disable closed captioning for each participant individually, consider making it Auto-on for the call, but rendering captions conditionally based on each participant’s client-side preference.

Rendering closed captions

The next two sections show you how to render closed captions in your call UI and how to start and stop them.

Closed Captions UI


/**
 * This class named ClosedCaptionUiModel is provided as an example to
 * simplify UI writing logic. This class is NOT part of the StreamVideo SDK and
 * should be implemented in your app as needed.
 */
public data class ClosedCaptionUiModel(val speaker: String, val text: String)

/**
 * This extension function CallClosedCaption.toClosedCaptionUiModel is provided as an example to
 * simplify state handling. This class is NOT part of the StreamVideo SDK and
 * should be implemented in your app as needed.
 */
public fun CallClosedCaption.toClosedCaptionUiModel(call: Call): ClosedCaptionUiModel {
  val participants = call.state.participants.value
  val user = participants.firstOrNull { it.userId.value == this.speakerId }
  return ClosedCaptionUiModel(
    speaker = user?.userNameOrId?.value ?: "N/A",
    text = this.text,
  )
}


/**
 * Render closed captions list UI.
 */

@Composable
public fun ClosedCaptionsListUiContainer(
  call: Call,
  closedCaptionUiState: ClosedCaptionUiState,
) {
  if (closedCaptionUiState == ClosedCaptionUiState.Running && closedCaptions.isNotEmpty()) {
    Box(
      modifier = Modifier
        .fillMaxSize()
        .offset(y = config.yOffset)
        .padding(horizontal = config.horizontalMargin),

      contentAlignment = Alignment.BottomCenter,
    ) {
      ClosedCaptionList(closedCaptions.map { it.toClosedCaptionUiModel(call) })
    }
  }
}

@Composable
public fun ClosedCaptionList(captions: List<ClosedCaptionUiModel>) {
  LazyColumn(
    modifier = Modifier
      .background(
        color = Color.Black.copy(alpha = 0.5),
        shape = RoundedCornerShape(16.dp),
      )
      .fillMaxWidth()
      .padding(12.dp),
    userScrollEnabled = false,
    horizontalAlignment = Alignment.CenterHorizontally,
  ) {
    itemsIndexed(captions.takeLast(config.maxVisibleCaptions)) { index, item ->
      ClosedCaptionUi(item, index != captions.size - 1)
    }
  }
}

@Composable
public fun ClosedCaptionUi(
  closedCaptionUiModel: ClosedCaptionUiModel,
  semiFade: Boolean,
) {
  val alpha = if (semiFade) 0.6f else 1f

  val formattedSpeakerText = closedCaptionUiModel.speaker + ":"

  Row(
    modifier = Modifier.alpha(alpha),
    horizontalArrangement = Arrangement.spacedBy(8.dp),
  ) {
    Text(formattedSpeakerText, color = Color.YELLOW)
    Text(
      closedCaptionUiModel.text,
      color = Color.WHITE,
      modifier = Modifier.wrapContentWidth(),
    )
  }
}

Toggling closed captions

Only users who have permission to start or stop closed captions can toggle closed captions on and off. This can be set up in the Stream Dashboard in the Permissions section.

Here is how you can create a UI that toggles closed captions depending on user permissions:

First we will create a state holder class for easy management of state and will name it as ClosedCaptionUiState. Then we will write actual composeable funcation named as ClosedCaptionsToggleUi in code snippet below which will render the Toggle UI

/**
 * The ClosedCaptionUiState sealed class is provided as an example to
 * simplify state handling. This class is NOT part of the StreamVideo SDK and
 * should be implemented in your app as needed.
 **/
sealed class ClosedCaptionUiState {
    /**
     * Indicates that closed captions are available for the current call but are not actively running/displaying.
     * This state usually occurs when the captioning feature is supported but not yet activated/displayed.
     */
    data object Available : ClosedCaptionUiState()

    /**
     * Indicates that closed captions are actively running and displaying captions during the call.
     */
    data object Running : ClosedCaptionUiState()

    /**
     * Indicates that closed captions are unavailable for the current call.
     * This state is used when the feature is disabled or not supported.
     */
    data object UnAvailable : ClosedCaptionUiState()

    public fun TranscriptionSettingsResponse.ClosedCaptionMode.toClosedCaptionUiState(): ClosedCaptionUiState {
        return when (this) {
            is TranscriptionSettingsResponse.ClosedCaptionMode.Available,
            is TranscriptionSettingsResponse.ClosedCaptionMode.AutoOn,
                ->
                Available
            else ->
                UnAvailable
        }
    }
}

Next we are going to write the UI for toggling the closed captions. Here is how you can do that:

@Composable
fun ClosedCaptionsToggleUi(call: Call) {

  val ccMode by call.state.ccMode.collectAsStateWithLifecycle()
  val captioning by call.state.isCaptioning.collectAsStateWithLifecycle()

  var closedCaptionUiState: ClosedCaptionUiState by remember {
    mutableStateOf(ccMode.toClosedCaptionUiState())
  }

  val updateClosedCaptionUiState: (ClosedCaptionUiState) -> Unit = { newState ->
    closedCaptionUiState = newState
  }
  // UI Click logic
  val onClosedCaptionsClick: () -> Unit = {
    scope.launch {
      when (closedCaptionUiState) {
        is ClosedCaptionUiState.Running -> {
          updateClosedCaptionUiState(ClosedCaptionUiState.Available)
        }
        is ClosedCaptionUiState.Available -> {
          if (captioning) {
            updateClosedCaptionUiState(ClosedCaptionUiState.Running)
          } else {
            call.startClosedCaptions()
          }
        }
        else -> {
          throw Exception(
            "This state $closedCaptionUiState should not invoke any ui operation",
          )
        }
      }
    }
  }

  // Render  closed captions toggle ui based  on UI state.
  return when (closedCaptionUiState) {
    is ClosedCaptionUiState.Available -> {
      Row(
        modifier = Modifier
          .clickable(onClick = onClosedCaptionsClick), verticalAlignment = (Alignment.CenterVertically)
      ) {
        Icon(Icons.Default.ClosedCaptionOff, contentDescription = "Start Closed Captions")
        Text("Start Closed Captions")
      }
    }

    is ClosedCaptionUiState.Running -> {
      Row(
        modifier = Modifier
          .clickable(onClick = onClosedCaptionsClick), verticalAlignment = (Alignment.CenterVertically)
      ) {
        Icon(Icons.Default.ClosedCaption, contentDescription = "Stop Closed Captions")
        Text("Stop Closed Captions")
      }
    }

    is ClosedCaptionUiState.UnAvailable -> {
      Row(
        modifier = Modifier
          .clickable(onClick = onClosedCaptionsClick), verticalAlignment = (Alignment.CenterVertically)
      ) {
        Icon(Icons.Default.ClosedCaptionDisabled, contentDescription = "Closed Captions Unavailable")
        Text("Closed Captions Unavailable")
      }
    }
  }

}

Note that starting and stopping closed captioning affects all call participants. When closed captions are running, all call participants receive them.

And now we can add these newly created components anywhere in our call UI.

Advanced usage

If the default behavior is not enough, there are several ways to tweak it.

Tweak visibility settings

By default, we keep a maximum of two captions visible, and each caption is visible for a maximum duration of 2.7 seconds (unless it is pushed out earlier by newer captions). This allows for a nice real-time experience while making sure there’s enough time to read each caption.


val newClosedCaptionsSettings = ClosedCaptionsSettings(
  visibilityDurationMs = 3000,
  autoDismissCaptions = true,
  maxVisibleCaptions = 3,
)

call.updateClosedCaptionsSettings(newClosedCaptionsSettings)

Be careful when setting both visibilityDurationMs and maxVisibleCaptions to zero: this means that captions will be kept in call state indefinetely.

Build your own logic

Finally, if the behavior provided by the SDK isn’t enough, you can always subscribe to the caption events and build your own logic. Here is how to do it:

You can subscribe to any of the following closed caption events based on your requirement

Events	Description
`ClosedCaptionEvent`	Sent when closed captions are active, used to display them on the call screen.
`ClosedCaptionStartedEvent`	Sent when closed captions start, used to indicate they are active.
`ClosedCaptionEndedEvent`	Sent when closed captions end, used to indicate they have stopped.


val sub = client.subscribeFor<ClosedCaptionEvent> { event ->
  logger.d { event.toString() }
}
// stop listening
sub.dispose()

Public Apis

APIs	Description
`callState.isCaptioning`	Tracks whether closed captioning is currently active for the call.
`callState.closedCaptions`	Holds the current list of closed captions. This list is updated dynamically and contains at most `ClosedCaptionsSettings.maxVisibleCaptions` captions.
`callState.ccMode`	Holds the current closed caption mode for the video call. This object contains information about closed captioning feature availability. Possible values: - `ClosedCaptionMode.Available` - `ClosedCaptionMode.Disabled` - `ClosedCaptionMode.AutoOn` - `ClosedCaptionMode.Unknown`

By combining the Toggle UI and List UI, you can create a robust and customizable Closed Captions experience in your application. The StreamVideo SDK makes it easy to manage state, appearance, and user interactions for Closed Captions seamlessly.

Call Quality Rating

Ringing