Video & Audio filters

Apply background blur and replacement filters using BackgroundFiltersProvider and useBackgroundFilters().

Best Practices

• Check isSupported before showing filter controls.
• Handle onError callback - filters auto-disable on failure.
• Monitor performance.degraded to warn users about CPU issues.
• Keep audio in parent window when using Document PiP to avoid autoplay restrictions.
• Safari support is experimental - enable with forceSafariSupport={true}.

Video Filters

Step 1 - Enable and initialize the filters

Background filters are provided through the following APIs and components:

• <BackgroundFiltersProvider /> - a React context provider that will allow you to use the filters API in your application
• useBackgroundFilters() - a React hook that will allow you to access the filters API in your application

A basic integration looks like this:

import {
  BackgroundFiltersProvider,
  Call,
  StreamCall,
  StreamVideo,
  StreamVideoClient,
} from "@stream-io/video-react-sdk";

export default function App() {
  let client: StreamVideoClient; /* = ... */
  let call: Call; /* = ... */
  return (
    <StreamVideo client={client}>
      <StreamCall call={call}>
        <BackgroundFiltersProvider
          backgroundFilter="blur" // initial filter
          backgroundImages={[
            "https://my-domain.com/bg/random-bg-1.jpg",
            "https://my-domain.com/bg/random-bg-2.jpg",
          ]}
        >
          <MyUILayout />
          <MyBackgroundFilterSettings /> {/* your settings UI */}
        </BackgroundFiltersProvider>
      </StreamCall>
    </StreamVideo>
  );
}

Step 2 - Use the API to control the filters

Once you have the BackgroundFiltersProvider rendered in your application, you can use the useBackgroundFilters() hook to access the filters API and control the behavior of the filters.

import { useBackgroundFilters } from "@stream-io/video-react-sdk";

export const MyBackgroundFilterSettings = () => {
  const {
    isSupported, // checks if these filters can run on this device
    isReady, // checks if the filters are ready to be enabled
    isLoading, // indicates that the filters are being loaded
    disableBackgroundFilter, // disables the filter
    applyBackgroundBlurFilter, // applies the blur filter
    applyBackgroundImageFilter, // applies the image filter
    backgroundImages, // list of available images
  } = useBackgroundFilters();

  if (!isSupported) {
    return <div>Background filters are not supported on this device</div>;
  }

  if (!isReady || isLoading) {
    return <div className="my-loading-indicator" />;
  }

  return (
    <div className="my-video-filters">
      <button onClick={disableBackgroundFilter}>Disable</button>
      <button onClick={() => applyBackgroundBlurFilter("high")}>Blur</button>
      <button onClick={() => applyBackgroundBlurFilter("medium")}>Blur</button>
      <button onClick={() => applyBackgroundBlurFilter("low")}>Blur</button>
      <ul>
        {backgroundImages.map((image) => (
          <li key={image}>
            <img src={image} alt="background" />
            <button onClick={() => applyBackgroundImageFilter(image)}>
              Apply background
            </button>
          </li>
        ))}
      </ul>
    </div>
  );
};

Step 3 - Handle errors

The performance of the video filter depends on the capabilities of the user device. Even on supported devices, the filter may not work properly, for example, if the device is resource-constrained.

By default, the malfunctioning filter is disabled. You can add the onError callback to handle such situations. Use it to display an error notification, to disable the camera completely, etc.

import { BackgroundFiltersProvider } from "@stream-io/video-react-sdk";

const call = useCall();

<BackgroundFiltersProvider
  backgroundFilter="blur"
  onError={(error) => {
    console.error(
      "Blur filter encountered an error, camera will be disabled",
      error,
    );
    call?.camera.disable();
  }}
>
  {/* ... */}
</BackgroundFiltersProvider>;

Performance metrics

Background filters run real-time segmentation and may be demanding on some devices. To help you surface performance issues to your users, the SDK exposes a built-in performance monitor.

const { performance } = useBackgroundFilters();

if (performance.degraded) {
  console.log("Background filters degraded:", performance?.reason);
}

return (
  <>
    {performance.degraded && (
      <div className="warning-banner">
        Consider disabling background filters for optimal performance.
      </div>
    )}
    {/* ... */}
  </>
);

Degradation reasons include:

• frame-drop - background filters are causing drop in frame rate
• cpu-throttling - drop in performance due to CPU throttling

The provider uses the camera's source frames as the baseline and calculates a smoothed processed to source frames ratio using an Exponential Moving Average (EMA). This avoids temporary spikes and keeps degradation detection stable and non-flickery.

Self-hosting MediaPipe assets (CSP compatibility)

By default, the background blur and replacement filters load their ML model and WebAssembly files at runtime from the unpkg.com CDN. The default basePath resolves to:

https://unpkg.com/@stream-io/video-filters-web@<version>/mediapipe

This works out of the box, but it can be a problem if your application enforces a strict Content Security Policy (CSP). You would have to allowlist unpkg.com in your connect-src (and related) directives, or - preferably - avoid the third-party dependency altogether by hosting the assets on your own domain and pointing the SDK to them.

Step 1 - Copy the assets into your app

The assets ship inside the @stream-io/video-filters-web package, in its mediapipe/ folder:

mediapipe/
├── models/  # .tflite segmentation models
└── wasm/    # MediaPipe vision WebAssembly + JS glue code

Copy that folder into your app's public/static assets directory, preserving the structure:

rm -rf public/mediapipe
cp -R node_modules/@stream-io/video-filters-web/mediapipe public/mediapipe

The rm -rf keeps the command idempotent: without it, re-running cp -R nests the source under the existing folder (public/mediapipe/mediapipe/...) and leaves stale files behind.

Step 2 - Override basePath

Set the basePath prop on <BackgroundFiltersProvider> to the location that serves your copied mediapipe folder (the directory that contains models/ and wasm/). Note that the public/ directory is served from the web root, so files in public/mediapipe are exposed at /mediapipe (no /public prefix):

<BackgroundFiltersProvider
  basePath="https://my.domain.com/mediapipe"
  // ...existing config
>
  {/* ... */}
</BackgroundFiltersProvider>

A same-origin path works too (e.g. basePath="/mediapipe"), which keeps everything within your own 'self' CSP directive.

Keep the assets in sync with the SDK

The hosted assets must match the installed SDK version - mismatched models, WASM, and SDK code can be incompatible. To prevent version drift, copy the folder automatically on install instead of committing a one-off copy. Add a postinstall script (or a step in your build) to your package.json.

{
  "scripts": {
    "postinstall": "node -e \"require('fs').cpSync('node_modules/@stream-io/video-filters-web/mediapipe', 'public/mediapipe', { recursive: true })\""
  }
}

Every install then refreshes public/mediapipe from whatever SDK version is installed.

Once the assets are self-hosted and basePath is set, the app no longer makes requests to unpkg.com, so you can remove it from your CSP allowlist.

Legacy API

For compatibility reasons, we still provide the original background filter implementation. This uses our previous segmentation model and pipeline instead of the new MediaPipe-based implementation.

You can opt-in by enabling the useLegacyFilter flag:

import { BackgroundFiltersProvider } from "@stream-io/video-react-sdk";

const call = useCall();

<BackgroundFiltersProvider useLegacyFilter>
  {/* ... */}
</BackgroundFiltersProvider>;

Caveats

Safari

By default, Safari is excluded from the supported browsers for video filters. The reason is the fact that Safari applies aggressive timer throttling whenever the tab goes to the background. This causes the video frame rate to drastically go down or appear completely frozen.

However, we expose a way to still enable the filters on Safari 17.4+, knowing that they may not work as expected in every case:

import { BackgroundFiltersProvider } from "@stream-io/video-react-sdk";

<BackgroundFiltersProvider
  forceSafariSupport={true} // enable Safari support
  {...otherProps}
>
  {/* ... */}
</BackgroundFiltersProvider>;

Audio Filters

Check the microphone.registerFilter() API to register custom audio filters.