# Video & Audio filters

Apply background blur and replacement filters using `BackgroundFiltersProvider` and `useBackgroundFilters()`.

## Best Practices

- Check `isSupported` before showing filter controls.
- Handle `onError` callback - filters auto-disable on failure.
- Monitor `performance.degraded` to warn users about CPU issues.
- Keep audio in parent window when using Document PiP to avoid autoplay restrictions.
- Safari support is experimental - enable with `forceSafariSupport={true}`.

> **Note:** Supported on Chrome 124+, Firefox 124+, Edge 124+. Mobile support coming soon.

## Video Filters

### Step 1 - Enable and initialize the filters

Background filters are provided through the following APIs and components:

- `<BackgroundFiltersProvider />` - a React context provider that will allow you to use the filters API in your application
- `useBackgroundFilters()` - a React hook that will allow you to access the filters API in your application

A basic integration looks like this:

```tsx
import {
  BackgroundFiltersProvider,
  Call,
  StreamCall,
  StreamVideo,
  StreamVideoClient,
} from "@stream-io/video-react-sdk";

export default function App() {
  let client: StreamVideoClient; /* = ... */
  let call: Call; /* = ... */
  return (
    <StreamVideo client={client}>
      <StreamCall call={call}>
        <BackgroundFiltersProvider
          backgroundFilter="blur" // initial filter
          backgroundImages={[
            "https://my-domain.com/bg/random-bg-1.jpg",
            "https://my-domain.com/bg/random-bg-2.jpg",
          ]}
        >
          <MyUILayout />
          <MyBackgroundFilterSettings /> {/* your settings UI */}
        </BackgroundFiltersProvider>
      </StreamCall>
    </StreamVideo>
  );
}
```

### Step 2 - Use the API to control the filters

Once you have the `BackgroundFiltersProvider` rendered in your application,
you can use the `useBackgroundFilters()` hook to access the filters API and control the behavior of the filters.

```tsx
import { useBackgroundFilters } from "@stream-io/video-react-sdk";

export const MyBackgroundFilterSettings = () => {
  const {
    isSupported, // checks if these filters can run on this device
    isReady, // checks if the filters are ready to be enabled
    isLoading, // indicates that the filters are being loaded
    disableBackgroundFilter, // disables the filter
    applyBackgroundBlurFilter, // applies the blur filter
    applyBackgroundImageFilter, // applies the image filter
    backgroundImages, // list of available images
  } = useBackgroundFilters();

  if (!isSupported) {
    return <div>Background filters are not supported on this device</div>;
  }

  if (!isReady || isLoading) {
    return <div className="my-loading-indicator" />;
  }

  return (
    <div className="my-video-filters">
      <button onClick={disableBackgroundFilter}>Disable</button>
      <button onClick={() => applyBackgroundBlurFilter("high")}>Blur</button>
      <button onClick={() => applyBackgroundBlurFilter("medium")}>Blur</button>
      <button onClick={() => applyBackgroundBlurFilter("low")}>Blur</button>
      <ul>
        {backgroundImages.map((image) => (
          <li key={image}>
            <img src={image} alt="background" />
            <button onClick={() => applyBackgroundImageFilter(image)}>
              Apply background
            </button>
          </li>
        ))}
      </ul>
    </div>
  );
};
```

>
> **Note:** `applyBackgroundBlurFilter` also takes a more fine-grained numeric argument, that specifies the strength of the applied Gaussian blur filter: `applyBackgroundBlurFilter(5.1)`. Note that higher values make the filter more resource intensive.
>

### Step 3 - Handle errors

The performance of the video filter depends on the capabilities of the user device. Even on supported devices, the filter may not work properly, for example, if the device is resource-constrained.

By default, the malfunctioning filter is disabled. You can add the `onError` callback to handle such situations. Use it to display an error notification, to disable the camera completely, etc.

```tsx
import { BackgroundFiltersProvider } from "@stream-io/video-react-sdk";

const call = useCall();

<BackgroundFiltersProvider
  backgroundFilter="blur"
  onError={(error) => {
    console.error(
      "Blur filter encountered an error, camera will be disabled",
      error,
    );
    call?.camera.disable();
  }}
>
  {/* ... */}
</BackgroundFiltersProvider>;
```

### Performance metrics

Background filters run real-time segmentation and may be demanding on some devices.
To help you surface performance issues to your users, the SDK exposes a built-in performance monitor.

```tsx
const { performance } = useBackgroundFilters();

if (performance.degraded) {
  console.log("Background filters degraded:", performance?.reason);
}

return (
  <>
    {performance.degraded && (
      <div className="warning-banner">
        Consider disabling background filters for optimal performance.
      </div>
    )}
    {/* ... */}
  </>
);
```

Degradation reasons include:

- `frame-drop` - background filters are causing drop in frame rate
- `cpu-throttling` - drop in performance due to CPU throttling

The provider uses the camera's source frames as the baseline and calculates a smoothed processed to source frames ratio using an Exponential Moving Average (EMA). This avoids temporary spikes and keeps degradation detection stable and
non-flickery.

### Self-hosting MediaPipe assets (CSP compatibility)

By default, the background blur and replacement filters load their ML model and WebAssembly files at runtime from the [unpkg.com](https://unpkg.com/) CDN. The default `basePath` resolves to:

```text
https://unpkg.com/@stream-io/video-filters-web@<version>/mediapipe
```

This works out of the box, but it can be a problem if your application enforces a strict [Content Security Policy (CSP)](https://developer.mozilla.org/en-US/docs/Web/HTTP/Guides/CSP). You would have to allowlist `unpkg.com` in your `connect-src` (and related) directives, or - preferably - avoid the third-party dependency altogether by hosting the assets on your own domain and pointing the SDK to them.

#### Step 1 - Copy the assets into your app

The assets ship inside the `@stream-io/video-filters-web` package, in its `mediapipe/` folder:

```text
mediapipe/
├── models/  # .tflite segmentation models
└── wasm/    # MediaPipe vision WebAssembly + JS glue code
```

Copy that folder into your app's public/static assets directory, preserving the structure:

```bash
rm -rf public/mediapipe
cp -R node_modules/@stream-io/video-filters-web/mediapipe public/mediapipe
```

The `rm -rf` keeps the command idempotent: without it, re-running `cp -R` nests the source under the existing folder (`public/mediapipe/mediapipe/...`) and leaves stale files behind.

#### Step 2 - Override `basePath`

Set the `basePath` prop on `<BackgroundFiltersProvider>` to the location that serves your copied `mediapipe` folder (the directory that contains `models/` and `wasm/`). Note that the `public/` directory is served from the web root, so files in `public/mediapipe` are exposed at `/mediapipe` (no `/public` prefix):

```tsx
<BackgroundFiltersProvider
  basePath="https://my.domain.com/mediapipe"
  // ...existing config
>
  {/* ... */}
</BackgroundFiltersProvider>
```

A same-origin path works too (e.g. `basePath="/mediapipe"`), which keeps everything within your own `'self'` CSP directive.

#### Keep the assets in sync with the SDK

The hosted assets must match the installed SDK version - mismatched models, WASM, and SDK code can be incompatible. To prevent version drift, copy the folder automatically on install instead of committing a one-off copy. Add a `postinstall` script (or a step in your build) to your `package.json`.

```json
{
  "scripts": {
    "postinstall": "node -e \"require('fs').cpSync('node_modules/@stream-io/video-filters-web/mediapipe', 'public/mediapipe', { recursive: true })\""
  }
}
```

Every install then refreshes `public/mediapipe` from whatever SDK version is installed.

>
> **Note:** If you opt into the [legacy filter](#legacy-api) (`useLegacyFilter`), it loads a TensorFlow Lite model and WASM from a separate `tf/` folder in the same package (default `https://unpkg.com/@stream-io/video-filters-web@<version>/tf`). The same `basePath` prop is reused, so to self-host the legacy filter, copy `node_modules/@stream-io/video-filters-web/tf` and point `basePath` at it instead - or use the `tfFilePath` and `modelFilePath` props for fine-grained control.
>

Once the assets are self-hosted and `basePath` is set, the app no longer makes requests to `unpkg.com`, so you can remove it from your CSP allowlist.

### Legacy API

For compatibility reasons, we still provide the original background filter implementation.
This uses our previous segmentation model and pipeline instead of the new MediaPipe-based implementation.

You can opt-in by enabling the `useLegacyFilter` flag:

```tsx
import { BackgroundFiltersProvider } from "@stream-io/video-react-sdk";

const call = useCall();

<BackgroundFiltersProvider useLegacyFilter>
  {/* ... */}
</BackgroundFiltersProvider>;
```

### Caveats

#### Safari

By default, Safari is excluded from the supported browsers for video filters.
The reason is the fact that Safari applies aggressive timer throttling whenever the tab goes to the background.
This causes the video frame rate to drastically go down or appear completely frozen.

However, we expose a way to still enable the filters on Safari 17.4+, knowing that they may not work as expected in every case:

```tsx
import { BackgroundFiltersProvider } from "@stream-io/video-react-sdk";

<BackgroundFiltersProvider
  forceSafariSupport={true} // enable Safari support
  {...otherProps}
>
  {/* ... */}
</BackgroundFiltersProvider>;
```

## Audio Filters

Check the [`microphone.registerFilter()`](https://getstream.io/video/docs/react/v2/guides/camera-and-microphone/#camera-and-microphone-aiml-filters) API to register custom audio filters.

---

For the most recent version of this documentation, visit [https://getstream.io/video/docs/react/v2/advanced/apply-video-filters/](https://getstream.io/video/docs/react/v2/advanced/apply-video-filters/).