Custom Video Filters with React Native Community CLI

Learn how to create custom video filters in a React Native CLI app, using a grayscale filter as an example.

Step 1 - Add your custom filter natively in Android and iOS

Create a video filter by implementing VideoFrameProcessorFactoryInterface from @stream-io/react-native-webrtc. Example rotation filter:

RotationFilterFactory.kt

import com.oney.WebRTCModule.videoEffects.VideoFrameProcessor
import com.oney.WebRTCModule.videoEffects.VideoFrameProcessorFactoryInterface
import org.webrtc.VideoFrame

class RotationFilterFactory : VideoFrameProcessorFactoryInterface {
    override fun build(): VideoFrameProcessor {
        return VideoFrameProcessor { frame, textureHelper ->
            VideoFrame(
                frame.buffer.toI420(),
                180, // apply rotation to the video frame
                frame.timestampNs
            )
        }
    }
}

For easier Bitmap processing, use VideoFrameProcessorWithBitmapFilter from @stream-io/video-filters-react-native. Extend BitmapVideoFilter to receive a Bitmap for each frame that you can manipulate directly.

Note:

BitmapVideoFilter is less performant due to YUV ↔ ARGB conversion overhead.

Example: grayscale video filter

Create a grayscale filter by extending BitmapVideoFilter:

GrayScaleVideoFilterFactory.kt

import android.graphics.Bitmap
import android.graphics.Canvas
import android.graphics.ColorMatrix
import android.graphics.ColorMatrixColorFilter
import android.graphics.Paint
import com.oney.WebRTCModule.videoEffects.VideoFrameProcessor
import com.oney.WebRTCModule.videoEffects.VideoFrameProcessorFactoryInterface
import com.streamio.videofiltersreactnative.common.BitmapVideoFilter
import com.streamio.videofiltersreactnative.common.VideoFrameProcessorWithBitmapFilter

class GrayScaleVideoFilterFactory : VideoFrameProcessorFactoryInterface {
  override fun build(): VideoFrameProcessor {
    return VideoFrameProcessorWithBitmapFilter {
      GrayScaleFilter()
    }
  }
}
private class GrayScaleFilter : BitmapVideoFilter() {
    override fun applyFilter(videoFrameBitmap: Bitmap) {
        val canvas = Canvas(videoFrameBitmap)
        val paint = Paint().apply {
            val colorMatrix = ColorMatrix().apply {
                // map the saturation of the color to grayscale
                setSaturation(0f)
            }
            colorFilter = ColorMatrixColorFilter(colorMatrix)
        }
        canvas.drawBitmap(videoFrameBitmap, 0f, 0f, paint)
    }
}

Create an object conforming to VideoFrameProcessorDelegate protocol from @stream-io/video-filters-react-native that inherits from NSObject.

For easier CIImage processing, copy VideoFilters.swift into your app. The VideoFilter class receives each frame as CIImage and converts it to an output CIImage, giving you full control over the processing pipeline. For raw video frame access, adapt the VideoFilter class implementation.

Import the necessary headers in the bridging header file (Xcode offers to create one when adding your first Swift file):

MyApp-Bridging-Header.h

#import "ProcessorProvider.h"

Example: grayscale video filter

Create a grayscale filter by extending VideoFilter:

GrayScaleVideoFrameProcessor.swift

import Foundation

final class GrayScaleVideoFrameProcessor: VideoFilter {
    @available(*, unavailable)
    override public init(
        filter: @escaping (Input) -> CIImage
    ) { fatalError() }

    init() {
        super.init(
            filter: { input in
                let filter = CIFilter(name: "CIPhotoEffectMono")
                filter?.setValue(input.originalImage, forKey: kCIInputImageKey)

                let outputImage: CIImage = filter?.outputImage ?? input.originalImage
                return outputImage
            }
        )
    }
}

Step 2 - Register this filter in a Turbo Native Module

Expose a registerVideoFilters method from a Turbo Native Module in your app. The method adds the filter to ProcessorProvider, which takes no time, so it runs synchronously.

Define the module spec

Create the typed spec that Codegen uses to generate the native interfaces:

specs/NativeVideoEffectsModule.ts

import { TurboModuleRegistry, type TurboModule } from "react-native";

export interface Spec extends TurboModule {
  registerVideoFilters(): boolean;
}

export default TurboModuleRegistry.getEnforcing<Spec>("VideoEffectsModule");

Info:

The method returns a boolean rather than void. Turbo Native Module methods that return void run asynchronously, and the filter must be registered before you apply it.

Configure Codegen in your app's package.json. If your app already has a codegenConfig, put the spec in its jsSrcsDir and add the modulesProvider entry to it:

package.json

"codegenConfig": {
  "name": "AppSpecs",
  "type": "modules",
  "jsSrcsDir": "specs",
  "android": {
    "javaPackageName": "com.example.myapp.videoeffects"
  },
  "ios": {
    "modulesProvider": {
      "VideoEffectsModule": "VideoEffectsModule"
    }
  }
}

Implement the module natively

Extend the generated NativeVideoEffectsModuleSpec class and register the filter with ProcessorProvider:

VideoEffectsModule.kt

package com.example.myapp.videoeffects

import com.facebook.react.bridge.ReactApplicationContext
import com.oney.WebRTCModule.videoEffects.ProcessorProvider
// Example import path based on a typical project structure. Update this to match your project's package name and directory structure.
import com.example.myapp.videofilters.GrayScaleVideoFilterFactory

class VideoEffectsModule(reactContext: ReactApplicationContext) :
  NativeVideoEffectsModuleSpec(reactContext) {

  override fun registerVideoFilters(): Boolean {
    ProcessorProvider.addProcessor("grayscale", GrayScaleVideoFilterFactory())
    return true
  }

  companion object {
    const val NAME = NativeVideoEffectsModuleSpec.NAME
  }
}

Create a package that provides the module:

VideoEffectsPackage.kt

package com.example.myapp.videoeffects

import com.facebook.react.BaseReactPackage
import com.facebook.react.bridge.NativeModule
import com.facebook.react.bridge.ReactApplicationContext
import com.facebook.react.module.model.ReactModuleInfo
import com.facebook.react.module.model.ReactModuleInfoProvider

class VideoEffectsPackage : BaseReactPackage() {
  override fun getModule(name: String, reactContext: ReactApplicationContext): NativeModule? =
    if (name == VideoEffectsModule.NAME) {
      VideoEffectsModule(reactContext)
    } else {
      null
    }

  override fun getReactModuleInfoProvider() = ReactModuleInfoProvider {
    mapOf(
      VideoEffectsModule.NAME to ReactModuleInfo(
        name = VideoEffectsModule.NAME,
        className = VideoEffectsModule.NAME,
        canOverrideExistingModule = false,
        needsEagerInit = false,
        isCxxModule = false,
        isTurboModule = true
      )
    )
  }
}

Add the package to the package list in MainApplication.kt:

MainApplication.kt

PackageList(this).packages.apply {
  add(VideoEffectsPackage())
}

Write the implementation in Swift. Declare the class and its methods public and expose them to Objective-C:

VideoEffectsModuleImpl.swift

import Foundation

@objcMembers public class VideoEffectsModuleImpl: NSObject {
  public func registerVideoFilters() {
    ProcessorProvider.addProcessor(GrayScaleVideoFrameProcessor(), forName: "grayscale")
  }
}

Swift can't implement the generated C++ interface directly, so add a thin Objective-C++ adapter that conforms to the generated NativeVideoEffectsModuleSpec protocol and forwards to Swift:

VideoEffectsModule.h

#import <Foundation/Foundation.h>
#import <AppSpecs/AppSpecs.h>

NS_ASSUME_NONNULL_BEGIN

@interface VideoEffectsModule : NSObject <NativeVideoEffectsModuleSpec>

@end

NS_ASSUME_NONNULL_END

VideoEffectsModule.mm

#import "VideoEffectsModule.h"
// This file is compiled without modules, so the Swift header's @imports are skipped.
// Import the Objective-C headers for the types it references first.
#import <React-RCTAppDelegate/RCTDefaultReactNativeFactoryDelegate.h>
#import "ProcessorProvider.h"
// Replace MyApp with your app target's name.
#import "MyApp-Swift.h"

@implementation VideoEffectsModule {
  VideoEffectsModuleImpl *_impl;
}

- (instancetype)init
{
  if (self = [super init]) {
    _impl = [VideoEffectsModuleImpl new];
  }
  return self;
}

- (NSNumber *)registerVideoFilters
{
  [_impl registerVideoFilters];
  return @YES;
}

- (std::shared_ptr<facebook::react::TurboModule>)getTurboModule:
    (const facebook::react::ObjCTurboModule::InitParams &)params
{
  return std::make_shared<facebook::react::NativeVideoEffectsModuleSpecJSI>(params);
}

+ (NSString *)moduleName
{
  return @"VideoEffectsModule";
}

@end

MyApp-Swift.h declares every Objective-C-visible Swift class in your app, including the template's AppDelegate types. If the build reports an unknown type or protocol in it, import that type's framework header before it (for example, <UserNotifications/UserNotifications.h> when your AppDelegate adopts UNUserNotificationCenterDelegate).

The modulesProvider entry in package.json registers the adapter, so it doesn't need RCT_EXPORT_MODULE. Run bundle exec pod install in the ios folder to generate the Codegen files.

Info:

When calling addProcessor, provide a filter name (e.g., grayscale) to use later in JavaScript.

Step 3 - Apply the video filter in JavaScript

Call mediaStreamTrack._setVideoEffect(name) to apply the filter. Use disableAllFilters from useBackgroundFilters() to disable filters. Example hook (media stream is in the Call instance from useCall):

import {
  useBackgroundFilters,
  useCall,
} from "@stream-io/video-react-native-sdk";
import { useRef, useCallback, useState } from "react";

import { MediaStream } from "@stream-io/react-native-webrtc";

// Update the path to match where you placed the spec.
import NativeVideoEffectsModule from "../specs/NativeVideoEffectsModule";

type CustomFilters = "GrayScale" | "MyOtherCustomFilter";

export const useCustomVideoFilters = () => {
  const call = useCall();
  const isFiltersRegisteredRef = useRef(false);
  const { disableAllFilters } = useBackgroundFilters();
  const [currentCustomFilter, setCustomFilter] = useState<CustomFilters>();

  const applyGrayScaleFilter = useCallback(() => {
    if (!isFiltersRegisteredRef.current) {
      // registering is needed only once per the app's lifetime
      NativeVideoEffectsModule.registerVideoFilters();
      isFiltersRegisteredRef.current = true;
    }
    disableAllFilters(); // disable any other filter
    (call?.camera.state.mediaStream as MediaStream | undefined)
      ?.getVideoTracks()
      .forEach((track) => {
        track._setVideoEffect("grayscale"); // set the grayscale filter
      });
    setCustomFilter("GrayScale");
  }, [call, disableAllFilters]);

  const disableCustomFilter = useCallback(() => {
    disableAllFilters();
    setCustomFilter(undefined);
  }, [disableAllFilters]);

  return {
    currentCustomFilter,
    applyGrayScaleFilter,
    disableCustomFilter,
  };
};

Call applyGrayScaleFilter while in a call:

The grayscale filter implementations are available in our React Native CLI sample app.