# Video & Audio Filters ## Video Filters Some apps allow filters to be applied to the current user's video, such as blurred or virtual background, AR elements (glasses, moustaches etc) or color filters (such as sepia, bloom etc). Our SDK offers background blurring and virtual backgrounds out of the box and also has support for injecting your custom filter into the calling experience. ### AI Filters We ship two AI video filters, _background blur_ and _virtual background_, in a separate artifact. To use these filters, add the video filter library as a dependency: ```groovy dependencies { implementation("io.getstream:stream-video-android-filters-video:$stream_version") } ``` #### Background Blur Use the `BlurredBackgroundVideoFilter` class to add a background blur effect to the video feed. Pass an instance of it to the `videoFilter` call property. ```kotlin call.videoFilter = BlurredBackgroundVideoFilter() ``` The `BlurredBackgroundVideoFilter` class has the following parameters: | Parameter | Description | Default Value | | --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------- | | `blurIntensity` | Intensity of the blur effect. Three options available in the `BlurIntensity` enum: `LIGHT`, `MEDIUM`, `HEAVY`. | `BlurIntensity.MEDIUM` | | `foregroundThreshold` | Confidence threshold for the foreground. Pixels with a confidence value greater than or equal to this threshold are considered to be in the foreground. Coerced between 0 and 1, inclusive. | `0.99999` | #### Virtual Backgrounds Pass an instance of `VirtualBackgroundFilter` to the `videoFilter` call property to set a custom image as the participant's background. ```kotlin call.videoFilter = VirtualBackgroundVideoFilter(context, R.drawable.background_image) ``` The `VirtualBackgroundFilter` class has the following parameters: | Parameter | Description | Default Value | | --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------- | | `context` | Context used to access resources. | None | | `backgroundImage` | The drawable resource ID of the image to be used as a virtual background. | None | | `foregroundThreshold` | Confidence threshold for the foreground. Pixels with a confidence value greater than or equal to this threshold are considered to be in the foreground. Coerced between 0 and 1, inclusive. | `0.99999` | ## Adding Your Custom Filter You can inject your own filter through the `Call.videoFilter` property. You will receive each frame of the user's local video, so you have complete freedom over the image processing that you perform. There are two types of filters that you can create: - By extending the `BitmapVideoFilter` abstract class. This gives you a bitmap for each video frame, which you can manipulate directly. ```kotlin abstract class BitmapVideoFilter : VideoFilter() { fun applyFilter(videoFrameBitmap: Bitmap) } ``` You need to manipulate the original bitmap instance to apply the filters. You can of course create a new bitmap in the process, but you need to draw it on the `videoFrameBitmap` instance you get in the `filter` callback. > > **Caution:** **Important:** Do NOT call `bitmap.recycle()` on the `videoFrameBitmap` parameter. The SDK manages the bitmap lifecycle internally. Recycling it will cause black frames or crashes. > - By extending the `RawVideoFilter` abstract class. This gives you direct access to the `VideoFrame` WebRTC object. ```kotlin abstract class RawVideoFilter : VideoFilter() { abstract fun applyFilter( videoFrame: VideoFrame, surfaceTextureHelper: SurfaceTextureHelper, ): VideoFrame } ``` > > **Note:** `BitmapVideoFilter` is less performant than `RawVideoFilter` due to the overhead of certain operations, like _YUV ↔ ARGB_ conversions. > ### Example: Black & White Filter We can create and set a simple black and white filter like this: ```kotlin call.videoFilter = object : BitmapVideoFilter() { override fun applyFilter(videoFrameBitmap: Bitmap) { val c = Canvas(videoFrameBitmap) val paint = Paint() val cm = ColorMatrix() cm.setSaturation(0f) val f = ColorMatrixColorFilter(cm) paint.colorFilter = f c.drawBitmap(videoFrameBitmap, 0f, 0f, paint) } } ``` ## Audio Filters The Stream Video SDK also supports custom audio processing of the local track. This opens up possibilities for custom echo filtering, voice changing or other audio effects. If you want to do custom audio processing, you need to provide your own implementation of the `AudioFilter` interface to `Call.audioFilter`. The `AudioFilter` interface is defined like this: ```kotlin interface AudioFilter { /** * Invoked after an audio sample is recorded. Can be used to manipulate * the ByteBuffer before it's fed into WebRTC. Currently the audio in the * ByteBuffer is always PCM 16bit and the buffer sample size is ~10ms. * * @param audioFormat format in android.media.AudioFormat */ fun applyFilter(audioFormat: Int, channelCount: Int, sampleRate: Int, sampleData: ByteBuffer) } ``` ### Example: Robot Voice In the following example, we will build a simple audio filter that gives the user's voice a robotic touch. ```kotlin // We assume that you already have a call instance (call is started) // Create a simple filter (pitch modification) and assign it to the call call.audioFilter = object : AudioFilter { override fun applyFilter(audioFormat: Int, channelCount: Int, sampleRate: Int, sampleData: ByteBuffer) { // You can modify the pitch factor to achieve a bit different effect val pitchShiftFactor = 0.8f val inputBuffer = audioBuffer.duplicate() inputBuffer.order(ByteOrder.LITTLE_ENDIAN) // Set byte order for correct handling of PCM data val numSamples = inputBuffer.remaining() / 2 // Assuming 16-bit PCM audio val outputBuffer = ByteBuffer.allocate(inputBuffer.capacity()) outputBuffer.order(ByteOrder.LITTLE_ENDIAN) for (channel in 0 until numChannels) { val channelBuffer = ShortArray(numSamples) inputBuffer.asShortBuffer().get(channelBuffer) for (i in 0 until numSamples) { val originalIndex = (i * pitchShiftFactor).toInt() if (originalIndex >= 0 && originalIndex < numSamples) { outputBuffer.putShort(channelBuffer[originalIndex]) } else { // Fill with silence if the index is out of bounds outputBuffer.putShort(0) } } } outputBuffer.flip() audioBuffer.clear() audioBuffer.put(outputBuffer) audioBuffer.flip() } } ``` This is a simple algorithm that just does index shifting. For a more complex scenario you can use a voice processing library. The important part is that you update the `channelBuffer` with the filtered values. --- For the most recent version of this documentation, visit [https://getstream.io/video/docs/android/advanced/apply-video-filters/](https://getstream.io/video/docs/android/advanced/apply-video-filters/).