Introduction
In this course, we have learned the various concepts and architectures of WebRTC and how it standardizes video calling between various devices and frameworks. Now, it’s our turn to implement it on each framework.
WebRTC on Flutter is usually implemented through the flutter_webrtc library, which has the requisite WebRTC code for all platforms that Flutter supports. The plugin abstracts away several difficult-to-implement parts of WebRTC, and the app built in this article is based on the example code given in the plugin.
In this tutorial, we’ll add a WebRTC-based calling solution to a Flutter application.
Setting Up the flutter_webrtc Plugin
Various components must be set up to facilitate a full video-calling experience. The first one is adding the base WebRTC plugin to your Flutter app. In this lesson, we only focus on Android and iOS, but please note that additional setup may be needed to set up similar experiences on other platforms.
First up, add the plugin to your pubspec.yaml file:
dependencies:
flutter_webrtc: ^0.9.48
For iOS, we need to let the platform know that we will use the microphone and camera for calls through the Info.plist file:
<key>NSCameraUsageDescription</key>
<string>$(PRODUCT_NAME) Camera Usage!</string>
<key>NSMicrophoneUsageDescription</key>
<string>$(PRODUCT_NAME) Microphone Usage!</string>
Similarly, for Android, we declare the same in the AndroidManifest.xml file:
<uses-feature android:name="android.hardware.camera" />
<uses-feature android:name="android.hardware.camera.autofocus" />
<uses-permission android:name="android.permission.CAMERA" />
<uses-permission android:name="android.permission.RECORD_AUDIO" />
<uses-permission android:name="android.permission.ACCESS_NETWORK_STATE" />
<uses-permission android:name="android.permission.CHANGE_NETWORK_STATE" />
<uses-permission android:name="android.permission.MODIFY_AUDIO_SETTINGS" />
You can also add additional permissions in order to use Bluetooth devices:
<uses-permission android:name="android.permission.BLUETOOTH" android:maxSdkVersion="30" />
<uses-permission android:name="android.permission.BLUETOOTH_ADMIN" android:maxSdkVersion="30" />
Additionally, you must set your app to target Java 8 through the app-level build.gradle file for Android:
android {
//...
compileOptions {
sourceCompatibility JavaVersion.VERSION_1_8
targetCompatibility JavaVersion.VERSION_1_8
}
}
Adding a WebSocket
Any WebRTC-based application needs to communicate with a plethora of servers and components. To do this, we must first set up a WebSocket that allows us to connect to these servers.
Here is a class that establishes a connection with the server mentioned above via the URL provided:
import 'dart:io';
import 'dart:math';
import 'dart:convert';
import 'dart:async';
class SimpleWebSocket {
String _url;
var _socket;
Function()? onOpen;
Function(dynamic msg)? onMessage;
Function(int? code, String? reaso)? onClose;
SimpleWebSocket(this._url);
connect() async {
try {
_socket = await _connectForSelfSignedCert(_url);
onOpen?.call();
_socket.listen((data) {
onMessage?.call(data);
}, onDone: () {
onClose?.call(_socket.closeCode, _socket.closeReason);
});
} catch (e) {
onClose?.call(500, e.toString());
}
}
send(data) {
if (_socket != null) {
_socket.add(data);
print('send: $data');
}
}
close() {
if (_socket != null) _socket.close();
}
Future<WebSocket> _connectForSelfSignedCert(url) async {
try {
Random r = Random();
String key = base64.encode(List<int>.generate(8, (_) => r.nextInt(255)));
HttpClient client = HttpClient(context: SecurityContext());
client.badCertificateCallback =
(X509Certificate cert, String host, int port) {
print(
'SimpleWebSocket: Allow self-signed certificate => $host:$port. ');
return true;
};
HttpClientRequest request =
await client.getUrl(Uri.parse(url)); // form the correct url here
request.headers.add('Connection', 'Upgrade');
request.headers.add('Upgrade', 'websocket');
request.headers.add(
'Sec-WebSocket-Version', '13'); // insert the correct version here
request.headers.add('Sec-WebSocket-Key', key.toLowerCase());
HttpClientResponse response = await request.close();
// ignore: close_sinks
var socket = await response.detachSocket();
var webSocket = WebSocket.fromUpgradedSocket(
socket,
protocol: 'signaling',
serverSide: false,
);
return webSocket;
} catch (e) {
throw e;
}
}
}
Signaling Server
While the overall WebRTC framework standardizes several aspects of creating a video calling experience, the one thing it leaves out is the signaling server. When creating a calling platform, you need to set up your own signaling server in order to connect devices. We go through the process of creating your own signaling server in a previous lesson. For this lesson, for the sake of brevity, we will use the signaling server that the flutter_webrtc plugin provides.
The signaling server project deals with several connection aspects in WebRTC, not just basic signaling. The class in our project dealing with the connection to the signaling server must create a WebSocket and listen for changes as well as any messages sent over the socket:
Future<void> connect() async {
var url = 'https://$_host:$_port/ws';
_socket = SimpleWebSocket(url);
print('connect to $url');
if (_turnCredential == null) {
try {
_turnCredential = await getTurnCredential(_host, _port);
_iceServers = {
'iceServers': [
{
'urls': _turnCredential['uris'][0],
'username': _turnCredential['username'],
'credential': _turnCredential['password']
},
]
};
} catch (e) {}
}
_socket?.onOpen = () {
print('onOpen');
onSignalingStateChange?.call(SignalingState.connectionOpen);
_send('new', {
'name': DeviceInfo.label,
'id': _selfId,
'user_agent': DeviceInfo.userAgent
});
};
_socket?.onMessage = (message) {
print('Received data: ' + message);
onMessage(_decoder.convert(message));
};
_socket?.onClose = (int? code, String? reason) {
print('Closed by server [$code => $reason]!');
onSignalingStateChange?.call(SignalingState.connectionClosed);
};
await _socket?.connect();
}
The entire class that connects and manages the signaling server connection is larger and is simplified here but you can view in full in the project repository.
Transferring data between peers
When a peer (a device on the network) wants to establish a call with another, it needs to create and send an offer to the other. It also needs to specify the session description, which defines several details about the potential call. See the lesson on the Session Description Protocol (SDP) for more information.
Future<void> _createOffer(Session session, String media) async {
try {
RTCSessionDescription s =
await session.pc!.createOffer(media == 'data' ? _dcConstraints : {});
await session.pc!.setLocalDescription(_fixSdp(s));
_send('offer', {
'to': session.pid,
'from': _selfId,
'description': {'sdp': s.sdp, 'type': s.type},
'session_id': session.sid,
'media': media,
});
} catch (e) {
print(e.toString());
}
}
Once the other peer receives the offer from the first, it needs to accept or reject the offer. To do this, it creates an answer and sends it to the first peer. If accepted, they can initiate a call session and start exchanging data.
Future<void> _createAnswer(Session session, String media) async {
try {
RTCSessionDescription s =
await session.pc!.createAnswer(media == 'data' ? _dcConstraints : {});
await session.pc!.setLocalDescription(_fixSdp(s));
_send('answer', {
'to': session.pid,
'from': _selfId,
'description': {'sdp': s.sdp, 'type': s.type},
'session_id': session.sid,
});
} catch (e) {
print(e.toString());
}
}
We must monitor any messages on the socket, such as offers, answers, peers, candidates, and more. Based on the data received, we can update local information on each device and initiate calls:
void onMessage(message) async {
Map<String, dynamic> mapData = message;
var data = mapData['data'];
switch (mapData['type']) {
case 'peers':
{
List<dynamic> peers = data;
if (onPeersUpdate != null) {
Map<String, dynamic> event = <String, dynamic>{};
event['self'] = _selfId;
event['peers'] = peers;
onPeersUpdate?.call(event);
}
}
break;
case 'offer':
{
var peerId = data['from'];
var description = data['description'];
var media = data['media'];
var sessionId = data['session_id'];
var session = _sessions[sessionId];
var newSession = await _createSession(session,
peerId: peerId,
sessionId: sessionId,
media: media,
screenSharing: false);
_sessions[sessionId] = newSession;
await newSession.pc?.setRemoteDescription(
RTCSessionDescription(description['sdp'], description['type']));
if (newSession.remoteCandidates.isNotEmpty) {
newSession.remoteCandidates.forEach((candidate) async {
await newSession.pc?.addCandidate(candidate);
});
newSession.remoteCandidates.clear();
}
onCallStateChange?.call(newSession, CallState.callStateNew);
onCallStateChange?.call(newSession, CallState.callStateRinging);
}
break;
case 'answer':
{
var description = data['description'];
var sessionId = data['session_id'];
var session = _sessions[sessionId];
session?.pc?.setRemoteDescription(
RTCSessionDescription(description['sdp'], description['type']));
onCallStateChange?.call(session!, CallState.callStateConnected);
}
break;
case 'candidate':
{
var peerId = data['from'];
var candidateMap = data['candidate'];
var sessionId = data['session_id'];
var session = _sessions[sessionId];
RTCIceCandidate candidate = RTCIceCandidate(candidateMap['candidate'],
candidateMap['sdpMid'], candidateMap['sdpMLineIndex']);
if (session != null) {
if (session.pc != null) {
await session.pc?.addCandidate(candidate);
} else {
session.remoteCandidates.add(candidate);
}
} else {
_sessions[sessionId] = Session(pid: peerId, sid: sessionId)
..remoteCandidates.add(candidate);
}
}
break;
case 'leave':
{
var peerId = data as String;
_closeSessionByPeerId(peerId);
}
break;
case 'bye':
{
var sessionId = data['session_id'];
print('bye: ' + sessionId);
var session = _sessions.remove(sessionId);
if (session != null) {
onCallStateChange?.call(session, CallState.callStateBye);
_closeSession(session);
}
}
break;
case 'keepalive':
{
print('keepalive response!');
}
break;
default:
break;
}
}
Building the video renderer
Once we receive the data stream from other peers, we can start displaying all users' videos. To do this, we create an RTCVideoRenderer which represents a single user’s video stream data. In this call between two users, we create a local renderer and a remote renderer representing the two participants.
We can then listen to remote and local stream changes using onAddRemoteStream and onLocalStream respectively (part of the signaling class in the project). These renderers can then be passed along to the RTCVideoView widget in order to be displayed:
RTCVideoRenderer _localRenderer = RTCVideoRenderer();
RTCVideoRenderer _remoteRenderer = RTCVideoRenderer();
// ...
_signaling?.onLocalStream = ((stream) {
_localRenderer.srcObject = stream;
setState(() {});
});
_signaling?.onAddRemoteStream = ((_, stream) {
_remoteRenderer.srcObject = stream;
setState(() {});
});
// When declaring widgets
RTCVideoView(_localRenderer, mirror: true)
RTCVideoView(_remoteRenderer)
Adding Controls
Simply creating a video feed isn’t enough. We need to add controls for the users to turn their input devices, such as the camera and microphone, on or off. The flutter_webrtc plugin helps us via the Helper class which allows us to easily switch cameras. We can disable the audio stream directly to enable or disable the microphone.
void switchCamera() {
if (_localStream != null) {
Helper.switchCamera(_localStream!.getVideoTracks()[0]);
}
}
void muteMic() {
if (_localStream != null) {
bool enabled = _localStream!.getAudioTracks()[0].enabled;
_localStream!.getAudioTracks()[0].enabled = !enabled;
}
}
The Helper class also enables you to do several other things, such as selecting an audio input/output device, enabling/disabling speakerphone, and more:
static Future<void> selectAudioOutput(String deviceId) async {
await navigator.mediaDevices
.selectAudioOutput(AudioOutputOptions(deviceId: deviceId));
}
static Future<void> selectAudioInput(String deviceId) =>
NativeAudioManagement.selectAudioInput(deviceId);
static Future<void> setSpeakerphoneOn(bool enable) =>
NativeAudioManagement.setSpeakerphoneOn(enable);
And that’s it! Combining these aspects creates a simple, minimal video-calling experience with WebRTC and Flutter.
The Stream Video Flutter SDK
While this guide provides a hands-on approach to building video streaming capabilities from scratch using WebRTC and Flutter, we understand the complexity and time investment required to create a robust real-time communication solution. For developers seeking a more streamlined, ready-to-use option that minimizes integration efforts, Stream's Video SDK for Flutter might be a suitable option.
To kickstart your development with Stream's Video SDK for Flutter, we recommend exploring the following tutorials tailored to specific use cases:
- Video-Calling App Tutorial: Ideal for developers aiming to integrate video-sharing experience into their Flutter applications.
- Audio-Room Tutorial: Perfect for those interested in creating any form of audio interaction.
- Livestreaming Tutorial: Demonstrates creating your own live-streaming experience.
Conclusion
You can find the full tutorial on the Stream Github here.
In this tutorial, we've taken a deep dive into implementing real-time video streaming in Flutter using WebRTC, focusing on the practical steps necessary to set up a peer-to-peer video-sharing connection. With this knowledge, you're now equipped to explore the vast potential of interactive media in your Flutter projects.