PencilKit: Add Collaborative Whiteboard, Chat, & Video Calling To SwiftUI Apps

This tutorial teaches you how to implement freeform drawing, chat messaging, voice calling, and video calling into your SwiftUI apps. We will use Apple's PencilKit framework, equipped with an intuitive drawing canvas and a rich set of tools for making handwritten notes and sketches.

On the drawing canvas of the app, users can initiate voice and video calls, record, and share screens. Additionally, the whiteboard provides a seamless way to share and collaborate on ideas and whatever you draw with other channel members via chat messaging.

Find an Apple Pencil or make your fingers ready to draw with an iOS/SwiftUI app you build yourself in this tutorial. How fun is that? Let’s begin.

Prerequisites

Completing the tutorial requires the following installations.

• Xcode 15: Our development environment.
• Stream Chat SwiftUI: To provide messaging functionality and the ability to attach free-form notes and drawings to messages.
• Stream Video SwiftUI: To provide audio and video calling functionalities.
• PencilKit: A framework for capturing finger and Apple Pencil inputs as drawings.

You can test all the app’s features except the video calling capability with an iOS simulator. However, to get the best testing experience, you should use an Apple Pencil with an iPad or your finger to test it on an iPhone.

Explore the Final Sample Project

The video above represents the final project you will build in this tutorial. You can download it from GitHub. As shown in the video, collaboration tools are those on the top-right of the drawing canvas, the icons on the bottom-right are drawing tools, and those on the bottom-left are for making corrections to drawings. The app also provides default PencilKit drawing tools like the toolset on the top-left and a ruler for sketching straight and diagonal lines.

Project Setup

Let's create a new SwiftUI project in Xcode, name it FaceBoard, or use any preferred name, and do the following to make it ready for coding our demo app.

Add Folder Groups and Swift Files

In your newly created SwiftUI project, add the groups (folders) and Swift files shown above. You can also find each of them in the GitHub project. Let's add the Package Dependencies in the following section.

Install Stream Chat and Video SDKs

The image in the previous section shows the following package dependencies.

• StreamChat: To provide the app's chat feature.
• StreamChatSwiftUI: Customizable and reusable SwiftUI component for building chat experiences.
• StreamVideo: The core video calling SDK consists of SwiftUI components.
• StreamWebRTC: To provide the app's WebRTC based video calling feature.
• SwiftProtobuf: An alternative to JSON and XML, helping to serialize structured data.

Let's install Stream’s Chat and Video SDKs for iOS to bring all the five package dependencies above. In your Xcode project, go to File -> Add Package Dependencies, copy and paste the following URLs, and follow the steps to install the chat and video SDKs.

• Stream Chat SwiftUI: https://getstream.io/tutorials/ios-chat.

• Stream Video SwiftUI: https://getstream.io/video/docs/ios

Set Permissions For Users' Protected Assets

Users can make freeform sketches and notes in our app's drawing canvas, save them to their iOS device's Photos Library, and send them to other collaborators as message attachments. You should set permission in the Xcode project to allow the app to pick saved drawings from the user's Photos Library and send them to others.

The app's audio and video calling feature also requires people's microphones and cameras to make calls. The protected user assets above also require privacy configuration in Xcode. Select your main project’s folder, click the Info tab, and add the privacies for photos, camera, and microphone usage as highlighted in the image above.

PencilKit Overview

PencilKit is a freeform drawing framework from Apple that allows the implementation of low-latency drawing in macOS, iOS, and visionOS apps by capturing and displaying drawing inputs from users' fingers and Apple Pencil. This framework lets developers quickly incorporate hand-drawn content, note-taking, and document or image markup into their apps. PencilKit has default drawing tools for creating, erasing, undoing, and selecting. The sketching environment supports tilt sensitivity and palm rejection to ensure pixel-perfect precision drawing for integrated apps.

PencilKit provides seamless support for the following features when you integrate it into your iOS app.

• Precision finger and pencil drawing: Users can draw, jot notes, and make precise illustrations.
• Ultra low-latency drawing: Drawing with the finger or Apple Pencil on the iOS device's screen feels as responsive as sketching with a physical pencil and paper.
• Pressure-sensitivity for illustrations: Its drawing tools, like the Fountain Pen, respond seamlessly to light and deep presses (pressure) to draw thin and thick lines and curves.
• Tilt-sensitivity for shading: The drawing canvas automatically supports the tilt-to-shade feature when using an Apple Pencil.

Collaboration Features: Drawing, Messaging, and Calling

With the sample SwiftUI drawing app we will create in this tutorial, you can work alone to jot your ideas, screen share your whiteboard, make video calls, and send drawings as message attachments to collaborate with others. To send jotted ideas to others, save whatever you draw on the canvas by tapping the save button on the top right. This action saves drawings on the canvas to the iOS Photos Library. Then, tap the chat icon to open a list of collaborators (chat channel list). Scroll through the collaborators, select one, pick the saved drawing from the Photos Library, and attach it to a message to share. The video above demonstrates drawing, saving, and attaching a drawn image to a message.

When you follow the steps below to complete the tutorial or run the app after downloading it from GitHub, tap the video button 🎥 on the top-right to initiate a call. Once the call starts, you can invite collaborators to join the call by tapping the person.2 symbol on the top-right of the screen.

Add Collaborative Messaging Support

Previously, we looked at how to install the Stream's Chat SDKs and Video SDKs. Let's dive into what their setups involve.
The folder structure and files involved are shown in the image below for the chat part.

Create the folder ChatMessaging and add the contents of the following Swift files.

• ChatSetup.swift
• CustomChannelList.swift

In this tutorial, we will not go into setting up the chat SDK. However, we have excellent resources that explain how to set it up in steps. Check out the chat tutorial in the iOS documentation to learn more.

Add Collaborative Video Calling Support

Similarly, the video calling settings are in the folder, as shown in the image below. Add the following Swift files and their content in the links below.

• CallContainerSetup.swift
• ParticipantsView.swift
• FloatingParticipantView.swift

Check out this YouTube video and the video calling tutorial in our documentation to learn how to configure the video SDK.

Create the Freeform Drawing Canvas

To add a freeform drawing board to our SwiftUI project, let's implement it in a single Swift file for simplicity. Rename ContentView.swift that comes with the project creation with FreeFormDrawingView.swift. Feel free to choose any name you want. Replace the content of this file with the following sample code.

//
//  FreeFormDrawingView.swift
//  FaceBoard

import SwiftUI
import PencilKit
import StreamVideo
import StreamVideoSwiftUI
import StreamChat
import StreamChatSwiftUI

struct FreeFormDrawingView: View {

    @ObservedObject var viewModel: CallViewModel
    // Define a state variable to capture touches from the user's finger and Apple pencil.

Let's summarize the code we just added. To add a drawing canvas and tools to any SwiftUI project, you should:

1. First, make it available as import PencilKit.
2. In the FreeFormDrawingView struct, define a PKCanvasView object to capture users' finger and Apple Pencil inputs @State private var canvas = PKCanvasView().
3. Next, define the following properties for drawing, color, pencil type, and undoing.

  @State private var isDrawing = true
  @State private var color: Color = .black
  @State private var pencilType: PKInkingTool.InkType = .pencil
  @State private var colorPicker = false
  @Environment(\.undoManager) private var undoManager

4. SwiftUI does not support PencilKit natively. Therefore, we should create a DrawingView struct and make it UIViewRepresentable. In this struct, we define binding properties for updating drawing inputs about whether a user is drawing, pencil type, and color. We create a computed variable to update the pencil type whenever a user picks a different pencil. Then, we create the function makeUIView to enable finger and Apple Pencil drawing and show the default tool picker when the canvas becomes a first responder. The updateUIView function watches the binding variables for changes.

struct DrawingView: UIViewRepresentable {
  // Capture drawings for saving in the photos library
  @Binding var canvas: PKCanvasView
  @Binding var isDrawing: Bool
  // Ability to switch a pencil
  @Binding var pencilType: PKInkingTool.InkType
  // Ability to change a pencil color
  @Binding var color: Color

  //let ink = PKInkingTool(.pencil, color: .black)
  // Update ink type
  var ink: PKInkingTool {
    PKInkingTool(pencilType, color: UIColor(color))
  }

Let's navigate back to the FreeFormDrawingView struct. In the body computed property, we add a NavigationStack and create an instance of the drawing view DrawingView(canvas: $canvas, isDrawing: $isDrawing, pencilType: $pencilType, color: $color). Then, we add a .toolbar and append the drawing tools, chat, and video buttons to the various sections of it. Eventually, we use the saveDrawing function to store whatever users scribble on the canvas in the iOS device's Photo Library.

func saveDrawing() {
    // Get the drawing image from the canvas
    let drawingImage = canvas.drawing.image(from: canvas.drawing.bounds, scale: 1.0)

    // Save drawings to the Photos Album
    UIImageWriteToSavedPhotosAlbum(drawingImage, nil, nil, nil)
  }

Test the app

In the main app’s file FaceBoardApp.swift, let’s modify the Scene to display the drawing canvas FreeFormDrawingView and CallContainerSetup that contains the video call configurations.

//
// FaceBoardApp.swift
// FaceBoard
//
import SwiftUI
import StreamVideo
import StreamVideoSwiftUI
@main
struct FaceBoardApp: App {
  var body: some Scene {
    WindowGroup {
      ZStack {
        CallContainerSetup()
        FreeFormDrawingView(viewModel: CallViewModel())
      }
    }
  }
}

Since these two files are in the entry point of the app WindowGroup, the whiteboard, its tools, and the calling functionality become immediately available when you run the app.

What’s Next?

You discovered the fundamentals of creating a SwiftUI collaborative whiteboard and freeform drawing app in this article. To go beyond the basics, head to the PencilKit, Stream Chat, and Video for iOS documentation to learn more about building an iOS drawing app to help people work together or alone to bring their ideas to life.