PencilKit: Add Collaborative Whiteboard, Chat, & Video Calling To SwiftUI Apps

Visual collaboration apps like Apple’s Freeform (iOS and Mac) and Zoom Whiteboard provide new ways for teams to ideate, make freeform visuals and sketches, and work together.

Amos G.
Amos G.
Published February 2, 2024
PencilKit header

This tutorial teaches you how to implement freeform drawing, chat messaging, voice calling, and video calling into your SwiftUI apps. We will use Apple's PencilKit framework, equipped with an intuitive drawing canvas and a rich set of tools for making handwritten notes and sketches.

On the drawing canvas of the app, users can initiate voice and video calls, record, and share screens. Additionally, the whiteboard provides a seamless way to share and collaborate on ideas and whatever you draw with other channel members via chat messaging.

Find an Apple Pencil or make your fingers ready to draw with an iOS/SwiftUI app you build yourself in this tutorial. How fun is that? Let’s begin.

Prerequisites

Completing the tutorial requires the following installations.

You can test all the app’s features except the video calling capability with an iOS simulator. However, to get the best testing experience, you should use an Apple Pencil with an iPad or your finger to test it on an iPhone.

Explore the Final Sample Project

The video above represents the final project you will build in this tutorial. You can download it from GitHub. As shown in the video, collaboration tools are those on the top-right of the drawing canvas, the icons on the bottom-right are drawing tools, and those on the bottom-left are for making corrections to drawings. The app also provides default PencilKit drawing tools like the toolset on the top-left and a ruler for sketching straight and diagonal lines.

Project Setup

Let's create a new SwiftUI project in Xcode, name it FaceBoard, or use any preferred name, and do the following to make it ready for coding our demo app.

Add Folder Groups and Swift Files

Swift Files structure

In your newly created SwiftUI project, add the groups (folders) and Swift files shown above. You can also find each of them in the GitHub project. Let's add the Package Dependencies in the following section.

Install Stream Chat and Video SDKs

The image in the previous section shows the following package dependencies.

  • StreamChat: To provide the app's chat feature.
  • StreamChatSwiftUI: Customizable and reusable SwiftUI component for building chat experiences.
  • StreamVideo: The core video calling SDK consists of SwiftUI components.
  • StreamWebRTC: To provide the app's WebRTC based video calling feature.
  • SwiftProtobuf: An alternative to JSON and XML, helping to serialize structured data.

Let's install Stream’s Chat and Video SDKs for iOS to bring all the five package dependencies above. In your Xcode project, go to File -> Add Package Dependencies, copy and paste the following URLs, and follow the steps to install the chat and video SDKs.

Chat SDK installation window Stream Video SwiftUI*

Set Permissions For Users' Protected Assets

Privacy configuration

Users can make freeform sketches and notes in our app's drawing canvas, save them to their iOS device's Photos Library, and send them to other collaborators as message attachments. You should set permission in the Xcode project to allow the app to pick saved drawings from the user's Photos Library and send them to others.

The app's audio and video calling feature also requires people's microphones and cameras to make calls. The protected user assets above also require privacy configuration in Xcode. Select your main project’s folder, click the Info tab, and add the privacies for photos, camera, and microphone usage as highlighted in the image above.

PencilKit Overview

PencilKit drawing canvas

PencilKit is a freeform drawing framework from Apple that allows the implementation of low-latency drawing in macOS, iOS, and visionOS apps by capturing and displaying drawing inputs from users' fingers and Apple Pencil. This framework lets developers quickly incorporate hand-drawn content, note-taking, and document or image markup into their apps. PencilKit has default drawing tools for creating, erasing, undoing, and selecting. The sketching environment supports tilt sensitivity and palm rejection to ensure pixel-perfect precision drawing for integrated apps.

PencilKit provides seamless support for the following features when you integrate it into your iOS app.

  • Precision finger and pencil drawing: Users can draw, jot notes, and make precise illustrations.
  • Ultra low-latency drawing: Drawing with the finger or Apple Pencil on the iOS device's screen feels as responsive as sketching with a physical pencil and paper.
  • Pressure-sensitivity for illustrations: Its drawing tools, like the Fountain Pen, respond seamlessly to light and deep presses (pressure) to draw thin and thick lines and curves.
  • Tilt-sensitivity for shading: The drawing canvas automatically supports the tilt-to-shade feature when using an Apple Pencil.
Building your own app? Get early access to our Livestream or Video Calling API and launch in days!

Collaboration Features: Drawing, Messaging, and Calling

With the sample SwiftUI drawing app we will create in this tutorial, you can work alone to jot your ideas, screen share your whiteboard, make video calls, and send drawings as message attachments to collaborate with others. To send jotted ideas to others, save whatever you draw on the canvas by tapping the save button on the top right. This action saves drawings on the canvas to the iOS Photos Library. Then, tap the chat icon to open a list of collaborators (chat channel list). Scroll through the collaborators, select one, pick the saved drawing from the Photos Library, and attach it to a message to share. The video above demonstrates drawing, saving, and attaching a drawn image to a message.

When you follow the steps below to complete the tutorial or run the app after downloading it from GitHub, tap the video button 🎥 on the top-right to initiate a call. Once the call starts, you can invite collaborators to join the call by tapping the person.2 symbol on the top-right of the screen.

Add Collaborative Messaging Support

Collaborative Messaging Support

Previously, we looked at how to install the Stream's Chat SDKs and Video SDKs. Let's dive into what their setups involve.
The folder structure and files involved are shown in the image below for the chat part.

Chat folder

Create the folder ChatMessaging and add the contents of the following Swift files.

In this tutorial, we will not go into setting up the chat SDK. However, we have excellent resources that explain how to set it up in steps. Check out the chat tutorial in the iOS documentation to learn more.

Add Collaborative Video Calling Support

 Video Calling Support

Similarly, the video calling settings are in the folder, as shown in the image below. Add the following Swift files and their content in the links below.

Call setup folder

Check out this YouTube video and the video calling tutorial in our documentation to learn how to configure the video SDK.

Create the Freeform Drawing Canvas

To add a freeform drawing board to our SwiftUI project, let's implement it in a single Swift file for simplicity. Rename ContentView.swift that comes with the project creation with FreeFormDrawingView.swift. Feel free to choose any name you want. Replace the content of this file with the following sample code.

swift
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
// // FreeFormDrawingView.swift // FaceBoard import SwiftUI import PencilKit import StreamVideo import StreamVideoSwiftUI import StreamChat import StreamChatSwiftUI struct FreeFormDrawingView: View { @ObservedObject var viewModel: CallViewModel // Define a state variable to capture touches from the user's finger and Apple pencil. @State private var canvas = PKCanvasView() @State private var isDrawing = true @State private var color: Color = .black @State private var pencilType: PKInkingTool.InkType = .pencil @State private var colorPicker = false @State private var isMessaging = false @State private var isVideoCalling = false @State private var isScreenSharing = false @State private var isRecording = false @Environment(\.dismiss) private var dismiss @Environment(\.undoManager) private var undoManager var body: some View { NavigationStack { // Drawing View DrawingView(canvas: $canvas, isDrawing: $isDrawing, pencilType: $pencilType, color: $color) .navigationBarTitleDisplayMode(.inline) .toolbar { ToolbarItemGroup(placement: .bottomBar) { Button { // Clear the canvas. Reset the drawing canvas.drawing = PKDrawing() } label: { Image(systemName: "scissors") } Button { // Undo drawing undoManager?.undo() } label: { Image(systemName: "arrow.uturn.backward") } Button { // Redo drawing undoManager?.redo() } label: { Image(systemName: "arrow.uturn.forward") } Button { // Erase tool isDrawing = false } label: { Image(systemName: "eraser.line.dashed") } Divider() .rotationEffect(.degrees(90)) Button { // Tool picker //let toolPicker = PKToolPicker.init() @State var toolPicker = PKToolPicker() toolPicker.setVisible(true, forFirstResponder: canvas) toolPicker.addObserver(canvas) canvas.becomeFirstResponder() } label: { Image(systemName: "pencil.tip.crop.circle.badge.plus") } // Menu for pencil types and color Menu { Button { // Menu: Pick a color colorPicker.toggle() } label: { Label("Color", systemImage: "paintpalette") } Button { // Menu: Pencil isDrawing = true pencilType = .pencil } label: { Label("Pencil", systemImage: "pencil") } Button { // Menu: pen isDrawing = true pencilType = .pen } label: { Label("Pen", systemImage: "pencil.tip") } Button { // Menu: Marker isDrawing = true pencilType = .marker } label: { Label("Marker", systemImage: "paintbrush.pointed") } Button { // Menu: Monoline isDrawing = true pencilType = .monoline } label: { Label("Monoline", systemImage: "pencil.line") } Button { // Menu: pen isDrawing = true pencilType = .fountainPen } label: { Label("Fountain", systemImage: "paintbrush.pointed.fill") } Button { // Menu: Watercolor isDrawing = true pencilType = .watercolor } label: { Label("Watercolor", systemImage: "eyedropper.halffull") } Button { // Menu: Crayon isDrawing = true pencilType = .crayon } label: { Label("Crayon", systemImage: "pencil.tip") } } label: { Image(systemName: "hand.draw") } .sheet(isPresented: $colorPicker) { ColorPicker("Pick color", selection: $color) .padding() } Spacer() // Drawing Tools Button { // Pencil isDrawing = true pencilType = .pencil } label: { Label("Pencil", systemImage: "pencil.and.scribble") } Button { // Pen isDrawing = true pencilType = .pen } label: { Label("Pen", systemImage: "applepencil.tip") } Button { // Monoline isDrawing = true pencilType = .monoline } label: { Label("Monoline", systemImage: "pencil.line") } Button { // Fountain: Variable scribbling isDrawing = true pencilType = .fountainPen } label: { Label("Fountain", systemImage: "scribble.variable") } Button { // Marker isDrawing = true pencilType = .marker } label: { Label("Marker", systemImage: "paintbrush.pointed") } Button { // Crayon isDrawing = true pencilType = .crayon } label: { Label("Crayon", systemImage: "paintbrush") } Button { // Water Color isDrawing = true pencilType = .watercolor } label: { Label("Watercolor", systemImage: "eyedropper.halffull") } Divider() .rotationEffect(.degrees(90)) // Color picker Button { // Pick a color colorPicker.toggle() } label: { Label("Color", systemImage: "paintpalette") } Button { // Set ruler as active canvas.isRulerActive.toggle() } label: { Image(systemName: "pencil.and.ruler.fill") } } // Collaboration tools ToolbarItemGroup(placement: .topBarTrailing) { // Chat messaging Button { isMessaging.toggle() } label: { VStack { Image(systemName: "message") Text("Chat") .font(.caption2) } } .sheet(isPresented: $isMessaging, content: ChatSetup.init) // Video calling Button { isVideoCalling.toggle() } label: { VStack { Image(systemName: "video") Text("Call") .font(.caption2) } } .sheet(isPresented: $isVideoCalling, content: CallContainerSetup.init) // Screen sharing Button { isScreenSharing ? viewModel.stopScreensharing() : viewModel.startScreensharing(type: .inApp) isScreenSharing.toggle() } label: { VStack { Image(systemName: isScreenSharing ? "shared.with.you.slash" : "shared.with.you") .foregroundStyle(isScreenSharing ? .red : .blue) .contentTransition(.symbolEffect(.replace)) .contentTransition(.interpolate) withAnimation { Text(isScreenSharing ? "Stop" : "Share") .font(.caption2) .foregroundStyle(isScreenSharing ? .red : .blue) .contentTransition(.interpolate) } } } // Screen recording Button { isRecording.toggle() } label: { //Image(systemName: "rectangle.dashed.badge.record") VStack { Image(systemName: isRecording ? "rectangle.inset.filled.badge.record" : "rectangle.dashed.badge.record") .foregroundStyle(isRecording ? .red : .blue) .contentTransition(.symbolEffect(.replace)) .contentTransition(.interpolate) withAnimation { Text(isRecording ? "Stop" : "Record") .font(.caption2) .foregroundStyle(isRecording ? .red : .blue) .contentTransition(.interpolate) } } } Divider() .rotationEffect(.degrees(90)) // Save your creativity Button { saveDrawing() } label: { VStack { Image(systemName: "square.and.arrow.down.on.square") Text("Save") .font(.caption2) } } } } } } // Save drawings to Photos func saveDrawing() { // Get the drawing image from the canvas let drawingImage = canvas.drawing.image(from: canvas.drawing.bounds, scale: 1.0) // Save drawings to the Photos Album UIImageWriteToSavedPhotosAlbum(drawingImage, nil, nil, nil) } } struct DrawingView: UIViewRepresentable { // Capture drawings for saving in the photos library @Binding var canvas: PKCanvasView @Binding var isDrawing: Bool // Ability to switch a pencil @Binding var pencilType: PKInkingTool.InkType // Ability to change a pencil color @Binding var color: Color //let ink = PKInkingTool(.pencil, color: .black) // Update ink type var ink: PKInkingTool { PKInkingTool(pencilType, color: UIColor(color)) } let eraser = PKEraserTool(.bitmap) func makeUIView(context: Context) -> PKCanvasView { // Allow finger and pencil drawing canvas.drawingPolicy = .anyInput // Eraser tool canvas.tool = isDrawing ? ink : eraser canvas.alwaysBounceVertical = true // Toolpicker let toolPicker = PKToolPicker.init() toolPicker.setVisible(true, forFirstResponder: canvas) toolPicker.addObserver(canvas) // Notify when the picker configuration changes canvas.becomeFirstResponder() return canvas } func updateUIView(_ uiView: PKCanvasView, context: Context) { // Update tool whenever the main view updates uiView.tool = isDrawing ? ink : eraser } }

Let's summarize the code we just added. To add a drawing canvas and tools to any SwiftUI project, you should:

  1. First, make it available as import PencilKit.
  2. In the FreeFormDrawingView struct, define a PKCanvasView object to capture users' finger and Apple Pencil inputs @State private var canvas = PKCanvasView().
  3. Next, define the following properties for drawing, color, pencil type, and undoing.
swift
1
2
3
4
5
@State private var isDrawing = true @State private var color: Color = .black @State private var pencilType: PKInkingTool.InkType = .pencil @State private var colorPicker = false @Environment(\.undoManager) private var undoManager
  1. SwiftUI does not support PencilKit natively. Therefore, we should create a DrawingView struct and make it UIViewRepresentable. In this struct, we define binding properties for updating drawing inputs about whether a user is drawing, pencil type, and color. We create a computed variable to update the pencil type whenever a user picks a different pencil. Then, we create the function makeUIView to enable finger and Apple Pencil drawing and show the default tool picker when the canvas becomes a first responder. The updateUIView function watches the binding variables for changes.
swift
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
struct DrawingView: UIViewRepresentable { // Capture drawings for saving in the photos library @Binding var canvas: PKCanvasView @Binding var isDrawing: Bool // Ability to switch a pencil @Binding var pencilType: PKInkingTool.InkType // Ability to change a pencil color @Binding var color: Color //let ink = PKInkingTool(.pencil, color: .black) // Update ink type var ink: PKInkingTool { PKInkingTool(pencilType, color: UIColor(color)) } let eraser = PKEraserTool(.bitmap) func makeUIView(context: Context) -> PKCanvasView { // Allow finger and pencil drawing canvas.drawingPolicy = .anyInput // Eraser tool canvas.tool = isDrawing ? ink : eraser canvas.alwaysBounceVertical = true // Toolpicker let toolPicker = PKToolPicker.init() toolPicker.setVisible(true, forFirstResponder: canvas) toolPicker.addObserver(canvas) // Notify when the picker configuration changes canvas.becomeFirstResponder() return canvas } // makeUIView func updateUIView(_ uiView: PKCanvasView, context: Context) { // Update tool whenever the main view updates uiView.tool = isDrawing ? ink : eraser } // updateUIView } // DrawingView

Let's navigate back to the FreeFormDrawingView struct. In the body computed property, we add a NavigationStack and create an instance of the drawing view DrawingView(canvas: $canvas, isDrawing: $isDrawing, pencilType: $pencilType, color: $color). Then, we add a .toolbar and append the drawing tools, chat, and video buttons to the various sections of it. Eventually, we use the saveDrawing function to store whatever users scribble on the canvas in the iOS device's Photo Library.

swift
1
2
3
4
5
6
7
func saveDrawing() { // Get the drawing image from the canvas let drawingImage = canvas.drawing.image(from: canvas.drawing.bounds, scale: 1.0) // Save drawings to the Photos Album UIImageWriteToSavedPhotosAlbum(drawingImage, nil, nil, nil) }

Test the app

In the main app’s file FaceBoardApp.swift, let’s modify the Scene to display the drawing canvas FreeFormDrawingView and CallContainerSetup that contains the video call configurations.

swift
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
// // FaceBoardApp.swift // FaceBoard // import SwiftUI import StreamVideo import StreamVideoSwiftUI @main struct FaceBoardApp: App { var body: some Scene { WindowGroup { ZStack { CallContainerSetup() FreeFormDrawingView(viewModel: CallViewModel()) } } } }

Since these two files are in the entry point of the app WindowGroup, the whiteboard, its tools, and the calling functionality become immediately available when you run the app.

What’s Next?

You discovered the fundamentals of creating a SwiftUI collaborative whiteboard and freeform drawing app in this article. To go beyond the basics, head to the PencilKit, Stream Chat, and Video for iOS documentation to learn more about building an iOS drawing app to help people work together or alone to bring their ideas to life.

Integrating Video With Your App?
We've built a Video and Audio solution just for you. Check out our APIs and SDKs.
Learn more ->