Personal Assistant

A personal assistant knows what's on the user's plate. It reads their calendar, adds their reminders, searches their email, keeps every conversation they've had with it, and listens when they'd rather talk than type.

Note:

What you'll build

  • A side menu with the user's conversations, titled and searchable
  • The user's calendar and reminders, as tools the agent runs on the phone
  • A question before the agent adds a reminder
  • The user's Gmail, which they connect from the conversation
  • Photos the user attaches to a question
  • A voice mode in the same conversation

Starts from: the Quickstart, with agents, AgentMessage and AgentTransport in your project. It targets iOS 17, for the EventKit APIs its calendar tools use.

Best practices

  • Run on the phone what only the phone has, such as the calendar, reminders or location. Only a tool's answer leaves the phone.
  • Ask for each permission when a tool first needs it, not when the app starts.
  • Ask before a tool that changes something, such as adding a reminder.
  • Let users connect their own accounts, such as Gmail, from the conversation. The agent never sees their password.

Define the assistant

Make the agent from the voice template, so it can talk as well as write, and give it each user's Gmail as a connector:

getstream agents init assistant --template voice
agents/assistant/agent.yaml
name: assistant
mode: voice
llm: llm-fast
connectors:
  - name: gmail # each user signs in to their own Gmail
    tools: [search_threads, get_thread, get_message]

Write what the assistant is for in instructions.md, then sync it with getstream agents sync assistant. User connection lists the other connectors, such as Google Calendar and Notion.

Give the agent the calendar and reminders

A tool the app gives the agent runs on the phone. When the model calls it, its closure runs and returns the answer the model sees. addReminder declares an approval, so the user is asked first, with the reminder's title:

import EventKit
import Foundation
import StreamAgents

let listEvents = AgentTool(
    name: "list_events",
    description: "The events on the user's calendar for one day.",
    parameters: .strings(["day": "the day, as YYYY-MM-DD"], required: ["day"])
) { arguments in
    let store = EKEventStore()
    guard try await store.requestFullAccessToEvents() else { return "No access to the calendar." }
    let start = try Date.ISO8601FormatStyle(timeZone: .current).year().month().day()
        .parse(arguments["day"]?.stringValue ?? "")
    let events = store.events(matching: store.predicateForEvents(withStart: start, end: start + 86_400, calendars: nil))
    return events.map { "\($0.startDate.formatted(date: .omitted, time: .shortened)) \($0.title ?? "")" }
        .joined(separator: "\n")
}

let addReminder = AgentTool(
    name: "add_reminder",
    description: "Adds a reminder to the user's Reminders app.",
    parameters: .strings(["title": "what to remind them of", "due": "when, in ISO 8601"], required: ["title"]),
    approval: .init(title: "Add this reminder?", reasonArgument: "title")
) { arguments in
    let store = EKEventStore()
    guard try await store.requestFullAccessToReminders() else { return "No access to Reminders." }
    let reminder = EKReminder(eventStore: store)
    reminder.title = arguments["title"]?.stringValue
    reminder.calendar = store.defaultCalendarForNewReminders()
    if let due = arguments["due"]?.stringValue {
        reminder.addAlarm(EKAlarm(absoluteDate: try Date(due, strategy: .iso8601)))
    }
    try store.save(reminder, commit: true)
    return "Added the reminder."
}

Add NSCalendarsFullAccessUsageDescription and NSRemindersFullAccessUsageDescription to your Info.plist.

Open a conversation

Pass the tools every time a conversation opens, new or carried on. A new one also names where a connector's sign-in comes back to, in your app's own URL scheme:

import Foundation
import StreamAgents

@MainActor
func openConversation(_ session: Session?) async throws -> AgentTransport {
    if let session {
        return try await AgentTransport(agents.attach(sessionID: session.id, tools: [listEvents, addReminder]))
    }
    var options = SessionOptions(agent: "assistant")
    options.tools = [listEvents, addReminder]
    options.returnURL = URL(string: "com.example.assistant:/connected")
    return try await AgentTransport(agents.chat(options))
}

List the user's conversations

Each conversation is a session, which the router titles after its first answers. ConversationList lists the user's sessions with the assistant, most recent first, and searches them:

import SwiftUI
import StreamAgents

struct ConversationList: View {
    let onSelect: (Session?) -> Void
    @State private var sessions: [Session] = []
    @State private var search = ""

    var body: some View {
        List {
            Button("New conversation", systemImage: "square.and.pencil") { onSelect(nil) }
            ForEach(sessions) { session in
                Button(session.title.isEmpty ? "New conversation" : session.title) { onSelect(session) }
            }
        }
        .searchable(text: $search)
        .task(id: search) {
            let all = agents.agent("assistant").sessions
            sessions = (try? await search.isEmpty ? all.query() : all.search(search))?.items ?? []
        }
    }
}

Put it together

SidebarView puts the list beside the open conversation, and the user opens it by dragging from the leading edge. The conversation's approver gets an AIToolAuthorizer, so when the agent first needs Gmail, the reply shows a Connect Gmail card that signs the user in:

import StreamChatAI
import SwiftUI
import StreamAgents

struct AssistantHome: View {
    @State private var transport: AgentTransport?
    @State private var isMenuOpen = false
    @State private var isTalking = false

    var body: some View {
        SidebarView(isOpen: $isMenuOpen, excludedBottomHeight: 80) {
            ConversationList { session in
                isMenuOpen = false
                Task { transport = try? await openConversation(session) }
            }
        } content: {
            NavigationStack {
                if let transport {
                    AssistantConversationView(transport: transport)
                        .id(transport.session.id)
                        .toolbar { Button("Talk", systemImage: "waveform") { isTalking = true } }
                }
            }
        }
        .task { transport = try? await openConversation(nil) }
        .fullScreenCover(isPresented: $isTalking) {
            if let transport { VoiceModeView(session: transport.session) }
        }
    }
}

struct AssistantConversationView: View {
    @StateObject private var conversation: AIConversation<AgentMessage>

    init(transport: AgentTransport) {
        var approver = transport.approver
        approver.authorizer = AIToolAuthorizer(callbackScheme: "com.example.assistant") { call in
            try await transport.session.declineLogin(call.id)
        }
        _conversation = StateObject(wrappedValue: AIConversation(transport: transport, approver: approver))
    }

    var body: some View {
        AIConversationView(conversation: conversation)
            .safeAreaInset(edge: .bottom) {
                AIComposerView(conversation: conversation)
            }
    }
}

The reply shows each tool call as the agent makes it, and the question of add_reminder under its call.

Send photos

The quickstart's AgentTransport sends only the text. To send the photos the user attaches, replace its send(_:), and add import UIKit to its file:

func send(_ request: AIConversationRequest<AgentMessage>) async throws {
    let images = request.attachments.compactMap { url in
        UIImage(contentsOfFile: url.path)?.jpegData(compressionQuality: 0.7)
            .map { ImageSource(url: "data:image/jpeg;base64," + $0.base64EncodedString()) }
    }
    _ = try await session.responses.create(request.text, images: images)
}

Pick a model that reads images in agent.yaml. responses.create also takes a reasoningEffort, for an option such as "Think harder" in the composer.

Talk instead of typing

The Talk button switches the open conversation to voice. The agent joins a call with everything said so far, and what's said on the call is written into the same conversation:

import SwiftUI
import StreamAgents

struct VoiceModeView: View {
    let session: AgentSession
    @State private var voice: VoiceSession?
    @Environment(\.dismiss) private var dismiss

    var body: some View {
        if let voice {
            VoiceCallView(voice: voice) {
                Task {
                    _ = try? await session.stopVoice()
                    dismiss()
                }
            }
        } else {
            ProgressView().task {
                do {
                    _ = try await session.startVoice()
                    voice = try await VoiceSession.attach(agents: agents, sessionID: session.id)
                } catch {
                    dismiss()
                }
            }
        }
    }
}

VoiceCallView is the call screen of the Voice agent use case. When the user hangs up, stopVoice() takes the agent off the call, and the conversation goes on in writing.

Next steps