Voice Agent

Talking is often faster than typing, and it's the only way to use an app with busy hands. A voice agent listens to the user, answers out loud, and lets them interrupt it, like a phone call with your assistant.

Note:

What you'll build

  • A call screen that puts your agent on a call with the user
  • The agent as a glowing globe that moves with its voice
  • What was said, written out under the globe
  • Mute and hang up
  • The question a tool asks before it runs, over the call

Starts from: the Quickstart, with the user signed in to agents.

Info:

VoiceSession does the hard part. It starts your agent on a call, and joins the call from the phone with Stream Video, which comes with StreamAgents. The agent's speech-to-text, model and text-to-speech run on Stream, and when the user talks over the agent, it stops and listens.

Best practices

  • Show what the agent is doing at every moment. A silent screen while the agent thinks feels broken.
  • Drive the globe from the agent's real voice, so it swells with its words.
  • Write out what was said. Users glance at it to check a name, a number or an address.
  • Make hanging up obvious, so the agent stops listening.

Give the agent a voice

Any agent can talk, with the router's default speech models. Make it from the voice template to choose its models, voice and greeting:

getstream agents init my-agent --template voice
getstream agents sync my-agent

Add NSMicrophoneUsageDescription to your Info.plist. STT and TTS & Voices list the models and voices.

Start the call

VoiceAgentView starts a voice session with the agent you name, and shows the call once the agent is on it. Present it from anywhere, such as with .fullScreenCover:

import StreamChatAI
import SwiftUI
import StreamAgents

struct VoiceAgentView: View {
    let agent: String
    @State private var voice: VoiceSession?
    @Environment(\.dismiss) private var dismiss

    var body: some View {
        if let voice {
            VoiceCallView(voice: voice) { dismiss() }
        } else {
            AIVoiceGlowView(state: .connecting)
                .frame(width: 220, height: 220)
                .task {
                    do {
                        voice = try await VoiceSession.start(agents: agents, agent: agent)
                    } catch {
                        dismiss()
                    }
                }
        }
    }
}

Show the call

VoiceCallView joins the call from the phone, with the microphone on, and shows the agent, what was said, and the controls. voice.session.turns holds the conversation as it's said, so the transcript grows as the user and the agent talk:

import StreamChatAI
import SwiftUI
import StreamAgents

struct VoiceCallView: View {
    @ObservedObject var voice: VoiceSession
    let onHangUp: () -> Void

    var body: some View {
        VStack {
            if let call = voice.call {
                AgentGlow(session: voice.session, call: call.state)
            } else {
                AIVoiceGlowView(state: .connecting)
                    .frame(width: 220, height: 220)
            }
            AIConversationView(messages: voice.session.turns.map(TurnMessage.init))
            HStack(spacing: 40) {
                Button(voice.isMuted ? "Unmute" : "Mute", systemImage: voice.isMuted ? "mic.slash.fill" : "mic.fill") {
                    Task { await voice.setMuted(!voice.isMuted) }
                }
                Button("Hang up", systemImage: "phone.down.fill", role: .destructive) {
                    Task {
                        await voice.leave()
                        onHangUp()
                    }
                }
            }
            .labelStyle(.iconOnly)
            .font(.largeTitle)
        }
        .task {
            await voice.session.start()
            await voice.join()
        }
    }
}

struct TurnMessage: AIMessage {
    let turn: Turn

    var id: String { turn.id }
    var role: AIMessageRole { turn.speaker.isAgent ? .assistant : .user }
    var text: String { turn.text }
}

join() doesn't throw: if it fails, for example because the user denied the microphone, the error is in voice.failure.

Show what the agent is doing

AgentGlow shows the agent as an AIVoiceGlowView. The call says when the agent speaks, and how loud, and the session says when it listens or thinks:

import StreamChatAI
import StreamVideo
import SwiftUI
import StreamAgents

struct AgentGlow: View {
    @ObservedObject var session: AgentSession
    @ObservedObject var call: CallState

    var body: some View {
        let agent = call.remoteParticipants.first
        let isSpeaking = agent?.isSpeaking == true
        AIVoiceGlowView(
            state: isSpeaking ? .speaking : state,
            audioLevel: isSpeaking ? agent.map { Double($0.audioLevel) } : nil
        )
        .frame(width: 220, height: 220)
    }

    private var state: AIVoiceAgentState {
        switch session.state {
        case .listening: .listening
        case .responding, .working: .thinking
        case .idle, .ended: .idle
        }
    }
}

The glow takes its colors from the appearance, and AIVoiceWaveView can show the user's microphone the same way, from call.localParticipant. Voice Visualizations lists the options of both.

Ask before a tool runs

A tool that asks first waits in the session's approvals until the user answers. On a call, show the question over it, in the VStack of VoiceCallView:

if let request = voice.session.approvals.first {
    AIToolApprovalCard(approval: AIToolApproval(title: request.approval.title, reason: request.reason)) { allowed in
        Task { try? await voice.session.decide(request.id, allowed: allowed) }
    }
}

Show the agent the camera

An agent with the vision skill can look through the phone's camera. Join with the camera on with voice.join(camera: true), or turn it on during the call with voice.setCameraEnabled(true). Video covers what the agent sees.

Next steps