Voice Agent

Talking is often faster than typing, and it's the only way to use an app with busy hands. A voice agent listens to the user, answers out loud, and lets them interrupt it, like a phone call with your assistant.

Note:

What you'll build

  • A call screen that puts your agent on a call with the user
  • The agent as a glowing globe that moves with its voice
  • What was said, written out under the globe
  • Mute and hang up
  • The question a tool asks before it runs, over the call

Starts from: the Quickstart, with the user signed in to agents.

Info:

VoiceSession does the hard part. It starts your agent on a call, and joins the call from the phone with Stream Video, which comes with the agents SDK. The agent's speech-to-text, model and text-to-speech run on Stream, and when the user talks over the agent, it stops and listens.

Best practices

  • Show what the agent is doing at every moment. A silent screen while the agent thinks feels broken.
  • Drive the globe from the agent's real voice, so it swells with its words.
  • Write out what was said. Users glance at it to check a name, a number or an address.
  • Make hanging up obvious, so the agent stops listening.

Give the agent a voice

Any agent can talk, with the router's default speech models. Make it from the voice template to choose its models, voice and greeting:

getstream agents init my-agent --template voice
getstream agents sync my-agent

The call needs the microphone. RECORD_AUDIO is already in your manifest, from Stream Video, but the app has to ask for it at runtime before it joins. STT and TTS & Voices list the models and voices.

Start the call

VoiceAgentScreen asks for the microphone, starts a voice session with the agent you name, and shows the call once the agent is on it. Show it from anywhere, such as a destination of your navigation graph:

import android.Manifest
import androidx.activity.compose.rememberLauncherForActivityResult
import androidx.activity.result.contract.ActivityResultContracts
import io.getstream.chat.android.ai.compose.ui.component.AIVoiceAgentState
import io.getstream.chat.android.ai.compose.ui.component.AIVoiceGlow
import io.getstream.agents.android.core.AgentsException
import io.getstream.agents.android.core.SessionOptions
import io.getstream.agents.android.core.VoiceSession

@Composable
fun VoiceAgentScreen(agent: String, onClose: () -> Unit) {
    val context = LocalContext.current
    var voice by remember { mutableStateOf<VoiceSession?>(null) }
    // The call needs the microphone, so ask for it before the agent starts.
    val microphone = rememberLauncherForActivityResult(ActivityResultContracts.RequestPermission()) { granted ->
        if (!granted) onClose()
    }
    LaunchedEffect(Unit) { microphone.launch(Manifest.permission.RECORD_AUDIO) }

    val started = voice
    if (started != null) {
        VoiceCallScreen(started, onHangUp = onClose)
    } else {
        AIVoiceGlow(state = AIVoiceAgentState.Connecting, modifier = Modifier.size(220.dp))
        LaunchedEffect(agent) {
            try {
                // The agent joins the session's call right away.
                voice = VoiceSession.start(context, agents, SessionOptions(agent = agent))
            } catch (_: AgentsException) {
                onClose()
            }
        }
    }
}

Show the call

VoiceCallScreen joins the call from the phone, with the microphone on, and shows the agent, what was said, and the controls. voice.session.conversation holds the conversation as it's said, so the transcript grows as the user and the agent talk:

import androidx.compose.material.icons.Icons
import androidx.compose.material.icons.filled.CallEnd
import androidx.compose.material.icons.filled.Mic
import androidx.compose.material.icons.filled.MicOff
import io.getstream.chat.android.ai.compose.conversation.AIMessage
import io.getstream.chat.android.ai.compose.conversation.AIMessageRole
import io.getstream.chat.android.ai.compose.ui.component.AIConversation
import io.getstream.agents.android.core.Turn

@Composable
fun VoiceCallScreen(voice: VoiceSession, onHangUp: () -> Unit) {
    val scope = rememberCoroutineScope()
    val call by voice.call.collectAsStateWithLifecycle()
    val muted by voice.isMuted.collectAsStateWithLifecycle()
    val conversation by voice.session.conversation.collectAsStateWithLifecycle()

    // Joins from this device, with the microphone on and the agent on the speaker.
    LaunchedEffect(voice) { voice.join() }

    Column(Modifier.fillMaxSize(), horizontalAlignment = Alignment.CenterHorizontally) {
        val active = call
        if (active != null) {
            AgentGlow(voice.session, active)
        } else {
            AIVoiceGlow(state = AIVoiceAgentState.Connecting, modifier = Modifier.size(220.dp))
        }
        AIConversation(
            messages = conversation.turns.map(::TurnMessage),
            modifier = Modifier.weight(1f),
        )
        Row(horizontalArrangement = Arrangement.spacedBy(40.dp)) {
            FilledTonalIconButton(onClick = { voice.setMuted(!muted) }) {
                Icon(
                    if (muted) Icons.Filled.MicOff else Icons.Filled.Mic,
                    contentDescription = if (muted) "Unmute" else "Mute",
                )
            }
            FilledIconButton(
                onClick = {
                    scope.launch {
                        voice.leave()
                        onHangUp()
                    }
                },
            ) {
                Icon(Icons.Filled.CallEnd, contentDescription = "Hang up")
            }
        }
    }
}

/** A turn of the call, as the AI components show it. */
data class TurnMessage(val turn: Turn) : AIMessage {
    override val id: String = turn.id
    override val role: AIMessageRole = if (turn.isAgent) AIMessageRole.Assistant else AIMessageRole.User
    override val text: String = turn.text
}

join() doesn't throw: if it fails, for example because the user denied the microphone, the error is in voice.failure. leave() leaves the call and ends the session start opened.

Show what the agent is doing

AgentGlow shows the agent as an AIVoiceGlow. The call says when the agent speaks, and how loud, and the session says when it listens or thinks:

import io.getstream.video.android.core.Call
import io.getstream.agents.android.core.AgentSession
import io.getstream.agents.android.core.Conversation

@Composable
fun AgentGlow(session: AgentSession, call: Call) {
    val conversation by session.conversation.collectAsStateWithLifecycle()
    // The agent is the call's other participant.
    val participants by call.state.remoteParticipants.collectAsStateWithLifecycle()
    val agent = participants.firstOrNull()
    val speaking by remember(agent) { agent?.speaking ?: MutableStateFlow(false) }.collectAsStateWithLifecycle()
    val level = remember(agent) { agent?.audioLevel ?: MutableStateFlow(0f) }

    val state = when {
        speaking -> AIVoiceAgentState.Speaking
        else -> when (conversation.state) {
            Conversation.State.Listening -> AIVoiceAgentState.Listening
            Conversation.State.Responding, is Conversation.State.Working -> AIVoiceAgentState.Thinking
            Conversation.State.Idle, Conversation.State.Ended -> AIVoiceAgentState.Idle
        }
    }
    AIVoiceGlow(
        state = state,
        modifier = Modifier.size(220.dp),
        // Read once a frame, so the glow swells with the agent's words.
        audioLevel = if (speaking) ({ level.value }) else null,
    )
}

The glow takes the theme's primary color unless you pass it colors, and AIVoiceWave can show the user's microphone the same way, from call.state.me. Voice Visualizations lists the options of both.

Ask before a tool runs

A tool of your app that asks first, an AgentTool with an approval, waits in the session's approvals until the user answers. On a call, show the question over it with AIToolApprovalCard:

import io.getstream.chat.android.ai.compose.parts.AIToolApproval
import io.getstream.chat.android.ai.compose.ui.component.AIToolApprovalCard
import io.getstream.chat.android.ai.compose.ui.component.AIToolApprovalState
import io.getstream.agents.android.core.ToolApprovalRequest

@Composable
fun ApprovalCard(request: ToolApprovalRequest, session: AgentSession) {
    val scope = rememberCoroutineScope()
    var state by remember { mutableStateOf(AIToolApprovalState()) }
    AIToolApprovalCard(
        approval = AIToolApproval(
            title = request.approval.title,
            message = request.approval.message,
            reason = request.reason.ifEmpty { null },
        ),
        state = state,
        decide = { allowed ->
            state = AIToolApprovalState(isSending = true)
            scope.launch {
                try {
                    session.decide(request.id, allowed)
                } catch (_: AgentsException) {
                    state = AIToolApprovalState(failed = true)
                }
            }
        },
    )
}

Then add it to the Column of VoiceCallScreen, above the controls:

val approvals by voice.session.approvals.collectAsStateWithLifecycle()

approvals.firstOrNull()?.let { request ->
    // Keyed by the call, so the next question starts with no answer sending.
    key(request.id) { ApprovalCard(request, voice.session) }
}

Show the agent the camera

An agent with the vision skill can look through the phone's camera. Join with the camera on with voice.join(camera = true), or turn it on during the call with voice.setCameraEnabled(true). The camera starts on the back lens, and the agent sees what the call publishes. Ask for CAMERA first:

import androidx.compose.material.icons.filled.Videocam

@Composable
fun CameraButton(voice: VoiceSession) {
    // The camera needs its permission before it turns on.
    val camera = rememberLauncherForActivityResult(ActivityResultContracts.RequestPermission()) { granted ->
        if (granted) voice.setCameraEnabled(true)
    }
    FilledTonalIconButton(onClick = { camera.launch(Manifest.permission.CAMERA) }) {
        Icon(Icons.Filled.Videocam, contentDescription = "Show the agent the camera")
    }
}

Video covers what the agent sees.

Next steps