Voice Agent

Talking is often faster than typing, and it's the only way to use an app with busy hands. A voice agent listens to the user, answers out loud, and lets them interrupt it, like a phone call with your assistant.

Note:

What you'll build

  • A call screen that puts your agent on a call with the user
  • The agent as a glowing globe that moves with its voice
  • What was said, written out under the globe
  • Mute and hang up
  • The question a tool asks before it runs, over the call

Starts from: the Quickstart, with the user signed in to agents, the StreamAgents its signIn returns, kept where your screens reach it, such as final agents = signIn('jane'); in lib/agents.dart.

Info:

VoiceSession does the hard part. It starts your agent on a call, and joins the call from the phone with Stream Video, which comes with stream_agents_rtc. The agent's speech-to-text, model and text-to-speech run on Stream, and when the user talks over the agent, it stops and listens.

Best practices

  • Show what the agent is doing at every moment. A silent screen while the agent thinks feels broken.
  • Drive the globe from the agent's real voice, so it swells with its words.
  • Write out what was said. Users glance at it to check a name, a number or an address.
  • Make hanging up obvious, so the agent stops listening.

Give the agent a voice

Any agent can talk, with the router's default speech models. Make it from the voice template to choose its models, voice and greeting:

getstream agents init my-agent --template voice
getstream agents sync my-agent

The call needs the microphone and the network. Declare them: NSMicrophoneUsageDescription in your Info.plist, and INTERNET, ACCESS_NETWORK_STATE, RECORD_AUDIO, MODIFY_AUDIO_SETTINGS and BLUETOOTH_CONNECT in your AndroidManifest.xml. Android also has the app ask for the microphone at runtime before it joins. STT and TTS & Voices list the models and voices.

Start the call

VoiceAgentScreen asks for the microphone, starts a voice session with the agent you name, and shows the call once the agent is on it. Push it from anywhere, such as in a MaterialPageRoute with fullscreenDialog: true. It asks with permission_handler, so add it to your pubspec.yaml:

pubspec.yaml
dependencies:
  # Asks for the microphone and the camera on Android.
  permission_handler: ^12.0.3

Then the screen:

lib/voice_agent_screen.dart
import 'dart:async';
import 'dart:io' show Platform;

import 'package:flutter/material.dart';
import 'package:permission_handler/permission_handler.dart';
import 'package:stream_chat_flutter_ai/stream_chat_flutter_ai.dart';
import 'package:stream_agents_core/stream_agents_core.dart';
import 'package:stream_agents_rtc/stream_agents_rtc.dart';

import 'agents.dart';
import 'voice_call_screen.dart';

class VoiceAgentScreen extends StatefulWidget {
  const VoiceAgentScreen({super.key, required this.agent, this.tools = const []});

  /// The agent to talk to, by the name it was synced under.
  final String agent;

  /// The tools of your app the agent can call.
  final List<AgentTool> tools;

  @override
  State<VoiceAgentScreen> createState() => _VoiceAgentScreenState();
}

class _VoiceAgentScreenState extends State<VoiceAgentScreen> {
  VoiceSession? _voice;

  @override
  void initState() {
    super.initState();
    _start();
  }

  Future<void> _start() async {
    // The call needs the microphone, so ask for it before the agent starts. iOS asks by
    // itself as the call is joined.
    if (Platform.isAndroid && !await Permission.microphone.request().isGranted) {
      _close();
      return;
    }
    try {
      // The agent joins the session's call right away.
      final voice = await VoiceSession.start(
        agents,
        agent: widget.agent,
        options: SessionOptions(tools: widget.tools),
      );
      if (!mounted) {
        await voice.leave();
        return;
      }
      setState(() => _voice = voice);
    } on Object {
      _close();
    }
  }

  void _close() {
    if (mounted) Navigator.of(context).pop();
  }

  @override
  void dispose() {
    // Leaving again after a hang-up does nothing.
    unawaited(_voice?.leave());
    super.dispose();
  }

  @override
  Widget build(BuildContext context) => switch (_voice) {
    final voice? => VoiceCallScreen(voice: voice, onHangUp: _close),
    null => const Scaffold(
      body: Center(
        child: SizedBox.square(
          dimension: 220,
          child: AIVoiceGlowView(state: AIVoiceAgentState.connecting),
        ),
      ),
    ),
  };
}

Show the call

VoiceCallScreen joins the call from the phone, with the microphone on, and shows the agent, what was said, and the controls. voice.session.conversation holds the conversation as it's said, so the transcript grows as the user and the agent talk:

lib/voice_call_screen.dart
import 'package:flutter/material.dart';
import 'package:stream_chat_flutter_ai/stream_chat_flutter_ai.dart';
import 'package:stream_agents_core/stream_agents_core.dart';
import 'package:stream_agents_rtc/stream_agents_rtc.dart';

import 'agent_glow.dart';

class VoiceCallScreen extends StatefulWidget {
  const VoiceCallScreen({super.key, required this.voice, required this.onHangUp, this.toolSession});

  final VoiceSession voice;

  /// The session that holds your app's tools, if it isn't the call's own.
  final AgentSession? toolSession;

  /// Called once the user hung up and the phone left the call.
  final VoidCallback onHangUp;

  @override
  State<VoiceCallScreen> createState() => _VoiceCallScreenState();
}

class _VoiceCallScreenState extends State<VoiceCallScreen> {
  @override
  void initState() {
    super.initState();
    // Joins from this device, with the microphone on and the agent on the speaker.
    widget.voice.join();
  }

  Future<void> _hangUp() async {
    await widget.voice.leave();
    widget.onHangUp();
  }

  @override
  Widget build(BuildContext context) {
    final voice = widget.voice;
    return Scaffold(
      body: SafeArea(
        child: StreamBuilder(
          stream: voice.state.stream,
          initialData: voice.state.value,
          builder: (context, snapshot) {
            final state = snapshot.requireData;
            return Column(
              children: [
                const SizedBox(height: 24),
                if (state.isJoined)
                  AgentGlow(voice: voice)
                else
                  const SizedBox.square(
                    dimension: 220,
                    child: AIVoiceGlowView(state: AIVoiceAgentState.connecting),
                  ),
                Expanded(
                  child: StreamBuilder(
                    stream: voice.session.conversation.stream,
                    initialData: voice.session.conversation.value,
                    builder: (context, conversation) => AIConversationView(
                      messages: [for (final turn in conversation.requireData.turns) TurnMessage(turn)],
                    ),
                  ),
                ),
                if (state.failure case final failure?)
                  Padding(padding: const EdgeInsets.all(16), child: Text('$failure', textAlign: TextAlign.center)),
                Padding(
                  padding: const EdgeInsets.all(24),
                  child: Row(
                    mainAxisAlignment: MainAxisAlignment.center,
                    spacing: 40,
                    children: [
                      IconButton.filledTonal(
                        tooltip: state.isMuted ? 'Unmute' : 'Mute',
                        iconSize: 28,
                        onPressed: state.isJoined ? () => voice.setMuted(!state.isMuted) : null,
                        icon: Icon(state.isMuted ? Icons.mic_off : Icons.mic),
                      ),
                      IconButton.filled(
                        tooltip: 'Hang up',
                        iconSize: 28,
                        style: IconButton.styleFrom(backgroundColor: Colors.red),
                        onPressed: _hangUp,
                        icon: const Icon(Icons.call_end),
                      ),
                    ],
                  ),
                ),
              ],
            );
          },
        ),
      ),
    );
  }
}

/// A turn of the call, as the AI components show it.
class TurnMessage with AIMessage {
  TurnMessage(this.turn);

  final Turn turn;

  @override
  String get id => turn.id;

  @override
  AIMessageRole get role => turn.speaker.isAgent ? AIMessageRole.assistant : AIMessageRole.user;

  @override
  String get text => turn.text;
}

join() doesn't throw: if it fails, for example because the microphone was taken away, the error is in voice.state.value.failure, which the screen shows. leave() leaves the call and ends the session start opened.

Show what the agent is doing

AgentGlow shows the agent as an AIVoiceGlowView. The call says when the agent speaks, and how loud, and the session says when it listens or thinks:

lib/agent_glow.dart
import 'package:flutter/material.dart';
import 'package:stream_chat_flutter_ai/stream_chat_flutter_ai.dart';
import 'package:stream_agents_core/stream_agents_core.dart';
import 'package:stream_agents_rtc/stream_agents_rtc.dart';

class AgentGlow extends StatelessWidget {
  const AgentGlow({super.key, required this.voice});

  /// A voice session that has joined its call.
  final VoiceSession voice;

  @override
  Widget build(BuildContext context) {
    final call = voice.call!;
    return StreamBuilder(
      stream: call.state.valueStream,
      initialData: call.state.value,
      builder: (context, callState) {
        // The agent is the call's other participant.
        final agent = callState.requireData.callParticipants.where((p) => !p.isLocal).firstOrNull;
        final isSpeaking = agent?.isSpeaking ?? false;
        return StreamBuilder(
          stream: voice.session.conversation.stream,
          initialData: voice.session.conversation.value,
          builder: (context, conversation) => SizedBox.square(
            dimension: 220,
            child: AIVoiceGlowView(
              state: isSpeaking
                  ? AIVoiceAgentState.speaking
                  : switch (conversation.requireData.state) {
                      Listening() => AIVoiceAgentState.listening,
                      Responding() || Working() => AIVoiceAgentState.thinking,
                      Idle() || Ended() => AIVoiceAgentState.idle,
                    },
              // The agent's voice, so the glow swells with its words.
              audioLevel: isSpeaking ? agent!.audioLevel : null,
            ),
          ),
        );
      },
    );
  }
}

The glow takes the theme's accentPrimary unless you pass it colors, and AIVoiceWaveView can show the user's microphone the same way, from call.state.value.localParticipant. Voice Visualizations lists the options of both.

Ask before a tool runs

A tool of your app that asks first, an AgentTool with an approval, waits in the session's approvals until the user answers. Pass your tools to VoiceAgentScreen, which gives them to VoiceSession.start. On a call, show the question over it with AIToolApprovalCard:

lib/approval_card.dart
import 'package:flutter/material.dart';
import 'package:stream_chat_flutter_ai/stream_chat_flutter_ai.dart';
import 'package:stream_agents_core/stream_agents_core.dart';

class ApprovalCard extends StatefulWidget {
  const ApprovalCard({super.key, required this.request, required this.session});

  final ToolApprovalRequest request;
  final AgentSession session;

  @override
  State<ApprovalCard> createState() => _ApprovalCardState();
}

class _ApprovalCardState extends State<ApprovalCard> {
  AIToolApprovalState _state = const AIToolApprovalIdle();

  /// Sending lasts until the call leaves the session's approvals, which takes the card away.
  void _answer(AIToolApprovalDecision decision) {
    setState(() => _state = AIToolApprovalSending(decision));
    try {
      widget.session.decide(widget.request.id, allowed: decision == AIToolApprovalDecision.allowed);
    } on StateError catch (error) {
      setState(() => _state = AIToolApprovalFailed(decision, error));
    } on SocketClosedException catch (error) {
      setState(() => _state = AIToolApprovalFailed(decision, error));
    }
  }

  @override
  Widget build(BuildContext context) {
    final request = widget.request;
    return Padding(
      padding: const EdgeInsets.symmetric(horizontal: 16),
      child: AIToolApprovalCard(
        approval: AIToolApproval(
          title: request.approval.title,
          message: request.approval.message,
          reason: request.reason.isEmpty ? null : request.reason,
          allowTitle: request.approval.allowTitle,
          declineTitle: request.approval.declineTitle,
        ),
        state: _state,
        onAnswer: _answer,
      ),
    );
  }
}

Then import approval_card.dart in lib/voice_call_screen.dart, and add it to the Column of VoiceCallScreen, above the controls, where session is widget.toolSession ?? voice.session:

// The question of a tool that asks first, over the call.
StreamBuilder(
  stream: session.approvals.stream,
  initialData: session.approvals.value,
  builder: (context, approvals) => switch (approvals.requireData.firstOrNull) {
    // Keyed by the call, so the next question starts with no answer sending.
    final request? => ApprovalCard(key: ValueKey(request.id), request: request, session: session),
    null => const SizedBox.shrink(),
  },
),

Show the agent the camera

An agent with the vision skill can look through the phone's camera. Join with the camera on with voice.join(camera: true), or turn it on during the call with voice.setCameraEnabled(true). The camera starts on the back lens, and when the agent looks, the session answers with pictures the camera takes then. CameraButton asks for the camera on Android first, and needs NSCameraUsageDescription in your Info.plist and CAMERA in your AndroidManifest.xml:

lib/camera_button.dart
import 'dart:io' show Platform;

import 'package:flutter/material.dart';
import 'package:permission_handler/permission_handler.dart';
import 'package:stream_agents_rtc/stream_agents_rtc.dart';

class CameraButton extends StatelessWidget {
  const CameraButton({super.key, required this.voice});

  final VoiceSession voice;

  Future<void> _toggle(bool isOn) async {
    // The camera needs its permission before it turns on. iOS asks by itself.
    if (!isOn && Platform.isAndroid && !await Permission.camera.request().isGranted) return;
    await voice.setCameraEnabled(!isOn);
  }

  @override
  Widget build(BuildContext context) => StreamBuilder(
    stream: voice.state.stream,
    initialData: voice.state.value,
    builder: (context, snapshot) {
      final state = snapshot.requireData;
      return IconButton.filledTonal(
        tooltip: state.isCameraEnabled ? 'Stop showing the camera' : 'Show the agent the camera',
        iconSize: 28,
        onPressed: state.isJoined ? () => _toggle(state.isCameraEnabled) : null,
        icon: Icon(state.isCameraEnabled ? Icons.videocam : Icons.videocam_off),
      );
    },
  );
}

Then import camera_button.dart in lib/voice_call_screen.dart, and add it to the controls of VoiceCallScreen, between the mute and hang-up buttons:

CameraButton(voice: voice),

Video covers what the agent sees.

Next steps