# Voice Agent

Talking is often faster than typing, and it's the only way to use an app with busy hands. A voice agent listens to the user, answers out loud, and lets them interrupt it, like a phone call with your assistant.

>
> **Note:** **What you'll build**
>
> - A call screen that puts your agent on a call with the user
> - The agent as a glowing globe that moves with its voice
> - What was said, written out under the globe
> - Mute and hang up
> - The question a tool asks before it runs, over the call
>

**Starts from:** the [Quickstart](https://getstream.io/chat/docs/sdk/flutter/ai-components/), with the user signed in to `agents`, the `StreamAgents` its `signIn` returns, kept where your screens reach it, such as `final agents = signIn('jane');` in `lib/agents.dart`.

>
> **Info:** `VoiceSession` does the hard part. It starts your agent on a call, and joins the call from the phone with Stream Video, which comes with `stream_agents_rtc`. The agent's speech-to-text, model and text-to-speech run on Stream, and when the user talks over the agent, it stops and listens.
>

## Best practices

- Show what the agent is doing at every moment. A silent screen while the agent thinks feels broken.
- Drive the globe from the agent's real voice, so it swells with its words.
- Write out what was said. Users glance at it to check a name, a number or an address.
- Make hanging up obvious, so the agent stops listening.

## Give the agent a voice

Any agent can talk, with the router's default speech models. Make it from the voice template to choose its models, voice and greeting:

```sh
getstream agents init my-agent --template voice
getstream agents sync my-agent
```

The call needs the microphone and the network. Declare them: `NSMicrophoneUsageDescription` in your `Info.plist`, and `INTERNET`, `ACCESS_NETWORK_STATE`, `RECORD_AUDIO`, `MODIFY_AUDIO_SETTINGS` and `BLUETOOTH_CONNECT` in your `AndroidManifest.xml`. Android also has the app ask for the microphone at runtime before it joins. [STT](https://getstream.io/agents/docs/stt/?client=flutter) and [TTS & Voices](https://getstream.io/agents/docs/tts/?client=flutter) list the models and voices.

## Start the call

`VoiceAgentScreen` asks for the microphone, starts a voice session with the agent you name, and shows the call once the agent is on it. Push it from anywhere, such as in a `MaterialPageRoute` with `fullscreenDialog: true`. It asks with `permission_handler`, so add it to your `pubspec.yaml`:

```yaml title="pubspec.yaml"
dependencies:
  # Asks for the microphone and the camera on Android.
  permission_handler: ^12.0.3
```

Then the screen:

```dart title="lib/voice_agent_screen.dart"
import 'dart:async';
import 'dart:io' show Platform;

import 'package:flutter/material.dart';
import 'package:permission_handler/permission_handler.dart';
import 'package:stream_chat_flutter_ai/stream_chat_flutter_ai.dart';
import 'package:stream_agents_core/stream_agents_core.dart';
import 'package:stream_agents_rtc/stream_agents_rtc.dart';

import 'agents.dart';
import 'voice_call_screen.dart';

class VoiceAgentScreen extends StatefulWidget {
  const VoiceAgentScreen({super.key, required this.agent, this.tools = const []});

  /// The agent to talk to, by the name it was synced under.
  final String agent;

  /// The tools of your app the agent can call.
  final List<AgentTool> tools;

  @override
  State<VoiceAgentScreen> createState() => _VoiceAgentScreenState();
}

class _VoiceAgentScreenState extends State<VoiceAgentScreen> {
  VoiceSession? _voice;

  @override
  void initState() {
    super.initState();
    _start();
  }

  Future<void> _start() async {
    // The call needs the microphone, so ask for it before the agent starts. iOS asks by
    // itself as the call is joined.
    if (Platform.isAndroid && !await Permission.microphone.request().isGranted) {
      _close();
      return;
    }
    try {
      // The agent joins the session's call right away.
      final voice = await VoiceSession.start(
        agents,
        agent: widget.agent,
        options: SessionOptions(tools: widget.tools),
      );
      if (!mounted) {
        await voice.leave();
        return;
      }
      setState(() => _voice = voice);
    } on Object {
      _close();
    }
  }

  void _close() {
    if (mounted) Navigator.of(context).pop();
  }

  @override
  void dispose() {
    // Leaving again after a hang-up does nothing.
    unawaited(_voice?.leave());
    super.dispose();
  }

  @override
  Widget build(BuildContext context) => switch (_voice) {
    final voice? => VoiceCallScreen(voice: voice, onHangUp: _close),
    null => const Scaffold(
      body: Center(
        child: SizedBox.square(
          dimension: 220,
          child: AIVoiceGlowView(state: AIVoiceAgentState.connecting),
        ),
      ),
    ),
  };
}
```

## Show the call

`VoiceCallScreen` joins the call from the phone, with the microphone on, and shows the agent, what was said, and the controls. `voice.session.conversation` holds the conversation as it's said, so the transcript grows as the user and the agent talk:

```dart title="lib/voice_call_screen.dart"
import 'package:flutter/material.dart';
import 'package:stream_chat_flutter_ai/stream_chat_flutter_ai.dart';
import 'package:stream_agents_core/stream_agents_core.dart';
import 'package:stream_agents_rtc/stream_agents_rtc.dart';

import 'agent_glow.dart';

class VoiceCallScreen extends StatefulWidget {
  const VoiceCallScreen({super.key, required this.voice, required this.onHangUp, this.toolSession});

  final VoiceSession voice;

  /// The session that holds your app's tools, if it isn't the call's own.
  final AgentSession? toolSession;

  /// Called once the user hung up and the phone left the call.
  final VoidCallback onHangUp;

  @override
  State<VoiceCallScreen> createState() => _VoiceCallScreenState();
}

class _VoiceCallScreenState extends State<VoiceCallScreen> {
  @override
  void initState() {
    super.initState();
    // Joins from this device, with the microphone on and the agent on the speaker.
    widget.voice.join();
  }

  Future<void> _hangUp() async {
    await widget.voice.leave();
    widget.onHangUp();
  }

  @override
  Widget build(BuildContext context) {
    final voice = widget.voice;
    return Scaffold(
      body: SafeArea(
        child: StreamBuilder(
          stream: voice.state.stream,
          initialData: voice.state.value,
          builder: (context, snapshot) {
            final state = snapshot.requireData;
            return Column(
              children: [
                const SizedBox(height: 24),
                if (state.isJoined)
                  AgentGlow(voice: voice)
                else
                  const SizedBox.square(
                    dimension: 220,
                    child: AIVoiceGlowView(state: AIVoiceAgentState.connecting),
                  ),
                Expanded(
                  child: StreamBuilder(
                    stream: voice.session.conversation.stream,
                    initialData: voice.session.conversation.value,
                    builder: (context, conversation) => AIConversationView(
                      messages: [for (final turn in conversation.requireData.turns) TurnMessage(turn)],
                    ),
                  ),
                ),
                if (state.failure case final failure?)
                  Padding(padding: const EdgeInsets.all(16), child: Text('$failure', textAlign: TextAlign.center)),
                Padding(
                  padding: const EdgeInsets.all(24),
                  child: Row(
                    mainAxisAlignment: MainAxisAlignment.center,
                    spacing: 40,
                    children: [
                      IconButton.filledTonal(
                        tooltip: state.isMuted ? 'Unmute' : 'Mute',
                        iconSize: 28,
                        onPressed: state.isJoined ? () => voice.setMuted(!state.isMuted) : null,
                        icon: Icon(state.isMuted ? Icons.mic_off : Icons.mic),
                      ),
                      IconButton.filled(
                        tooltip: 'Hang up',
                        iconSize: 28,
                        style: IconButton.styleFrom(backgroundColor: Colors.red),
                        onPressed: _hangUp,
                        icon: const Icon(Icons.call_end),
                      ),
                    ],
                  ),
                ),
              ],
            );
          },
        ),
      ),
    );
  }
}

/// A turn of the call, as the AI components show it.
class TurnMessage with AIMessage {
  TurnMessage(this.turn);

  final Turn turn;

  @override
  String get id => turn.id;

  @override
  AIMessageRole get role => turn.speaker.isAgent ? AIMessageRole.assistant : AIMessageRole.user;

  @override
  String get text => turn.text;
}
```

`join()` doesn't throw: if it fails, for example because the microphone was taken away, the error is in `voice.state.value.failure`, which the screen shows. `leave()` leaves the call and ends the session `start` opened.

## Show what the agent is doing

`AgentGlow` shows the agent as an `AIVoiceGlowView`. The call says when the agent speaks, and how loud, and the session says when it listens or thinks:

```dart title="lib/agent_glow.dart"
import 'package:flutter/material.dart';
import 'package:stream_chat_flutter_ai/stream_chat_flutter_ai.dart';
import 'package:stream_agents_core/stream_agents_core.dart';
import 'package:stream_agents_rtc/stream_agents_rtc.dart';

class AgentGlow extends StatelessWidget {
  const AgentGlow({super.key, required this.voice});

  /// A voice session that has joined its call.
  final VoiceSession voice;

  @override
  Widget build(BuildContext context) {
    final call = voice.call!;
    return StreamBuilder(
      stream: call.state.valueStream,
      initialData: call.state.value,
      builder: (context, callState) {
        // The agent is the call's other participant.
        final agent = callState.requireData.callParticipants.where((p) => !p.isLocal).firstOrNull;
        final isSpeaking = agent?.isSpeaking ?? false;
        return StreamBuilder(
          stream: voice.session.conversation.stream,
          initialData: voice.session.conversation.value,
          builder: (context, conversation) => SizedBox.square(
            dimension: 220,
            child: AIVoiceGlowView(
              state: isSpeaking
                  ? AIVoiceAgentState.speaking
                  : switch (conversation.requireData.state) {
                      Listening() => AIVoiceAgentState.listening,
                      Responding() || Working() => AIVoiceAgentState.thinking,
                      Idle() || Ended() => AIVoiceAgentState.idle,
                    },
              // The agent's voice, so the glow swells with its words.
              audioLevel: isSpeaking ? agent!.audioLevel : null,
            ),
          ),
        );
      },
    );
  }
}
```

The glow takes the theme's `accentPrimary` unless you pass it [`colors`](https://getstream.io/chat/docs/sdk/flutter/ai-components/voice-visualizations/#colors), and `AIVoiceWaveView` can show the user's microphone the same way, from `call.state.value.localParticipant`. [Voice Visualizations](https://getstream.io/chat/docs/sdk/flutter/ai-components/voice-visualizations/) lists the options of both.

## Ask before a tool runs

A tool of your app that asks first, an `AgentTool` with an `approval`, waits in the session's `approvals` until the user answers. Pass your tools to `VoiceAgentScreen`, which gives them to `VoiceSession.start`. On a call, show the question over it with `AIToolApprovalCard`:

```dart title="lib/approval_card.dart"
import 'package:flutter/material.dart';
import 'package:stream_chat_flutter_ai/stream_chat_flutter_ai.dart';
import 'package:stream_agents_core/stream_agents_core.dart';

class ApprovalCard extends StatefulWidget {
  const ApprovalCard({super.key, required this.request, required this.session});

  final ToolApprovalRequest request;
  final AgentSession session;

  @override
  State<ApprovalCard> createState() => _ApprovalCardState();
}

class _ApprovalCardState extends State<ApprovalCard> {
  AIToolApprovalState _state = const AIToolApprovalIdle();

  /// Sending lasts until the call leaves the session's approvals, which takes the card away.
  void _answer(AIToolApprovalDecision decision) {
    setState(() => _state = AIToolApprovalSending(decision));
    try {
      widget.session.decide(widget.request.id, allowed: decision == AIToolApprovalDecision.allowed);
    } on StateError catch (error) {
      setState(() => _state = AIToolApprovalFailed(decision, error));
    } on SocketClosedException catch (error) {
      setState(() => _state = AIToolApprovalFailed(decision, error));
    }
  }

  @override
  Widget build(BuildContext context) {
    final request = widget.request;
    return Padding(
      padding: const EdgeInsets.symmetric(horizontal: 16),
      child: AIToolApprovalCard(
        approval: AIToolApproval(
          title: request.approval.title,
          message: request.approval.message,
          reason: request.reason.isEmpty ? null : request.reason,
          allowTitle: request.approval.allowTitle,
          declineTitle: request.approval.declineTitle,
        ),
        state: _state,
        onAnswer: _answer,
      ),
    );
  }
}
```

Then import `approval_card.dart` in `lib/voice_call_screen.dart`, and add it to the `Column` of `VoiceCallScreen`, above the controls, where `session` is `widget.toolSession ?? voice.session`:

```dart
// The question of a tool that asks first, over the call.
StreamBuilder(
  stream: session.approvals.stream,
  initialData: session.approvals.value,
  builder: (context, approvals) => switch (approvals.requireData.firstOrNull) {
    // Keyed by the call, so the next question starts with no answer sending.
    final request? => ApprovalCard(key: ValueKey(request.id), request: request, session: session),
    null => const SizedBox.shrink(),
  },
),
```

## Show the agent the camera

An agent with the `vision` skill can look through the phone's camera. Join with the camera on with `voice.join(camera: true)`, or turn it on during the call with `voice.setCameraEnabled(true)`. The camera starts on the back lens, and when the agent looks, the session answers with pictures the camera takes then. `CameraButton` asks for the camera on Android first, and needs `NSCameraUsageDescription` in your `Info.plist` and `CAMERA` in your `AndroidManifest.xml`:

```dart title="lib/camera_button.dart"
import 'dart:io' show Platform;

import 'package:flutter/material.dart';
import 'package:permission_handler/permission_handler.dart';
import 'package:stream_agents_rtc/stream_agents_rtc.dart';

class CameraButton extends StatelessWidget {
  const CameraButton({super.key, required this.voice});

  final VoiceSession voice;

  Future<void> _toggle(bool isOn) async {
    // The camera needs its permission before it turns on. iOS asks by itself.
    if (!isOn && Platform.isAndroid && !await Permission.camera.request().isGranted) return;
    await voice.setCameraEnabled(!isOn);
  }

  @override
  Widget build(BuildContext context) => StreamBuilder(
    stream: voice.state.stream,
    initialData: voice.state.value,
    builder: (context, snapshot) {
      final state = snapshot.requireData;
      return IconButton.filledTonal(
        tooltip: state.isCameraEnabled ? 'Stop showing the camera' : 'Show the agent the camera',
        iconSize: 28,
        onPressed: state.isJoined ? () => _toggle(state.isCameraEnabled) : null,
        icon: Icon(state.isCameraEnabled ? Icons.videocam : Icons.videocam_off),
      );
    },
  );
}
```

Then import `camera_button.dart` in `lib/voice_call_screen.dart`, and add it to the controls of `VoiceCallScreen`, between the mute and hang-up buttons:

```dart
CameraButton(voice: voice),
```

[Video](https://getstream.io/agents/docs/video/?client=flutter) covers what the agent sees.

## Next steps

- [Personal assistant](https://getstream.io/chat/docs/sdk/flutter/ai-components/use-cases/personal-assistant/#talk-instead-of-typing): switch a typed conversation to voice and back.
- [Sessions](https://getstream.io/agents/docs/sessions/?client=flutter#voice-ui-transcript-captions-and-status): the voice UI's states and how a voice session ends.
- [Purchase a phone number](https://getstream.io/agents/docs/telephony/purchase-number/?client=flutter): the same agent on the phone.

---

For the most recent version of this documentation, visit [https://getstream.io/chat/docs/sdk/flutter/ai-components/use-cases/voice-agent/](https://getstream.io/chat/docs/sdk/flutter/ai-components/use-cases/voice-agent/).