Info:
This is documentation for Stream Chat React Native SDK v9, which is no longer actively maintained. For up-to-date documentation, see the latest version (v10).

Voice Agent

Talking is often faster than typing, and it's the only way to use an app with busy hands. A voice agent listens to the user, answers out loud, and carries on, like a phone call with your assistant.

Note:

What you'll build

  • A voice mode button after the composer, which opens a full-screen voice conversation
  • The agent as a glowing globe that shows whether it's listening, thinking or speaking
  • The user's voice as a wave, while they talk
  • What was heard and what the agent said, written under the globe
  • The question a tool asks before it runs, over the voice screen

Starts from: the Quickstart, with a conversation controller and a session, as in the Quickstart's AgentConversation, on stream-chat-react-native or stream-chat-expo 9.14.0 or later, with dictation set up.

Info:

On React Native, voice runs on the phone. Dictation, the platform's speech recognition, listens to the user. The controller sends what it heard as a question of the conversation, and your agent answers in writing, as it always does. When the reply is done, a text-to-speech library reads it aloud, and the voice mode listens again. The user's words and the agent's replies are written into the same conversation, so the user can go back to typing at any time.

The call-based voice session that VoiceSession from @stream-io/stream-agents/video starts, with the agent's own speech models on a Stream Video call, is not part of these components, and this page doesn't use it.

Best practices

  • Show what the agent is doing at every moment. A silent screen while the agent thinks feels broken.
  • Write out what was heard and what was said. Users glance at it to check a name, a number or an address.
  • Tell the agent in its instructions that its answers are read aloud, so it keeps them short.
  • Make closing voice mode obvious, so the phone stops listening.
  • Ask before a tool that shares or changes something, as in a typed conversation.

Give the agent a voice

Any agent can be heard this way, since the phone does the listening and the speaking. Make it from the text template, and tell it how to write for speech:

getstream agents init my-agent --template text
agents/my-agent/instructions.md
Your answers are read aloud. Answer in short sentences and plain words. Don't use
lists, tables, headings or code, and say numbers and dates the way you'd say them.

Sync it with getstream agents sync my-agent. An agent made with the voice template adds speech-to-text and text-to-speech models, a voice and a greeting, which only a call uses.

Reading the reply aloud needs a text-to-speech library. This page uses expo-speech, which needs no permission:

npx expo install expo-speech

In a React Native CLI app, use a text-to-speech library of your choice, and call it where this page calls Speech. Listening needs the microphone and speech recognition permissions of dictation.

Listen, ask, speak

useVoiceConversation runs the loop of a voice conversation on the conversation's controller. It listens with dictation, sends what it heard, waits until the controller says the reply is done, reads the reply aloud, and listens again. It returns what the agent is doing, as a VoiceAgentState for VoiceGlow, and what was heard and said:

useVoiceConversation.ts
import * as Speech from "expo-speech";
import { useCallback, useEffect, useRef, useState } from "react";
import {
  type AIConversationController,
  type AIConversationControllerState,
  dictationStore,
  isAIConversationAnswering,
  useAIConversationState,
  useDictation,
  useDictationState,
  useDictationTranscript,
  type VoiceAgentState,
} from "stream-chat-react-native/ai-components";

// Marks voice mode's dictation as its own, so the composer underneath ignores it.
const DICTATION_SOURCE = { source: "voice-mode" };

const selector = ({ messages, status }: AIConversationControllerState) => ({
  messages,
  status,
});

/** Markdown read aloud as plain words. */
const toSpeech = (text: string) =>
  text
    .replace(/```[\s\S]*?```/g, " ")
    .replace(/!\[[^\]]*]\([^)]*\)/g, " ")
    .replace(/\[([^\]]+)]\([^)]*\)/g, "$1")
    .replace(/[*_#>`~|]/g, "")
    .replace(/\s+/g, " ")
    .trim();

export const useVoiceConversation = (
  conversation: AIConversationController,
  active: boolean,
) => {
  const [phase, setPhase] = useState<VoiceAgentState>("idle");
  const [reply, setReply] = useState("");
  const { cancel, start } = useDictation(undefined, DICTATION_SOURCE);
  const { error, state: dictationState } = useDictationState();
  const { transcript } = useDictationTranscript();
  const { messages, status } = useAIConversationState(conversation, selector);

  const messagesRef = useRef(messages);
  messagesRef.current = messages;
  // The question being answered, with the messages already there when it was sent,
  // so its reply is found by identity.
  const turn = useRef<{ seenIds: Set<string>; sawAnswering: boolean } | null>(
    null,
  );
  // An error left over from an earlier turn, ignored until dictation starts again.
  const staleError = useRef(dictationStore.getLatestValue().error);

  const listen = useCallback(() => {
    if (!active) return;
    turn.current = null;
    staleError.current = dictationStore.getLatestValue().error;
    setReply("");
    setPhase("connecting");
    void start();
  }, [active, start]);

  // Ends the agent's turn: reads its answer aloud if there is one, then listens again.
  const finishTurn = useCallback(() => {
    const seenIds = turn.current?.seenIds ?? new Set<string>();
    turn.current = null;
    if (!active) return;
    const answer = [...messagesRef.current]
      .reverse()
      .find(
        (message) =>
          message.role === "assistant" &&
          message.text &&
          !seenIds.has(message.id),
      );
    if (!answer) {
      listen();
      return;
    }
    setReply(answer.text);
    setPhase("speaking");
    Speech.speak(toSpeech(answer.text), {
      onDone: listen,
      onError: listen,
      // Its own audio session on iOS, so the reply is heard whatever dictation left the app's in.
      useApplicationAudioSession: false,
    });
  }, [active, listen]);

  // Starts listening when voice mode opens, and stops everything when it closes.
  useEffect(() => {
    if (!active) return;
    listen();
    return () => {
      turn.current = null;
      cancel();
      void Speech.stop();
      setPhase("idle");
    };
  }, [active, cancel, listen]);

  // Follows dictation: listening while it records, and sends what it heard once it stops.
  useEffect(() => {
    if (!active || turn.current) return;
    if (dictationState === "listening") {
      setPhase("listening");
      return;
    }
    if (dictationState !== "idle") return;
    if (phase !== "listening" && phase !== "connecting") return;
    if (error && error !== staleError.current) {
      setPhase("idle");
      return;
    }
    if (phase === "connecting") return;
    const text = transcript.trim();
    if (!text) {
      // Nothing was heard: keep listening.
      listen();
      return;
    }
    turn.current = {
      sawAnswering: false,
      seenIds: new Set(messagesRef.current.map((message) => message.id)),
    };
    setPhase("thinking");
    conversation.send({ text });
  }, [active, conversation, dictationState, error, listen, phase, transcript]);

  // Follows the agent: its turn ends once it has been answering and is done.
  useEffect(() => {
    const current = turn.current;
    if (!active || !current) return;
    if (isAIConversationAnswering(status)) {
      current.sawAnswering = true;
    } else if (status === "error") {
      turn.current = null;
      setPhase("idle");
    } else if (current.sawAnswering) {
      finishTurn();
    }
  }, [active, finishTurn, status]);

  return {
    // Only an error from the current attempt, not one left over while dictation restarts.
    error: phase === "idle" && error !== staleError.current ? error : null,
    listen,
    phase,
    reply,
    transcript,
  };
};

useDictation listens, useDictationState follows the recognizer, and useDictationTranscript has the words so far, which interimResults update while the user speaks. Dictation ends by itself after a stretch of silence, silenceTimeoutMs, which is when the question is sent. The controller moves from 'submitted' to 'streaming' to 'ready' as the agent answers, and to 'error' when it can't, so the loop never waits on a reply that isn't coming. Conversation state lists the states.

Show the voice screen

VoiceMode is a button that opens the voice screen. The screen shows the agent as a VoiceGlow, moved by the phase the hook returns, and the user's own voice as a VoiceWave, moved by the microphone. The StreamTheme around them picks a dark backdrop and their colors:

VoiceMode.tsx
import { useState } from "react";
import { Modal, Pressable, StyleSheet, Text, View } from "react-native";
import { useSafeAreaInsets } from "react-native-safe-area-context";
import type { Session } from "@stream-io/stream-agents";
import {
  type AIConversationController,
  StreamTheme,
  useDictationAudioLevel,
  type VoiceAgentState,
  VoiceGlow,
  VoiceWave,
} from "stream-chat-react-native/ai-components";

import { ToolQuestion } from "./ToolQuestion";
import { useVoiceConversation } from "./useVoiceConversation";

const BACKDROP = "#0b0b10";
const COLORS = ["#005fff", "#7b5cff", "#00c2ff"];

const LABELS: Record<VoiceAgentState, string> = {
  connecting: "Connecting...",
  idle: "Tap the glow to talk",
  listening: "Listening",
  speaking: "Speaking",
  thinking: "Thinking",
};

type VoiceModeProps = {
  conversation: AIConversationController;
  session: Session;
};

const VoiceScreen = ({
  conversation,
  onClose,
  session,
}: VoiceModeProps & { onClose: () => void }) => {
  const { bottom, top } = useSafeAreaInsets();
  const { error, listen, phase, reply, transcript } = useVoiceConversation(
    conversation,
    true,
  );
  // The microphone's level, which moves the wave while the user talks.
  const micLevel = useDictationAudioLevel();

  return (
    <StreamTheme
      style={{
        voiceVisualization: { backdropColor: BACKDROP, colors: COLORS },
      }}
    >
      <View
        style={[
          styles.screen,
          { paddingBottom: bottom + 24, paddingTop: top + 12 },
        ]}
      >
        <Pressable
          accessibilityLabel="Close voice mode"
          hitSlop={12}
          onPress={onClose}
          style={styles.close}
        >
          <Text style={styles.closeText}>Close</Text>
        </Pressable>
        <Pressable
          accessibilityHint="Starts listening"
          disabled={phase !== "idle"}
          onPress={listen}
          style={styles.glowArea}
        >
          <VoiceGlow state={phase} style={styles.glow} />
        </Pressable>
        <Text style={styles.label}>
          {error
            ? "Couldn't hear you. Tap the glow to try again."
            : LABELS[phase]}
        </Text>
        <Text numberOfLines={4} style={styles.caption}>
          {phase === "listening" ? transcript : reply}
        </Text>
        <ToolQuestion session={session} />
        <VoiceWave
          accessibilityLabel="Your voice"
          audioLevel={micLevel}
          state={phase === "listening" ? "listening" : "idle"}
          style={styles.wave}
        />
      </View>
    </StreamTheme>
  );
};

/** A button that opens a full-screen voice conversation with the agent. */
export const VoiceMode = (props: VoiceModeProps) => {
  const [open, setOpen] = useState(false);

  return (
    <>
      <Pressable
        accessibilityLabel="Start voice mode"
        accessibilityRole="button"
        onPress={() => setOpen(true)}
        style={styles.button}
      >
        <Text>Voice</Text>
      </Pressable>
      <Modal
        animationType="fade"
        onRequestClose={() => setOpen(false)}
        visible={open}
      >
        {open ? (
          <VoiceScreen {...props} onClose={() => setOpen(false)} />
        ) : null}
      </Modal>
    </>
  );
};

const styles = StyleSheet.create({
  button: {
    alignItems: "center",
    height: 44,
    justifyContent: "center",
    width: 56,
  },
  caption: {
    color: "#ffffff",
    fontSize: 17,
    paddingHorizontal: 24,
    textAlign: "center",
  },
  close: { alignItems: "flex-end", paddingHorizontal: 20 },
  closeText: { color: "#ffffff", fontSize: 17 },
  glow: { aspectRatio: 1, width: "100%" },
  glowArea: { alignSelf: "stretch", flex: 1, justifyContent: "center" },
  label: {
    color: "#c9cbd3",
    fontSize: 17,
    marginVertical: 16,
    textAlign: "center",
  },
  screen: { backgroundColor: BACKDROP, flex: 1 },
  wave: { alignSelf: "stretch", height: 140 },
});

Mount VoiceScreen only while the voice screen is open, as above: its hook starts listening when it mounts, and cancels dictation and speech when it unmounts, so closing the screen stops everything. Put the button after the composer with its trailing prop, and give it the controller and the session:

<ComposerView
  conversation={conversation}
  trailing={<VoiceMode conversation={conversation} session={session} />}
/>

The first time the voice screen opens, the system asks for the microphone, and for speech recognition on iOS. If the user says no, error is set, and the screen says it couldn't hear them. The VoiceMode here asks the same conversation as the composer, so what the user says shows up in the conversation behind it, as a question with its reply, with the reasoning and tool calls that led to it.

Show what the agent is doing

phase is already one of the glow's states, worked out from dictation and the controller together:

  • connecting until dictation is ready to hear.
  • listening while dictation records. The wave follows the microphone, and the transcript shows the words so far.
  • thinking from the moment the question is sent until the reply is done. The controller's activity says more, such as "Working on your calendar", when you want to show it, as Follow its state shows.
  • speaking while the phone reads the reply aloud. The glow animates speech on its own, since the phone gives no level to follow.
  • idle otherwise, and when listening failed. Tapping the glow listens again.

VoiceGlow and VoiceWave run on the UI thread and keep moving while the JS thread is busy. Voice visualizations lists the options of both, such as colors and expressivity, and how to feed them the level of a voice you measure yourself.

Ask before a tool runs

A tool the app gives the agent can ask the user before each call, as the Personal assistant use case's get_location does. In a typed conversation, the question shows under the call, in the conversation. Behind a full-screen voice screen it would be out of sight, so the screen shows it. A call that waits for an answer sits in session.conversation's approvals until session.decide answers it. Show the oldest one with AIToolApprovalCard:

ToolQuestion.tsx
import type { Session } from "@stream-io/stream-agents";
import { useSyncExternalStore } from "react";
import { AIToolApprovalCard } from "stream-chat-react-native/ai-components";

/** The oldest question a tool waits on. */
export const ToolQuestion = ({ session }: { session: Session }) => {
  const { approvals } = useSyncExternalStore(
    session.conversation.subscribe,
    session.conversation.get,
  );
  const request = approvals[0];
  if (!request) return null;

  return (
    <AIToolApprovalCard
      approval={{
        title: request.approval.title,
        message: request.approval.message,
        reason: request.reason,
        allowTitle: request.approval.allowTitle,
        declineTitle: request.approval.declineTitle,
      }}
      key={request.id}
      onDecide={(allowed) => session.decide(request.id, { allowed })}
    />
  );
};

Allowed, the tool runs and its result goes back to the agent. Declined, it never runs, and the agent is told the user said no. Tool approvals shows the same question in a typed conversation.

Switch between typing and talking

Typed and spoken turns are one conversation, with one history and one channel: the voice screen asks through the same controller as the composer. Closing the screen leaves you in the conversation, with everything that was said in it, and the composer's microphone button still dictates into the text field. Dictation started by the voice screen carries its own source, so the composer underneath, which only fills its text field from the dictation of its own button, leaves it alone.

Closing the screen stops the listening and the speaking. A question that was already sent is still answered: its reply arrives in the conversation, written, for the user to read.

Next steps