Info:
This is documentation for Stream Chat React Native SDK v9, which is no longer actively maintained. For up-to-date documentation, see the latest version (v10).

Personal Assistant

A personal assistant knows what's on the user's plate. It knows where they are and what time it is for them, searches their email, keeps every conversation they've had with it, and listens when they'd rather talk than type.

Note:

What you'll build

  • A menu with the user's conversations, titled and searchable
  • The user's local time and location, as tools the agent runs on the phone
  • A question before the agent reads the user's location
  • The user's Gmail, which they connect from the conversation, in the system's web sign-in
  • Photos the user attaches to a question
  • A voice mode in the same conversation

Starts from: the Quickstart, with api, chatClient, openConversation and createAgentTransport in your project, on stream-chat-react-native or stream-chat-expo 9.14.0 or later, and the VoiceMode button of the Voice agent use case.

Best practices

  • Run on the phone what only the phone has, such as the user's local time or location. Only a tool's answer leaves the device.
  • Let the system ask for a permission when a tool first needs it, not when the app starts.
  • Ask before a tool that shares or changes something, such as the user's location.
  • Let users connect their own accounts, such as Gmail, from the conversation. The agent never sees their password.

Define the assistant

Make the agent from the voice template, so it can talk as well as write, and give it each user's Gmail as a connector:

getstream agents init assistant --template voice
agents/assistant/agent.yaml
name: assistant
mode: voice
llm: llm-fast
connectors:
  - gmail # each user signs in to their own Gmail

Write what the assistant is for in instructions.md, then sync it with getstream agents sync assistant. Each user's account covers connectors that act as the user, and Catalog lists the others, such as Google Calendar and Notion.

Give the agent the phone's tools

A tool the app gives the agent runs on the phone. When the model calls it, its run runs and returns the answer the model sees. get_location declares an approval, so the user is asked first, with the reason the model gives:

tools.ts
import { Tools } from "@stream-io/stream-agents";
import * as Location from "expo-location";

export const tools = new Tools()
  .register({
    name: "get_local_context",
    description:
      "The user's current date, local time, timezone and language. Call it before " +
      "answering anything that depends on today or the time.",
    parameters: { type: "object", properties: {} },
    run: () => {
      const { locale, timeZone } = Intl.DateTimeFormat().resolvedOptions();
      return `It is ${new Date().toString()}, in ${timeZone}. Their language is ${locale}.`;
    },
  })
  .register({
    name: "get_location",
    description:
      "Where the user is. Only call it when the answer depends on it, " +
      "such as the weather or places nearby.",
    parameters: {
      type: "object",
      properties: {
        reason: {
          type: "string",
          description: "why you need it, in a few words",
        },
      },
      required: ["reason"],
    },
    displayTitle: "Checking where you are",
    approval: {
      title: "Share your location?",
      reasonArgument: "reason",
      allowTitle: "Share",
      declineTitle: "Not now",
    },
    run: async () => {
      const { status } = await Location.requestForegroundPermissionsAsync();
      if (status !== "granted") {
        throw new Error("The user didn't allow access to their location.");
      }
      const { coords } = await Location.getCurrentPositionAsync();
      return `Latitude ${coords.latitude}, longitude ${coords.longitude}.`;
    },
  });

The system asks for the location the first time get_location runs, once the user has allowed the call. This example reads it with expo-location, which needs its config plugin and permission texts. In a React Native CLI app, use the geolocation library of your choice and declare the permissions it asks for. A tool that throws tells the model what went wrong, so the agent can say so. Tool calling & Dispatch covers them in full.

Open a conversation

Pass the tools every time a conversation opens, new or carried on, since the session runs them only while the app holds it. A new one also names where a connector's sign-in comes back to, a URL on your app's own scheme, the phone's time zone, and a label that the list finds the assistant's conversations by. openConversation from the quickstart then opens the session's channel:

assistant.ts
import { api, type AgentChat, openConversation } from "./agent";
import { tools } from "./tools";

/** Your app's own URL scheme, in reverse domain form. */
export const SIGN_IN_SCHEME = "com.example.assistant";
export const returnUrl = `${SIGN_IN_SCHEME}://sign-in`;

/** Marks the conversations this app opens, for the list to find. */
export const conversationLabel = { opened_from: "assistant" };

export async function openAssistant(sessionId?: string): Promise<AgentChat> {
  const assistant = api.agent("assistant");
  const session = sessionId
    ? await assistant.sessions.resume(sessionId, { tools })
    : await assistant.sessions.create({
        tools,
        returnUrl,
        custom: conversationLabel,
        // The agent tells the date and the hour in it. A phone app
        // names it, since none is sent otherwise.
        timezone: Intl.DateTimeFormat().resolvedOptions().timeZone,
      });
  return openConversation(session);
}

The sign-in comes back to com.example.assistant://sign-in, so register the scheme in your app, as you would for a deep link. The router takes only https URLs or a scheme in reverse domain form. With Expo, set the scheme in app.json. On Android with React Native CLI, add an intent filter for it to your main activity, and on iOS add it to the URL types of Info.plist:

app.json
{
  "expo": {
    "scheme": "com.example.assistant"
  }
}

With Expo Router, the return URL would route to a page that doesn't exist, so keep the app on the screen the sign-in was opened from. The SDK reads the URL itself:

app/+native-intent.tsx
export const redirectSystemPath = ({
  path,
}: {
  initial: boolean;
  path: string | null;
}) => (path?.includes("sign-in") ? null : path);

List the user's conversations

Each conversation is a session, which the router titles after its first answers. ConversationList lists the conversations this app opened, by their label, and searches them. A conversation the user switched to voice keeps its label, so it stays in the list, and a call started elsewhere, which has none, stays out. It loads again whenever version changes, which AssistantHome below bumps when a conversation is created or answered:

ConversationList.tsx
import type { Schemas } from "@stream-io/stream-agents";
import { useEffect, useState } from "react";
import { FlatList, Pressable, Text, TextInput, View } from "react-native";

import { api } from "./agent";
import { conversationLabel } from "./assistant";

// When the agent last answered, which moves only when a conversation is used.
// The router orders by when one last changed, which opening it also does.
const lastUsed = (session: Schemas["Session"]) =>
  Date.parse(session.last_response_at ?? session.created_at ?? "") || 0;

export const ConversationList = ({
  onSelect,
  selectedId,
  version,
}: {
  onSelect: (sessionId: string | undefined) => void;
  /** The open conversation, marked in the list. */
  selectedId?: string;
  /** Changes when the list may have: a conversation created, or answered. */
  version: number;
}) => {
  const [sessions, setSessions] = useState<Schemas["Session"][]>([]);
  const [search, setSearch] = useState("");

  useEffect(() => {
    let isCurrent = true;
    const all = api.agent("assistant").sessions;
    // The conversations this app opened, typed or switched to voice.
    const filters = { limit: 50, custom: conversationLabel };
    const page = search ? all.search(search, filters) : all.query(filters);
    page.then(
      ({ items }) => {
        // An incognito session keeps no conversation to open.
        if (isCurrent) {
          setSessions(
            items
              .filter((each) => each.conversation_id)
              .sort((a, b) => lastUsed(b) - lastUsed(a)),
          );
        }
      },
      () => undefined,
    );
    return () => {
      isCurrent = false;
    };
  }, [search, version]);

  return (
    <View style={{ flex: 1, gap: 8, padding: 12 }}>
      <Pressable accessibilityRole="button" onPress={() => onSelect(undefined)}>
        <Text>New conversation</Text>
      </Pressable>
      <TextInput onChangeText={setSearch} placeholder="Search" value={search} />
      <FlatList
        data={sessions}
        keyExtractor={(session) => session.id}
        renderItem={({ item }) => (
          <Pressable
            accessibilityRole="button"
            accessibilityState={{ selected: item.id === selectedId }}
            onPress={() => onSelect(item.id)}
          >
            <Text numberOfLines={1}>{item.title || "New conversation"}</Text>
          </Pressable>
        )}
      />
    </View>
  );
};

Put it together

AssistantHome shows the list in a menu beside the open conversation, and opens a conversation when the user picks one in the list. Opening from the tap rather than from an effect creates one session per New conversation, and a conversation that's no longer shown stops being watched. The conversation's approver answers the questions of the phone's tools, and gets an authorizer from createAuthSessionAuthorizer, so when the agent first needs Gmail, the reply shows a Connect card that signs the user in, in the system's web authentication session:

AssistantHome.tsx
import { useCallback, useEffect, useRef, useState } from "react";
import { Modal, Pressable, Text, View } from "react-native";
import {
  AIClientIdentity,
  AIConversationController,
  type AIConversationControllerState,
  ComposerView,
  createAuthSessionAuthorizer,
  isAIConversationAnswering,
  useAIConversationState,
} from "stream-chat-react-native/ai-components";
import { AIConversation } from "stream-chat-react-native/ai-components/conversation";

import { type AgentChat, chatClient } from "./agent";
import { createAgentTransport } from "./agentTransport";
import { openAssistant, SIGN_IN_SCHEME } from "./assistant";
import { ConversationList } from "./ConversationList";

const messageOf = (error: unknown) =>
  error instanceof Error ? error.message : String(error);

export const AssistantHome = () => {
  const [chat, setChat] = useState<AgentChat>();
  const [isOpening, setIsOpening] = useState(false);
  const [failure, setFailure] = useState<string>();
  const [isMenuOpen, setIsMenuOpen] = useState(false);
  // Bumped when the list may have changed, so it loads again.
  const [listVersion, setListVersion] = useState(0);
  const refreshList = useCallback(() => setListVersion((n) => n + 1), []);
  // The latest pick, so an open that a later pick overtook is dropped.
  const latestPick = useRef(0);

  // Opens from the list's tap, never from an effect, so New conversation
  // creates one session per tap.
  const open = async (sessionId: string | undefined) => {
    const pick = ++latestPick.current;
    setIsMenuOpen(false);
    setChat(undefined);
    setFailure(undefined);
    setIsOpening(true);
    try {
      const opened = await openAssistant(sessionId);
      if (pick !== latestPick.current) {
        void opened.session.stopWatching();
        return;
      }
      setChat(opened);
      if (!sessionId) refreshList();
    } catch (error) {
      if (pick === latestPick.current) setFailure(messageOf(error));
    } finally {
      if (pick === latestPick.current) setIsOpening(false);
    }
  };

  // Stops watching a conversation once another one opens, or the screen goes.
  // The conversation goes on; picking it again carries it on.
  useEffect(() => {
    if (!chat) return;
    return () => void chat.session.stopWatching();
  }, [chat]);

  return (
    <View style={{ flex: 1 }}>
      <Pressable accessibilityRole="button" onPress={() => setIsMenuOpen(true)}>
        <Text>Conversations</Text>
      </Pressable>
      <Modal
        animationType="slide"
        onRequestClose={() => setIsMenuOpen(false)}
        visible={isMenuOpen}
      >
        <ConversationList
          onSelect={(sessionId) => void open(sessionId)}
          selectedId={chat?.session.id}
          version={listVersion}
        />
      </Modal>
      {chat ? (
        <AssistantConversation
          chat={chat}
          key={chat.session.id}
          onReplied={refreshList}
        />
      ) : (
        <Text style={{ textAlign: "center" }}>
          {failure ??
            (isOpening
              ? "Opening..."
              : "Pick a conversation, or start a new one.")}
        </Text>
      )}
    </View>
  );
};

const statusSelector = ({ status }: AIConversationControllerState) => ({
  status,
});

const AssistantConversation = ({
  chat,
  onReplied,
}: {
  chat: AgentChat;
  /** Called when a reply ends, which is when the router titles a new conversation. */
  onReplied: () => void;
}) => {
  const { channel, session } = chat;
  const [conversation] = useState(
    () =>
      new AIConversationController({
        actions: { edit: false, regenerate: false },
        approver: {
          clientId: AIClientIdentity.installId(),
          decide: async (call, allowed) => session.decide(call.id, { allowed }),
          userId: chatClient.userID ?? "",
          authorizer: createAuthSessionAuthorizer({
            callbackScheme: SIGN_IN_SCHEME,
            decline: async (call) => session.declineLogin(call.id),
          }),
        },
        transport: createAgentTransport(session, channel),
      }),
  );
  useEffect(() => () => conversation.dispose(), [conversation]);

  const { status } = useAIConversationState(conversation, statusSelector);
  const wasAnswering = useRef(false);
  useEffect(() => {
    const isAnswering = isAIConversationAnswering(status);
    if (wasAnswering.current && !isAnswering) onReplied();
    wasAnswering.current = isAnswering;
  }, [onReplied, status]);

  return (
    <View style={{ flex: 1 }}>
      <AIConversation conversation={conversation} />
      <ComposerView conversation={conversation} />
    </View>
  );
};

The screen renders inside StreamTheme and KeyboardProvider, as in the Quickstart. In a bigger app, put the list in the drawer of your navigation library instead of a Modal.

The reply shows the question of get_location under its call, and the Connect card when a Gmail call waits for the user. Connect opens the provider's sign-in in ASWebAuthenticationSession on iOS, and in the browser's Auth Tab or a Custom Tab on Android. When the user is back, the call carries on in the same reply. Not now calls session.declineLogin, so the call runs nothing and the agent is told the user didn't connect. Tool approvals and Tool sign-ins cover both cards. The sign-in needs the SDK's native code, so with Expo, run the app in a development build.

Send photos

The transport above sends only the text. To send the photos the user attaches, pick them with the composer, as Composer shows, and pass them to responses.create as images. The photos arrive in the transport's send as request.attachments, files on the phone. The router takes photos inline, as data URIs, so shrink each one to a size the model needs first:

photos.ts
import { manipulateAsync, SaveFormat } from "expo-image-manipulator";

const MAX_IMAGE_WIDTH = 1280;

/** A photo the user picked, as the data URI the router takes. */
export const toDataUri = async (uri: string) => {
  const { base64 } = await manipulateAsync(
    uri,
    [{ resize: { width: MAX_IMAGE_WIDTH } }],
    { base64: true, compress: 0.7, format: SaveFormat.JPEG },
  );
  return `data:image/jpeg;base64,${base64}`;
};

Send them from a transport that builds on the quickstart's, and add the chat option that goes with it:

agentTransport.ts
import type { Session } from "@stream-io/stream-agents";
import type { Channel } from "stream-chat";
import type {
  AIChatOption,
  AIConversationTransport,
} from "stream-chat-react-native/ai-components";

import { toDataUri } from "./photos";

// createAgentTransport is the quickstart's, in this file.
export const thinkHarder: AIChatOption = {
  id: "think_harder",
  title: "Think harder",
  description: "More time to reason before answering",
  shortTitle: "Think",
};

export const createAssistantTransport = (
  session: Session,
  channel: Channel,
): AIConversationTransport => ({
  ...createAgentTransport(session, channel),
  send: async ({ attachments, chatOption, text }) => {
    const images = await Promise.all(
      attachments
        .filter((asset) => !asset.type || asset.type.startsWith("image"))
        .map(async (asset) => ({ url: await toDataUri(asset.uri) })),
    );
    await session.responses.create(text, {
      ...(images.length > 0 ? { images } : {}),
      ...(chatOption?.id === thinkHarder.id ? { reasoningEffort: "high" } : {}),
    });
  },
});

Use createAssistantTransport in AssistantConversation, and pass chatOptions={[thinkHarder]} to the ComposerView. Pick a model that reads images in agent.yaml. reasoningEffort is for an option such as "Think harder", which the composer's chat options add. Photos need expo-image-picker or react-native-image-picker, with their permissions, and the resizing here uses expo-image-manipulator: with React Native CLI, use an image library of your choice.

Talk instead of typing

A voice mode button after the composer switches the open conversation to voice. Dictation listens, the controller sends what the user said as a question of the same conversation, and when the reply is done, the phone reads it aloud. What's said is written into the same conversation, so the user can go back to typing at any time. Pass the button as the composer's trailing:

<ComposerView
  chatOptions={[thinkHarder]}
  conversation={conversation}
  trailing={<VoiceMode conversation={conversation} session={session} />}
/>

VoiceMode is built in the Voice agent use case.

Next steps