# Quickstart

Building your own text, voice or video AI is easy.
In this example we'll set up your agent, knowledge, skills, sandbox, tool calling and chat UI.

Stream Agents is hosted on edge servers around the world for optimal latency.
It's open source so you can also [self-host](https://getstream.io/agents/docs/ios/self-hosting/).

Time to get started. If you get stuck at any point hit the "Ask AI" button above.
The ask AI feature was actually created using this framework as well.

## The quickest path

If you're in a hurry you can skip this tutorial and paste one of the 3 example prompts in Claude.

Build me a Go based voice AI that takes orders for my restaurant. Optimize for Dutch language and give me a phone number in Amsterdam: getstream.io/SKILL-agents.md

Build me a Go based video AI that evaluates which strawberries are ripe and counts them: getstream.io/SKILL-agents.md

Build me a Go based text AI that answers support questions, reads my SDKs source code to help customers: getstream.io/SKILL-agents.md

## Stream app setup

Get the beta [Stream CLI](https://getstream.io/cli/docs/install/) and pick or create the Stream app your agent runs in. The `getstream agents` commands are only in beta builds: a stable `getstream` answers them with "agents unavailable".
The installer puts `getstream` in `~/.local/bin` and, if that is not on your `PATH`, adds it to your shell's rc file (`~/.zshrc`, `~/.bashrc`, `~/.bash_profile` or `~/.profile`, or fish's `config.fish`).

```sh
curl -fsSL https://getstream.io/cli/beta/install.sh | bash # the agents commands are in the beta CLI
getstream --version # a beta build shows a date and commit, such as 2026-10-14-3178aae

getstream init # pick or create a Stream app
```

## Create an agent

As a first step lets define an agent. Each agent consists of a folder with this structure:

```text
agents/my-agent/
  knowledge/       ← indexed markdown and URLs
    faq.md
    urls.yaml
  skills/          ← skills loaded as needed
    research.md
  agent.yaml       ← config for the agent
  instructions.md  ← system prompt
  guardrail.md     ← prevent abuse
```

The quickest way to create an agent folder is to use one of the templates offered by the beta CLI from [Stream app setup](#stream-app-setup):

```sh
getstream agents init my-agent --template text # or --template voice
```

You'll see the `agents/my-agent/` folder appear, with a lot of settings to play with. Start by editing the system prompt in `instructions.md` to match your use case.
When ready, run:

```sh
getstream agents sync my-agent
```

Now your agent is synced to Stream. Lets talk to your agent and see how it does:

```sh
getstream agents test my-agent
```

Ask it about Acme first, such as "How do I get started with Acme?". The template's guardrail turns away questions that are not about Acme Corp.

As a next step lets edit `agent.yaml`. We'll enable GitHub.
This agent doesn't need to run code, so lets leave the sandbox disabled.

```yaml title="agents/my-agent/agent.yaml"
name: my-agent
mode: text
# the model to use, can reference a router or a model
llm: llm-fast
# connectors & MCP
connectors:
  - name: github
    connection: company # one GitHub account your app owns
# no sandbox: this agent doesn't run code
```

`connection: company` is one GitHub account your app owns, so nobody signs in. Create it first, as in [Shared account](https://getstream.io/agents/docs/ios/connectors/shared-account/).

Run the agent and ask: "What are the open issues in GetStream/awesome-ai-news?"

```sh
getstream agents test my-agent --sync
```

These building blocks:

- Skills
- Knowledge
- Tools
- Guardrail
- Sandbox

Enable you to build any agent. As a next step lets setup tool calling.

## Tool Calling

Tool calling makes it easy to integrate agents with your systems.

If you're building a customer support bot you typically want tool calling for:

- Creating a ticket
- Looking up a ticket
- Replying to a customer etc

If you're building legal AI you might want to expose various legal databases through tool calling.

The example below how to setup a tool for a weather lookup:

```ts label="JavaScript"
import { Client, Dispatch } from "@stream-io/calypso";

const client = new Client();
const agent = client.agent("my-agent");

agent.tools.register<{ location: string }>({
  name: "weather_lookup",
  description: "Look up the current weather for a location",
  parameters: {
    type: "object",
    properties: {
      location: { type: "string", description: "a city or region" },
    },
    required: ["location"],
  },
  run: ({ location }) => {
    const place = location.toLowerCase();
    if (place.includes("colorado")) return "sunny";
    if (place.includes("amsterdam")) return "rainy";
    if (place.includes("san francisco")) return "foggy";
    return `no forecast for ${location}`;
  },
});

await new Dispatch().host(agent).run();
```

```python label="Python"
import asyncio

from vision_agents.plugins import stream

agent = stream.Client().agent("my-agent")


@agent.register(description="Look up the current weather for a location")
async def weather_lookup(location: str) -> str:
    place = location.lower()
    if "colorado" in place:
        return "sunny"
    if "amsterdam" in place:
        return "rainy"
    if "san francisco" in place:
        return "foggy"
    return f"no forecast for {location}"


async def main() -> None:
    dispatch = stream.Dispatch()
    dispatch.host(agent)
    await dispatch.run()


asyncio.run(main())
```

```ruby label="Ruby"
require "getstream/vision_agents"

agent = GetStream::VisionAgents::Client.new.agent("my-agent")

agent.tools.register("weather_lookup",
                     description: "Look up the current weather for a location",
                     params: { location: "a city or region" }) do |location:|
  place = location.downcase
  if place.include?("colorado") then "sunny"
  elsif place.include?("amsterdam") then "rainy"
  elsif place.include?("san francisco") then "foggy"
  else "no forecast for #{location}"
  end
end

dispatch = GetStream::VisionAgents::Dispatch.new
dispatch.host(agent)
dispatch.run
```

```php label="PHP"
<?php

declare(strict_types=1);

use GetStream\VisionAgents\Client;
use GetStream\VisionAgents\Describe;
use GetStream\VisionAgents\Worker\Dispatch;

require __DIR__ . '/vendor/autoload.php';

$agent = (new Client())->agent('my-agent');

$agent->tools->register(
    'weather_lookup',
    'Look up the current weather for a location',
    function (#[Describe('a city or region')] string $location): string {
        $place = strtolower($location);
        return match (true) {
            str_contains($place, 'colorado') => 'sunny',
            str_contains($place, 'amsterdam') => 'rainy',
            str_contains($place, 'san francisco') => 'foggy',
            default => "no forecast for {$location}",
        };
    },
);

(new Dispatch())->host($agent)->run();
```

```go label="Go"
package main

import (
	"context"
	"log"
	"strings"

	"github.com/GetStream/Vision-Agents/sdks/go/agents"
	"github.com/GetStream/Vision-Agents/sdks/go/client"
	"github.com/GetStream/Vision-Agents/sdks/go/stream"
	"github.com/GetStream/Vision-Agents/sdks/go/tools"
)

type WeatherLookup struct {
	Location string `json:"location" schema:"a city or region"`
}

func weatherLookup(_ context.Context, in WeatherLookup) (any, error) {
	place := strings.ToLower(in.Location)
	switch {
	case strings.Contains(place, "colorado"):
		return "sunny", nil
	case strings.Contains(place, "amsterdam"):
		return "rainy", nil
	case strings.Contains(place, "san francisco"):
		return "foggy", nil
	}
	return "no forecast for " + in.Location, nil
}

func main() {
	api, err := client.New(stream.Backend{})
	if err != nil {
		log.Fatal(err)
	}
	agent := api.Agent("my-agent")
	if err := tools.Register(agent.Tools(), "weather_lookup", "Look up the current weather for a location", weatherLookup); err != nil {
		log.Fatal(err)
	}

	dispatch, err := agents.NewDispatch(agents.DispatchOptions{})
	if err != nil {
		log.Fatal(err)
	}
	dispatch.Host(agent, 0)
	log.Fatal(dispatch.Run(context.Background()))
}
```

```csharp label="C#"
using GetStream.VisionAgents;

using var client = new VisionAgentsClient();
var agent = client.Agent("my-agent");

agent.Tools.Register<WeatherLookup, string>("weather_lookup",
    "Look up the current weather for a location",
    (lookup, _) => Task.FromResult(Forecast(lookup.Location)));

await new Dispatch(new DispatchOptions { Client = client }).Host(agent).RunAsync();

static string Forecast(string location) => location.ToLowerInvariant() switch
{
    var place when place.Contains("colorado") => "sunny",
    var place when place.Contains("amsterdam") => "rainy",
    var place when place.Contains("san francisco") => "foggy",
    _ => $"no forecast for {location}",
};

record WeatherLookup(string Location);
```

```rust label="Rust"
use serde_json::json;
use vision_agents::{Client, Dispatch};

#[tokio::main]
async fn main() -> vision_agents::Result<()> {
    let client = Client::from_env()?;
    let agent = client.agent("my-agent");

    agent.tools.register(
        "weather_lookup",
        "Look up the current weather for a location",
        json!({
            "type": "object",
            "properties": { "location": { "type": "string", "description": "a city or region" } },
            "required": ["location"],
        }),
        |args| async move {
            let place = args["location"].as_str().unwrap_or_default().to_lowercase();
            let weather = if place.contains("colorado") {
                "sunny"
            } else if place.contains("amsterdam") {
                "rainy"
            } else if place.contains("san francisco") {
                "foggy"
            } else {
                "no forecast for that location"
            };
            Ok::<_, String>(weather)
        },
    );

    let dispatch = Dispatch::new(client);
    dispatch.host(&agent, None);
    dispatch.run().await
}
```

Leave the worker running. The template's guardrail only answers questions about Acme, so let the weather through in `agents/my-agent/guardrail.md` and sync again:

```text
Answer questions about Acme Corp: its products, plans, pricing, accounts and support,
and questions about the weather.
Greetings, thanks, goodbyes and questions about who the agent is or what it can help with
are permitted too.
```

Now ask the agent about the weather in Amsterdam, Colorado or San Francisco. The model calls `weather_lookup`, the worker answers, and the agent replies with it:

```sh
getstream agents test my-agent --sync
```

Approval, interruptions, progress updates and the rest are in [Tool calling & Dispatch](https://getstream.io/agents/docs/ios/tool-calling-and-dispatch/).
For a runnable Go agent with tools, knowledge and memory together, see the [caregiving example](https://github.com/GetStream/Vision-Agents/tree/accelerate/sdks/go/examples/caregiving).

## Agent UI

Next you typically want to build your agent UI. On either React, Android or iOS.

1. Install the SDK

```bash label="Browser"
npm install @stream-io/calypso stream-chat stream-chat-react @stream-io/video-react-sdk
```

```kotlin label="Android"
// settings.gradle.kts: the SDK is a snapshot from Stream's snapshot repository,
// and the Stream Chat it uses comes from Stream's Maven repository
dependencyResolutionManagement {
    repositories {
        google()
        mavenCentral()
        maven("https://stream-io-repo.com")
        maven("https://snapshots.stream-io-repo.com")
    }
}

// build.gradle.kts: needs compileSdk 37 and Kotlin 2.3 or newer
implementation("io.getstream:calypso-core:0.1.0-202611041811-SNAPSHOT")
implementation("io.getstream:stream-chat-android-ai-compose:7.18.0")
implementation("androidx.lifecycle:lifecycle-runtime-compose:<version>")
```

```swift label="iOS"
// Package.swift: needs iOS 16 and Xcode 27 or newer
platforms: [.iOS(.v16)],
dependencies: [
    .package(url: "https://github.com/GetStream/Vision-Agents.git", branch: "accelerate"),
    .package(url: "https://github.com/GetStream/stream-chat-swift.git", from: "5.15.0"),
],
targets: [
    .target(
        name: "MyApp",
        dependencies: [
            .product(name: "VisionAgentsCore", package: "Vision-Agents"),
            .product(name: "StreamChatAI", package: "stream-chat-swift"),
        ]
    ),
]
```

2. Mint a token

Your users sign in with a Stream user token.
Mint one for a test user with the CLI.

```sh
getstream token jlahey --ttl 2h
```

Note: In production you would generate the token in your backend when a user signs up or logs in.

3. Create a session

Open a session with your agent and ask it something:

```ts label="Browser"
import { Client, type Session } from "@stream-io/calypso";

// Stream's hosted router. Self-hosting covers pointing it at your own.
export const api = new Client({ apiKey: "your_api_key" });

/** Call once, when the user signs in. */
export async function signIn() {
  await api.setUser("jlahey", "paste_the_token_here");
}

/** Opens a session with the agent and asks it something. */
export async function startSession(): Promise<Session> {
  const session = await api.agent("my-agent").sessions.create();
  await session.responses.create("How do I get started with Acme?");
  return session;
}
```

```kotlin label="Android"
val agents = VisionAgents(apiKey = "your_api_key")
agents.setUser(User(id = "jlahey"), token = "paste_the_token_here")

val session = agents.agent("my-agent").chat()
session.responses.create("How do I get started with Acme?")
```

```swift label="iOS"
import VisionAgentsCore

let agents = VisionAgents(apiKey: "your_api_key")

/// Call once, when the user signs in.
func signIn() {
    agents.setUser(User(id: "jlahey"), token: "paste_the_token_here")
}

/// Opens a session with the agent and asks it something.
@MainActor
func startSession() async throws -> AgentSession {
    let session = try await agents.agent("my-agent").sessions.create()
    // The session's socket, which stops a reply and runs the tools this phone offers.
    await session.start()
    _ = try await session.responses.create("How do I get started with Acme?")
    return session
}
```

4. Add the chat UI

The chat UI components support thinking, tool calling, markdown, table responses, code, and [UI the agent writes](https://getstream.io/agents/docs/ios/generated-ui/) in A2UI or OpenUI.
The session's channel is its transcript: the agent does not answer a message sent to the channel, so the composer sends with `session.responses.create`, and the router writes your message into the channel itself.
This holds for voice sessions too. Each thing the caller says is written as a message, the agent's reply streams into its message while it is generated and settles once it is spoken, and a reply the caller cut off is marked interrupted, with its text usually cleared. So render the channel for the transcript and live captions; don't build a second transcript from `session.events()`.
Have a look how you can show the UI:

**Browser**

In JavaScript, import `@stream-io/calypso/chat` once, or `session.chat()` throws (`@stream-io/calypso/video` does the same for `session.video()`).

```tsx
import "@stream-io/calypso/chat";
import type { Session } from "@stream-io/calypso";
import { useMemo } from "react";
import type { Channel as ChatChannel, StreamChat } from "stream-chat";
import {
  AIStateIndicator,
  Channel,
  Chat,
  MessageComposer,
  MessageList,
  WithComponents,
} from "stream-chat-react";
import "stream-chat-react/css/index.css";

// The agent's replies carry `ai_generated: true`.
declare module "stream-chat" {
  interface CustomMessageData {
    ai_generated?: boolean;
  }
}

/** Opens the session's channel, as the chat's views show it. */
export async function openChat(session: Session) {
  // `session.chat()` connects to Stream Chat with the user set above.
  const { client, channel } = await session.chat();
  return {
    session,
    client: client as StreamChat,
    channel: channel as ChatChannel,
  };
}

export type AgentChat = Awaited<ReturnType<typeof openChat>>;

export const AgentConversation = ({ chat }: { chat: AgentChat }) => {
  // While the agent answers, the composer shows a Stop button. Its default sends
  // `ai_indicator.stop` to the channel, which the agent does not act on, so stop
  // the reply through the session instead.
  const StopButton = useMemo(
    () => () => (
      <button
        aria-label="Stop"
        className="str-chat__stop-ai-generation-button"
        onClick={() => chat.session.interrupt()}
        type="button"
      />
    ),
    [chat.session],
  );

  return (
    <Chat
      client={chat.client}
      // Renders the agent's replies as streaming markdown.
      isMessageAIGenerated={(message) => Boolean(message.ai_generated)}
    >
      <WithComponents overrides={{ StopAIGenerationButton: StopButton }}>
        <Channel channel={chat.channel}>
          <MessageList />
          {/* Thinking and generating states, for text and voice sessions. */}
          <AIStateIndicator />
          <MessageComposer
            overrideSubmitHandler={async ({ message }) => {
              await chat.session.responses.create(message.text ?? "");
            }}
          />
        </Channel>
      </WithComponents>
    </Chat>
  );
};
```

In React, `isMessageAIGenerated` makes `MessageList` render the agent's replies with streaming markdown, and `AIStateIndicator` shows while the agent thinks or writes. The [React AI integration guide](https://getstream.io/chat/docs/sdk/react/v14/guides/ai-integrations/sdk-integration/) covers both. For a custom message, use the [AI components](https://getstream.io/chat/docs/sdk/react/v14/guides/ai-integrations/) such as `StreamingMessage` and `AIMarkdown`. `AIConversation`, among them, lays the conversation out by turn with each reply's reasoning, tool calls, approvals and sign-ins. The [React assistant](https://github.com/GetStream/Vision-Agents/tree/accelerate/examples/voice_agents/react_assistant) sample builds a complete app on them, with a list of conversations, photos, voice mode, connector sign-ins and tools that run in the browser. Step 5 adds the call audio and a listening, thinking or speaking status.

**Android**

```kotlin
import android.content.Context
import androidx.activity.compose.setContent
import androidx.compose.foundation.layout.Column
import androidx.compose.foundation.layout.fillMaxSize
import androidx.compose.runtime.Composable
import androidx.compose.runtime.getValue
import androidx.compose.ui.Modifier
import androidx.lifecycle.compose.collectAsStateWithLifecycle
import androidx.lifecycle.lifecycleScope
import io.getstream.chat.android.ai.compose.conversation.AIMessage
import io.getstream.chat.android.ai.compose.conversation.AIMessageRole
import io.getstream.chat.android.ai.compose.parts.AIClientIdentity
import io.getstream.chat.android.ai.compose.parts.AIMessagePart
import io.getstream.chat.android.ai.compose.ui.component.AIConversation
import io.getstream.chat.android.ai.compose.ui.component.AIToolApprover
import io.getstream.chat.android.ai.compose.ui.component.ChatComposer
import io.getstream.chat.android.client.channel.state.ChannelState
import io.getstream.chat.android.models.Message
import io.getstream.visionagents.android.core.AgentSession
import io.getstream.visionagents.android.core.AgentsException
import io.getstream.visionagents.android.core.Conversation
import io.getstream.visionagents.android.core.channelState
import kotlinx.coroutines.CoroutineScope
import kotlinx.coroutines.ExperimentalCoroutinesApi
import kotlinx.coroutines.flow.SharingStarted
import kotlinx.coroutines.flow.StateFlow
import kotlinx.coroutines.flow.combine
import kotlinx.coroutines.flow.flatMapLatest
import kotlinx.coroutines.flow.flowOf
import kotlinx.coroutines.flow.map
import kotlinx.coroutines.flow.stateIn
import kotlinx.coroutines.launch

// In your activity, with the session from step 3.
lifecycleScope.launch {
    val chat = AgentChat.open(this@MainActivity, session, lifecycleScope)
    setContent { AgentConversation(chat, Modifier.fillMaxSize()) }
}

@Composable
fun AgentConversation(chat: AgentChat, modifier: Modifier = Modifier) {
    val messages by chat.messages.collectAsStateWithLifecycle()
    val answering by chat.isAnswering.collectAsStateWithLifecycle()

    Column(modifier) {
        // Each question moves to the top, and its reply streams in below it with its
        // reasoning and tool calls.
        AIConversation(
            messages = messages,
            modifier = Modifier.weight(1f),
            // The agent is on it, and its reply isn't in the channel yet.
            isThinking = answering && messages.lastOrNull()?.role != AIMessageRole.Assistant,
            // Asks the user before a tool that asks first runs.
            approver = chat.approver,
        )
        ChatComposer(
            // The agent does not answer a message sent to the channel: ask the session.
            onSendClick = { chat.send(it.text) },
            onStopClick = chat::stop,
            // While the agent answers, the composer shows a Stop button.
            isGenerating = answering,
        )
    }
}

/** The session's channel, as the AI components show it. */
@OptIn(ExperimentalCoroutinesApi::class)
class AgentChat private constructor(
    val session: AgentSession,
    val approver: AIToolApprover,
    channel: StateFlow<ChannelState?>,
    private val scope: CoroutineScope,
) {
    /** The channel's messages, oldest first, as Stream Chat keeps them up to date. */
    val messages: StateFlow<List<AgentMessage>> = channel
        .flatMapLatest { it?.messages ?: flowOf(emptyList()) }
        .map { messages -> messages.filter { it.deletedAt == null }.map(::AgentMessage) }
        .stateIn(scope, SharingStarted.Eagerly, emptyList())

    /** Whether a reply is on its way, from the first thought to the last word. */
    val isAnswering: StateFlow<Boolean> = combine(session.conversation, messages) { conversation, messages ->
        when (conversation.state) {
            Conversation.State.Responding, is Conversation.State.Working -> true
            else -> messages.lastOrNull()?.isGenerating == true
        }
    }.stateIn(scope, SharingStarted.Eagerly, false)

    fun send(text: String) {
        scope.launch {
            try {
                session.responses.create(text)
            } catch (_: AgentsException) {
                // Not sent: the user can send it again.
            }
        }
    }

    /** Stops the reply in flight. What was written so far is kept. */
    fun stop() {
        scope.launch {
            try {
                session.interrupt()
            } catch (_: AgentsException) {
                // The reply finished first.
            }
        }
    }

    companion object {
        /** Opens the session's channel. Pass a [scope] on the main thread, such as `lifecycleScope`. */
        suspend fun open(context: Context, session: AgentSession, scope: CoroutineScope): AgentChat {
            // The user set above, who answers the calls the agent addresses to them.
            val approver = AIToolApprover("jlahey", AIClientIdentity.installId(context), scope) { call, allowed ->
                session.decide(call.id, allowed)
            }
            // Connects to Stream Chat as the user set above and watches the session's channel.
            return AgentChat(session, approver, session.channelState(context, scope), scope)
        }
    }
}

/** A message of the channel, as the AI components show it. */
data class AgentMessage(val message: Message) : AIMessage {
    override val id: String = message.id
    // The agent's replies carry `ai_generated: true`.
    override val role: AIMessageRole =
        if (message.extraData["ai_generated"] == true) AIMessageRole.Assistant else AIMessageRole.User
    override val text: String = message.text
    override val isGenerating: Boolean = message.extraData["generating"] == true
    // The reply's reasoning and tool calls, which the router writes as its attachments. Stream Chat
    // lifts `name` out of an attachment's fields, and a tool call's step needs it back.
    override val parts: List<AIMessagePart> = AIMessagePart.parts(
        message.attachments.map { it.type.orEmpty() to it.extraData + ("name" to it.name) },
    )
}
```

On Android, the views are the [AI components](https://getstream.io/chat/docs/sdk/android/guides/ai-integrations/) of `stream-chat-android-ai-compose`, and the agents SDK brings no views of its own. Stream Chat's client, which `channelState` watches the channel with, comes with the agents SDK. `AIConversation` shows each reply's reasoning and tool calls above its streaming markdown answer, and shows the agent thinking until the reply has something to show. `AIToolApprover` asks the user before a tool that asks first runs, and answers with `session.decide(call.id, allowed)`. It keeps the user's answers, so in an app, open the chat in a `ViewModel` and pass its `viewModelScope`. `ChatComposer` asks for the microphone the first time the user taps dictation; `RECORD_AUDIO` is already in your manifest, from Stream Video, which comes with the agents SDK. Its photo picker needs no permission, but `send` here passes only the text: `responses.create` takes photos as `images`. The [Android assistant](https://github.com/GetStream/Vision-Agents/tree/accelerate/examples/voice_agents/android_assistant) sample builds a complete app on them, with a list of conversations, photos, voice mode, connector sign-ins and tools that run on the phone.

**iOS**

```swift
import StreamChat
import StreamChatAI
import SwiftUI
import VisionAgentsCore

/// Opens the session's channel, as the AI components show it.
@MainActor
func openChat(for session: AgentSession) async throws -> AgentChat {
    // `session.chat()` connects to Stream Chat as the user set above and opens the session's channel.
    let channel = try await session.chat()
    return AgentChat(session: session, channel: channel)
}

struct AgentConversation: View {
    @ObservedObject var chat: AgentChat
    @StateObject private var composer = AIComposerViewModel()

    var body: some View {
        VStack(spacing: 0) {
            // Each question moves to the top, and its reply streams in below it with its
            // reasoning and tool calls.
            AIConversationView(messages: chat.messages)
            AIComposerView(
                viewModel: composer,
                // The agent does not answer a message sent to the channel: ask the session.
                onMessageSend: { message in
                    Task { _ = try? await chat.session.responses.create(message.text) }
                },
                onStopGenerating: {
                    Task { try? await chat.session.interrupt() }
                }
            )
        }
        // While the agent writes, the composer shows a Stop button.
        .onReceive(chat.$isAnswering) { composer.isGenerating = $0 }
    }
}

/// The session's channel, as the AI components show it.
@MainActor
final class AgentChat: ObservableObject, ChatChannelControllerDelegate {
    let session: AgentSession
    @Published private(set) var messages: [AgentMessage] = []
    @Published private(set) var isAnswering = false

    private let channel: ChatChannelController

    init(session: AgentSession, channel: ChatChannelController) {
        self.session = session
        self.channel = channel
        channel.delegate = self
        channel.synchronize()
    }

    func channelController(
        _ channelController: ChatChannelController,
        didUpdateMessages changes: [ListChange<ChatMessage>]
    ) {
        // The channel lists its messages newest first, the conversation oldest first.
        messages = channelController.messages.reversed().map(AgentMessage.init)
        isAnswering = messages.last?.isGenerating == true
    }
}

/// A message of the channel, as the AI components show it.
struct AgentMessage: AIMessage {
    let message: ChatMessage

    var id: MessageId { message.id }
    // The agent's replies carry `ai_generated: true`.
    var role: AIMessageRole {
        message.extraData["ai_generated"]?.boolValue == true ? .assistant : .user
    }
    var text: String { message.text }
    var isGenerating: Bool { message.extraData["generating"]?.boolValue == true }
    // The reply's reasoning and tool calls, which the router writes as its attachments.
    var parts: [AIMessagePart] {
        AIMessagePart.parts(from: message.allAttachments.map { ($0.type.rawValue, $0.payload) })
    }
}
```

On iOS, the views are the [AI components](https://getstream.io/chat/docs/sdk/ios/guides/ai-integrations/) of StreamChatAI, and VisionAgentsCore brings no views of its own. StreamChat, which the snippet imports for the channel, comes with VisionAgentsCore. `AIComposerView` needs iOS 16. `AIConversationView` shows the reply's [reasoning and tool calls](https://getstream.io/chat/docs/sdk/ios/guides/ai-integrations/reasoning-and-tool-calls/) above its streaming markdown answer, and shows the agent thinking until the reply has something to show. To let the user answer a tool that asks first, pass an `AIToolApprover` as its `approver` and answer with `session.decide(call.id, allowed: allowed)`. The composer's photo picker and dictation need the camera, photo library, microphone and speech recognition [usage descriptions](https://getstream.io/chat/docs/sdk/ios/guides/ai-integrations/#usage-descriptions) in your `Info.plist`. The [Swift assistant](https://github.com/GetStream/Vision-Agents/tree/accelerate/examples/voice_agents/swift_assistant) sample builds a complete app on them, with a list of conversations, photos, voice mode, connector sign-ins and tools that run on the phone.

5. Add the voice UI

A voice session puts the agent on a call. Your app joins that call, and the agent hears the user and answers out loud.
What is said on the call is written into the session's channel, so the chat's views show the transcript here too.
Add a glow that shows whether the agent is listening, thinking or speaking, and buttons to mute and hang up:

**Browser**

```tsx
import { VoiceSession } from "@stream-io/calypso/video";
import {
  ParticipantsAudio,
  StreamCall,
  StreamVideo,
  useCallStateHooks,
} from "@stream-io/video-react-sdk";
import { useEffect, useSyncExternalStore } from "react";
import { Channel, Chat, MessageList } from "stream-chat-react";
import { VoiceGlow } from "stream-chat-react/ai-components";

/** Starts a session with voice on, and opens its transcript. Call it from a click. */
export async function startVoice() {
  // The agent joins the session's call right away.
  const voice = await VoiceSession.start(api.agent("my-agent"));
  // What is said on the call is written into the session's channel, so the chat's views show it.
  return { voice, chat: await openChat(voice.session) };
}

export const AgentVoice = ({
  voice,
  chat,
}: {
  voice: VoiceSession;
  chat: AgentChat;
}) => {
  // What the agent is doing, in the states the glow takes.
  const { call, client, muted, status } = useSyncExternalStore(
    voice.subscribe,
    voice.get,
  );

  // Joins the call from this device, with the microphone on. Run twice, it joins once.
  useEffect(() => {
    voice
      .join()
      .catch((error: unknown) =>
        console.error("Could not join the call", error),
      );
  }, [voice]);

  return (
    <div>
      <VoiceGlow state={status} style={{ width: 240, height: 240 }} />
      {call && client && (
        <StreamVideo client={client}>
          <StreamCall call={call}>
            <AgentAudio />
          </StreamCall>
        </StreamVideo>
      )}
      {/* What the user said and the agent's replies, as in the chat. */}
      <Chat
        client={chat.client}
        isMessageAIGenerated={(message) => Boolean(message.ai_generated)}
      >
        <Channel channel={chat.channel}>
          <MessageList />
        </Channel>
      </Chat>
      <button onClick={() => void voice.setMuted(!muted)}>
        {muted ? "Unmute" : "Mute"}
      </button>
      {/* Leaves the call and ends the session this page started. */}
      <button onClick={() => void voice.leave()}>Hang up</button>
    </div>
  );
};

/** Plays the agent. */
const AgentAudio = () => {
  const { useRemoteParticipants } = useCallStateHooks();
  return <ParticipantsAudio participants={useRemoteParticipants()} />;
};
```

`VoiceSession` joins the call with Stream Video, signed in as the user you passed to `setUser`. `join()` rejects if joining fails, for example because the user denied microphone access. Call `startVoice()` from a click: a browser asks for the microphone and plays audio only after one.
`voice.session` is a regular `Session`, so `openChat` from step 4 works here too, and the agent's replies stream into the transcript as they're generated.
`status` is worked out for you: speaking comes from the call, and the agent is thinking until its reply is heard, since a reply is written before it is spoken. `VoiceSession` is in `@stream-io/calypso` from version 0.2.3. Import it from `@stream-io/calypso/video`, which brings in `@stream-io/video-client` with the Video React SDK from step 1, so don't install that separately.

`VoiceGlow` is one of the React SDK's [AI components](https://getstream.io/chat/docs/sdk/react/v14/guides/ai-integrations/), in `stream-chat-react/ai-components` from version 14.14. For a waveform instead of a glow, use `VoiceWave`, which takes the same states.
To show the agent what the camera sees, join with `voice.join({ camera: true })` or call `voice.setCameraEnabled(true)`. The camera starts on the back lens where there is one, and the agent looks through it when asked, if its config has the `vision` skill and a vision subagent.

An agent made from the text template talks with the router's default speech-to-text and text-to-speech models. Make it with `--template voice` to choose those models and add a greeting.
To switch an open chat to voice, call `session.voice.start()`, then join with `VoiceSession.of(session)`. Leaving then leaves the session running; `voice.end()` ends it. The voice mode of the [React assistant](https://github.com/GetStream/Vision-Agents/tree/accelerate/examples/voice_agents/react_assistant) sample does this.

**Android**

```kotlin
import io.getstream.chat.android.ai.compose.ui.component.AIVoiceAgentState
import io.getstream.chat.android.ai.compose.ui.component.AIVoiceGlow
import io.getstream.visionagents.android.core.Conversation
import io.getstream.visionagents.android.core.SessionOptions
import io.getstream.visionagents.android.core.VoiceSession

/** Starts a session with voice on. Ask for RECORD_AUDIO before you call it. */
suspend fun startVoice(context: Context): VoiceSession =
    // The agent joins the session's call right away.
    VoiceSession.start(context, agents, SessionOptions(agent = "my-agent"))

@Composable
fun AgentVoice(voice: VoiceSession) {
    val scope = rememberCoroutineScope()
    val call by voice.call.collectAsStateWithLifecycle()
    val muted by voice.isMuted.collectAsStateWithLifecycle()
    val conversation by voice.session.conversation.collectAsStateWithLifecycle()
    // Who on the call is speaking: the agent, or the person holding the phone.
    val speakers by remember(call) { call?.state?.activeSpeakers ?: MutableStateFlow(emptyList()) }
        .collectAsStateWithLifecycle()

    // Joins the call from this device, with the microphone on and the agent on the speaker.
    LaunchedEffect(voice) { voice.join() }

    // Speaking comes from the call: a reply is written before it is spoken.
    val state = when {
        call == null -> AIVoiceAgentState.Connecting
        speakers.any { !it.isLocal } -> AIVoiceAgentState.Speaking
        speakers.any { it.isLocal } -> AIVoiceAgentState.Listening
        else -> when (conversation.state) {
            Conversation.State.Listening -> AIVoiceAgentState.Listening
            Conversation.State.Responding, is Conversation.State.Working -> AIVoiceAgentState.Thinking
            else -> AIVoiceAgentState.Idle
        }
    }

    Column(horizontalAlignment = Alignment.CenterHorizontally, verticalArrangement = Arrangement.spacedBy(16.dp)) {
        AIVoiceGlow(state = state, modifier = Modifier.size(240.dp))
        // What the user said and the agent's replies.
        LazyColumn(Modifier.weight(1f)) {
            items(conversation.turns) { turn -> Text(turn.text) }
        }
        Row(horizontalArrangement = Arrangement.spacedBy(24.dp)) {
            Button(onClick = { voice.setMuted(!muted) }) { Text(if (muted) "Unmute" else "Mute") }
            // Leaves the call and ends the session this device started.
            Button(onClick = { scope.launch { voice.leave() } }) { Text("Hang up") }
        }
    }
}
```

`VoiceSession` joins the call with Stream Video, signed in as the user you passed to `setUser`. `join()` doesn't throw. If joining fails, for example because the app doesn't hold `RECORD_AUDIO`, the error is in `voice.failure`, so ask for the microphone before `startVoice`.
`voice.session` is a regular `AgentSession`, so `AgentChat.open` from step 4 works here too. `conversation` holds the turns, each reply growing as it is generated.
You don't need to add Stream Video to your app: it comes with the agents SDK.

`AIVoiceGlow` is one of the [AI components](https://getstream.io/chat/docs/sdk/android/guides/ai-integrations/) of `stream-chat-android-ai-compose`, from step 1. For a waveform instead of a glow, use `AIVoiceWave`, which takes the same states.

An agent made from the text template talks with the router's default speech-to-text and text-to-speech models. Make it with `--template voice` to choose those models and add a greeting.
To switch an open chat to voice, call `session.startVoice()`, then join with `VoiceSession.attach(context, agents, session.id)`. `voice.end()` ends the session whoever started it. The voice mode of the [Android assistant](https://github.com/GetStream/Vision-Agents/tree/accelerate/examples/voice_agents/android_assistant) sample does this.

**iOS**

```swift
import StreamChatAI
import StreamVideo
import SwiftUI
import VisionAgentsCore

/// Starts a session with voice on, and opens its transcript.
@MainActor
func startVoice() async throws -> (voice: VoiceSession, chat: AgentChat) {
    // The agent joins the session's call right away.
    let voice = try await VoiceSession.start(agents: agents, agent: "my-agent")
    // The session's socket, which follows what the agent is doing and runs the tools this phone offers.
    await voice.session.start()
    // What is said on the call is written into the session's channel, so the chat's views show it.
    return (voice, try await openChat(for: voice.session))
}

struct AgentVoice: View {
    @ObservedObject var voice: VoiceSession
    @ObservedObject var chat: AgentChat

    var body: some View {
        VStack(spacing: 16) {
            if let call = voice.call {
                AgentStatus(voice: voice, call: call.state)
            } else {
                AIVoiceGlowView(state: .connecting)
                    .frame(width: 240, height: 240)
            }
            // What the user said and the agent's replies, as in the chat.
            AIConversationView(messages: chat.messages)
            HStack(spacing: 24) {
                Button {
                    Task { await voice.setMuted(!voice.isMuted) }
                } label: {
                    Image(systemName: voice.isMuted ? "mic.slash.fill" : "mic.fill")
                }
                // Leaves the call and ends the session this device started.
                Button(role: .destructive) {
                    Task { await voice.leave() }
                } label: {
                    Image(systemName: "phone.down.fill")
                }
            }
            .font(.title2)
            .buttonStyle(.bordered)
        }
        // Joins the call from this device, with the microphone on and the agent on the speaker.
        .task { await voice.join() }
    }
}

/// What the agent is doing, as the glow shows it.
struct AgentStatus: View {
    @ObservedObject var voice: VoiceSession
    // The call's participants, and which of them is speaking.
    @ObservedObject var call: CallState

    var body: some View {
        AIVoiceGlowView(state: state)
            .frame(width: 240, height: 240)
    }

    private var state: AIVoiceAgentState {
        // Speaking comes from the call: a reply is written before it is spoken,
        // and is done before the agent stops talking.
        if call.remoteParticipants.contains(where: \.isSpeaking) { return .speaking }
        if call.localParticipant?.isSpeaking == true { return .listening }
        switch voice.session.state {
        case .listening: return .listening
        case .responding, .working: return .thinking
        case .idle, .ended: return .idle
        }
    }
}
```

`VoiceSession` joins the call with Stream Video, signed in as the user you passed to `setUser`. `join()` doesn't throw. If joining fails, for example because the user denied microphone access, the error is in `voice.failure`.
`voice.session` is a regular `AgentSession`, so `openChat(for:)` from step 4 works here too, and the agent's replies stream into the transcript as they're generated.
You don't need to add StreamVideo to your package: it comes with VisionAgentsCore.

`AIVoiceGlowView` is one of StreamChatAI's [AI components](https://getstream.io/chat/docs/sdk/ios/guides/ai-integrations/). For a waveform instead of a glow, use `AIVoiceWaveView`, which takes the same states.
Add a microphone [usage description](https://getstream.io/chat/docs/sdk/ios/guides/ai-integrations/#usage-descriptions) to your `Info.plist`. Without it, iOS closes the app when it asks for the microphone.

An agent made from the text template talks with the router's default speech-to-text and text-to-speech models. Make it with `--template voice` to choose those models and add a greeting.
To switch an open chat to voice, call `session.startVoice()`, then join with `VoiceSession.attach(agents:sessionID:)`. The voice mode of the [Swift assistant](https://github.com/GetStream/Vision-Agents/tree/accelerate/examples/voice_agents/swift_assistant) sample does this.

---

For the most recent version of this documentation, visit [https://getstream.io/agents/docs/ios/](https://getstream.io/agents/docs/ios/).