Generative UI

The model writes UI when the agent's instructions ask it to: add the format's own instructions, with the components of your app it may place, to the agent's instructions.md. stream-chat-react writes those instructions from the same components your app shows. The capabilities() of the app's AIUIHost hold, for each format it reads, the fence the model writes in and the instructions for the model:

// Each format the host reads, such as A2UI and OpenUI, with its `fence` and `instructions`.
const { renderers } = await ui.capabilities();
console.log(JSON.stringify(renderers, null, 2));

The page can't change the agent, so print them while you build the app and add them to instructions.md, or send them to your server and have it add them. Write them again when your components change. You can also write them on your server or at build time, from the same component definitions:

import { a2uiInstructions } from "stream-chat-react/a2ui";
import { openUIInstructions } from "stream-chat-react/openui";

const a2ui = await a2uiInstructions({ components: [placeCard] });
const openui = await openUIInstructions([placeCard]);

See Tell the agent what the app shows in the React guide.

The model writes UI when the agent's instructions ask it to: add the format's own instructions, with the components of your app it may place, to the agent's instructions.md. StreamChatAI writes those instructions from the same components your app shows. The capabilities of the app's AIUIHost hold, for each format it reads, the fence the model writes in and the instructions for the model: what you gave the renderer as its instructions, then the format's own, written from its components, yours and the host's tools:

// The instructions of each format the host reads, such as A2UI's and OpenUI's.
let instructions = (ui.capabilities["renderers"]?.arrayValue ?? [])
    .compactMap { $0.dictionaryValue?["instructions"]?.stringValue }

The app can't change the agent, so print them while you build the app and add them to instructions.md, or send them to your server and have it add them. Write them again when your components change. A renderer's instructions parameter is where to say which answers to write as UI and when to use your components:

A2UIRenderer(instructions: "Write a trip plan as UI, with a place_card for each place. Answer everything else in Markdown.")

A2UI's capabilities also hold the catalog_id of the catalog with your components, getstream.io:app-catalog by default (the renderer's catalogId changes it). See Tell the agent what the app shows in the iOS guide.

The model writes UI when the agent's instructions ask it to: add the format's own instructions, with the components of your app it may place, to the agent's instructions.md. stream-chat-android-ai-compose writes those instructions from the same components your app shows. The capabilities of the app's AIUIHost hold, as plain Kotlin values, for each format it reads, the fence the model writes in and the instructions for the model: what you gave the renderer as its instructions, then the format's own, written from its components, yours and the host's tools:

// The instructions of each format the host reads, such as A2UI's and OpenUI's.
val instructions: List<String> = (ui.capabilities["renderers"] as? List<*>).orEmpty()
    .mapNotNull { renderer -> (renderer as? Map<*, *>)?.get("instructions") as? String }

The app can't change the agent, so print them while you build the app and add them to instructions.md, or send them to your server and have it add them. Write them again when your components change. A renderer's instructions parameter is where to say which answers to write as UI and when to use your components:

val renderer: A2UIRenderer = A2UIRenderer(
    instructions = "Write a trip plan as UI, with a place_card for each place. Answer everything else in Markdown.",
)

A2UI's capabilities also hold the catalog_id of the catalog with your components, getstream.io:app-catalog by default (the renderer's catalogId changes it). See Tell the agent what the app shows in the Android guide.

AIConversation from stream-chat-react shows a reply's UI in place of its text, brings it in as the agent writes it, and sends what the person does in it back to the agent. Its renderers of A2UI and OpenUI are in stream-chat-react/a2ui and stream-chat-react/openui, from version 14.16.0. Each is built on its format's official packages, which your app installs for the formats your agent writes, at the versions the React guide's Installation names.

Keep an AIUIHost with the renderers of the formats your agent writes and your components for as long as the conversation shows, so each reply's UI keeps its state, such as what the person typed into a form. Its send asks the session, with what the person did as uiAction:

import type { Session } from "@stream-io/stream-agents";
import { useMemo } from "react";
import type { LocalMessage } from "stream-chat";
import {
  AIConversation,
  AIUIHost,
  aiUIActionJSON,
  fromStreamMessage,
} from "stream-chat-react/ai-components";
import { createA2UIRenderer } from "stream-chat-react/a2ui";
import { createOpenUIRenderer } from "stream-chat-react/openui";
import "stream-chat-react/css/index.css";
import "stream-chat-react/css/ai-components.css";
import "@openuidev/react-ui/components.css";

import { placeCard } from "./placeCard";

// `messages` are the messages of the channel session.chat() opened.
export const AgentConversation = ({
  session,
  messages,
}: {
  session: Session;
  messages: LocalMessage[];
}) => {
  const ui = useMemo(
    () =>
      new AIUIHost({
        components: [placeCard],
        renderers: [createA2UIRenderer(), createOpenUIRenderer()],
        // What the person did, in their words, with the action for the agent.
        send: (action) =>
          session.responses.create(action.label, {
            uiAction: aiUIActionJSON(action),
          }),
      }),
    [session],
  );

  return (
    <AIConversation
      messages={messages.map((message) => fromStreamMessage(message))}
      ui={ui}
    />
  );
};

A reply shows its UI when one of the host's renderers reads its format and version, the first one listed asked first. Otherwise, it shows its text.

A component is a view of your app the agent can place, described once with the parameters the agent fills in and the events it reports. Every renderer places it by its name, so it works whichever format the agent writes:

import {
  AIUIParameterType,
  defineAIUIComponent,
  requiredAIUIParameter,
} from "stream-chat-react/ai-components";

export const placeCard = defineAIUIComponent<{ title: string }>(
  {
    name: "place_card",
    description: "A photo and a short summary of a place.",
    parameters: [
      requiredAIUIParameter(
        "title",
        AIUIParameterType.string,
        "The place's Wikipedia title.",
      ),
    ],
    events: [
      {
        name: "ask_about",
        description: "The person asks about the place.",
        parameters: [],
      },
    ],
  },
  ({ args, context }) => (
    <PlaceCard
      title={args.title}
      onClick={() =>
        context.emit("ask_about", `Tell me more about ${args.title}`)
      }
    />
  ),
);

A component can load what it shows in the browser, such as a place's photo, which then never reaches the agent. The React generative UI guide covers the rest: the renderers' options, theming, running your app's tools from a reply's UI and handling errors. The iOS and Android guides do the same in the apps.

StreamChatAI's AIConversationView shows a reply's UI in place of its text, brings it in as the agent writes it, and sends what the person does in it back to the agent. Its renderers of A2UI and OpenUI are the StreamChatA2UI and StreamChatOpenUI libraries, from version 5.16.0 of the Stream Chat Swift SDK. Add the library of each format your agent writes to your app target, next to StreamChatAI:

The quickstart's chat UI shows the session's channel with AgentTransport and AgentMessage. Give each message the UI the router moved out of its text, as its uiDocument:

struct AgentMessage: AIMessage {
    let message: ChatMessage

    // The id, role, text, isGenerating and parts, as in the quickstart.

    // The reply's UI, from the message's `custom.ai_ui`.
    var uiDocument: AIUIDocument? { AIUIDocument(customData: message.extraData) }
}

Keep an AIUIHost with the renderers of the formats your agent writes and your components in AgentTransport, so each reply's UI keeps its state, such as what the person typed into a form, for as long as the conversation shows. Its send asks the session, as the composer does:

import Combine
import StreamChat
import StreamChatA2UI
import StreamChatAI
import StreamChatOpenUI
import StreamAgents

@MainActor
final class AgentTransport: AIConversationTransport {
    let session: AgentSession
    /// The UI the agent writes for some replies, in A2UI or OpenUI, with the app's own views.
    let ui: AIUIHost<AIUIRendererPair<A2UIRenderer, OpenUIRenderer>>
    private let channel: ChatChannelController
    private var subscription: AnyCancellable?

    init(_ session: AgentSession) async throws {
        self.session = session
        ui = AIUIHost(components: [PlaceCardComponent()]) {
            A2UIRenderer()
            OpenUIRenderer()
        } send: { [session] action in
            // What the person did, in their words, with the action for the agent.
            _ = try await session.responses.create(action.label, uiAction: action.json)
        }
        channel = try await session.chat()
        await session.start()
    }

    // connect(_:), send(_:), stop() and approver as in the quickstart.
}

Then pass the host to the conversation, in the quickstart's AgentConversation:

struct AgentConversation: View {
    let ui: AIUIHost<AIUIRendererPair<A2UIRenderer, OpenUIRenderer>>
    @StateObject private var conversation: AIConversation<AgentMessage>

    init(transport: AgentTransport) {
        ui = transport.ui
        _conversation = StateObject(
            wrappedValue: AIConversation(transport: transport, approver: transport.approver)
        )
    }

    var body: some View {
        AIConversationView(conversation: conversation, ui: ui)
            .safeAreaInset(edge: .bottom) {
                AIComposerView(conversation: conversation)
            }
    }
}

A reply shows its UI when one of the host's renderers reads its format and version, the first one listed asked first. Otherwise, it shows its text. The UI comes in as the agent writes it: each part fades in, rising into place, after the parts before it. A reply loaded from the channel's history shows at once.

OpenUIRenderer needs iOS 17. With an earlier deployment target, keep a host of A2UIRenderer alone, an AIUIHost<A2UIRenderer>, before iOS 17.

A component is a view of your app the agent can place, described once with the parameters the agent fills in and the events it reports. Every renderer places it by its name, so it works whichever format the agent writes:

struct PlaceCardComponent: AIUIComponent {
    // Decoded off the main actor, also in an app whose code is on the main actor by default.
    nonisolated struct Arguments: Decodable {
        let title: String
    }

    let definition = AIUIComponentDefinition(
        name: "place_card",
        description: "A photo and a short summary of a place.",
        parameters: [.required("title", .string, description: "The place's Wikipedia title.")],
        events: [.init(name: "ask_about", description: "The person asks about the place.")]
    )

    func body(arguments: Arguments, context: AIUIComponentContext) -> some View {
        PlaceCard(title: arguments.title) {
            context.emit("ask_about", label: "Tell me more about \(arguments.title)")
        }
    }
}

A component can load what it shows on the device, such as a place's photo or the person's calendar, which then never reaches the agent. Its definition reads and writes JSON, so your apps and the agent's instructions can share one description of it, decoded with JSONDecoder as [AIUIComponentDefinition].

The renderers treat what the agent writes as untrusted: a reply's UI can't crash or freeze the app, and its links open through the host, which opens only web and email links by default. A reply's UI can also run your app's client side tools, passed to the host as tools, such as to load the person's calendar for the days of a plan. They are apart from the tools the agent calls, and the host runs one by itself only when its definition says it's read-only (readOnlyHint).

The iOS generative UI guide covers the rest: the renderers' options and theming, components, a renderer of your own for another format, and showing a reply's UI in your own views with AIUIView.

AIConversation, of the AI components in stream-chat-android-ai-compose, shows a reply's UI in place of its text, brings it in as the agent writes it, and sends what the person does in it back to the agent. Its renderers of A2UI and OpenUI are the stream-chat-android-ai-a2ui and stream-chat-android-ai-openui artifacts, from version 7.18.0 of the Stream Chat Android SDK. Add the artifact of each format your agent writes to your dependencies, next to stream-chat-android-ai-compose:

The quickstart's chat UI shows the session's channel with MessageList, which shows a reply's text. To show its UI, show the channel with AIConversation, from its messages as AIMessages. Give each message the UI the router moved out of its text, as its uiDocument:

/** A message of the session's channel, as the AI components show it. */
class AgentMessage(val message: Message) : AIMessage {
    override val id: String = message.id

    // The agent's replies carry `ai_generated: true`.
    override val role: AIMessageRole =
        if (message.extraData["ai_generated"] == true) AIMessageRole.Assistant else AIMessageRole.User
    override val text: String = message.text
    override val isGenerating: Boolean = message.extraData["generating"] == true

    // The reply's reasoning and tool calls, which the router writes as its attachments.
    // Stream Chat moves an attachment's `name` out of its extraData, and a tool call needs it back.
    override val parts: List<AIMessagePart> = AIMessagePart.parts(
        message.attachments.map { it.type.orEmpty() to it.extraData + ("name" to it.name) },
    )

    // The reply's UI, from the message's `custom.ai_ui`.
    override val uiDocument: AIUIDocument? = AIUIDocument.from(message.extraData)
}

Keep an AIUIHost with the renderers of the formats your agent writes and your components beside the messages, in AgentChat, so each reply's UI keeps its state, such as what the person typed into a form, for as long as the conversation shows. Its send asks the session, as the composer does:

import android.content.Context
import io.getstream.chat.android.ai.a2ui.generativeui.A2UIRenderer
import io.getstream.chat.android.ai.compose.generativeui.AIUIHost
import io.getstream.chat.android.ai.openui.generativeui.OpenUIRenderer
import io.getstream.chat.android.client.channel.state.ChannelState
import io.getstream.agents.android.core.AgentSession
import io.getstream.agents.android.core.AgentsException
import io.getstream.agents.android.core.channelState
import kotlinx.coroutines.CoroutineScope
import kotlinx.coroutines.ExperimentalCoroutinesApi
import kotlinx.coroutines.flow.MutableStateFlow
import kotlinx.coroutines.flow.SharingStarted
import kotlinx.coroutines.flow.StateFlow
import kotlinx.coroutines.flow.asStateFlow
import kotlinx.coroutines.flow.flatMapLatest
import kotlinx.coroutines.flow.flowOf
import kotlinx.coroutines.flow.map
import kotlinx.coroutines.flow.stateIn
import kotlinx.coroutines.launch

/** The session's channel, as the AI components show it. */
@OptIn(ExperimentalCoroutinesApi::class)
class AgentChat private constructor(
    val session: AgentSession,
    channel: StateFlow<ChannelState?>,
    scope: CoroutineScope,
) {
    private val failed = MutableStateFlow<String?>(null)

    /** The channel's messages, oldest first. */
    val messages: StateFlow<List<AgentMessage>> = channel
        .flatMapLatest { it?.messages ?: flowOf(emptyList()) }
        .map { messages -> messages.map(::AgentMessage) }
        .stateIn(scope, SharingStarted.Eagerly, emptyList())

    /** Why asking failed, or null. */
    val failure: StateFlow<String?> = failed.asStateFlow()

    /** The UI the agent writes for some replies, in A2UI or OpenUI, with the app's own views. */
    val ui: AIUIHost = AIUIHost(
        renderers = listOf(A2UIRenderer(), OpenUIRenderer()),
        components = listOf(placeCard),
    ) { action ->
        scope.launch {
            try {
                // What the person did, in their words, with the action for the agent.
                session.responses.create(action.label, uiAction = action.json)
            } catch (e: AgentsException) {
                failed.value = e.message
            }
        }
    }

    companion object {
        /** Opens the session's channel, in the scope of the screen's ViewModel. */
        suspend fun open(context: Context, session: AgentSession, scope: CoroutineScope): AgentChat =
            AgentChat(session, session.channelState(context, scope), scope)
    }
}

Open it in the viewModelScope of the ViewModel behind your screen, so the host outlives scrolling and configuration changes. channelState watches the session's channel on the client that chat(context) opens, and keeps its messages up to date, after a reconnect too. Then pass the host to the conversation:

val messages by chat.messages.collectAsStateWithLifecycle()
AIConversation(messages = messages, ui = chat.ui)

A reply shows its UI when one of the host's renderers reads its format and version, the first one listed asked first. Otherwise, it shows its text. The UI comes in as the agent writes it: each part fades in, rising into place, after the parts before it. A reply loaded from the channel's history shows at once. The host is confined to the main thread. Conversation View covers the rest of AIConversation, such as the agent thinking and a question while it's sent.

A component is a view of your app the agent can place, described once with the parameters the agent fills in and the events it reports. Every renderer places it by its name, so it works whichever format the agent writes:

val placeCard: AIUIComponent = AIUIComponent(
    AIUIComponentDefinition(
        name = "place_card",
        description = "A photo and a short summary of a place.",
        parameters = listOf(
            AIUIParameter.required("title", AIUIParameterType.String, description = "The place's Wikipedia title."),
        ),
        events = listOf(AIUIComponentDefinition.Event("ask_about", "The person asks about the place.")),
    ),
) { arguments, context ->
    val title = arguments["title"] as? String ?: return@AIUIComponent
    PlaceCard(title = title, onClick = { context.emit("ask_about", label = "Tell me more about $title") })
}

PlaceCard is a composable of your app. Its arguments are the JSON the agent wrote, as plain Kotlin values, so show nothing while they don't fit, such as while the agent is still writing them. A component can load what it shows on the device, such as a place's photo or the person's calendar, which then never reaches the agent. Its definition reads and writes JSON, so your apps and the agent's instructions can share one description of it, read with AIUIComponentDefinition.fromJsonString(text).

The renderers treat what the agent writes as untrusted: a reply's links open through the host, which opens only web and email links by default (linkSchemes), and its images load only from the web. A reply's UI can also run your app's client-side tools, passed to the host as tools, such as to load the person's calendar for the days of a plan. They are apart from the tools the agent calls, and the host runs one by itself only when it says it's read-only (isReadOnly).

The Android generative UI guide covers the rest: the renderers' options and theming, components, a renderer of your own for another format, and showing a reply's UI in your own views with AIUIView.

When the person does something in a reply's UI that's for the agent, such as tapping a place or submitting a form, the host calls its send with an AIUIAction. Ask the session with it, as AgentConversation does above: action.label is the question, in the person's words, such as "Tell me more about St. Naum", and aiUIActionJSON(action) is what they did, passed as uiAction to responses.create. See Send what the person did in a reply's UI.

The React guide sends the action to the channel as a message instead.

When the person does something in a reply's UI that's for the agent, such as tapping a place, choosing a follow-up or submitting a form, the host calls its send with an AIUIAction. Ask the session with it, as AgentChat does above: action.label is the question, in the person's words, such as "Tell me more about St. Naum", and action.json is what they did, passed as uiAction to responses.create. See Send what the person did in a reply's UI.

The iOS guide sends the action to the channel instead, with action.customData as the message's extra data.

The host sends a reply's actions one after another, in the order they happened, and logs a send that throws. To show the person that asking failed, catch the error in send, as the Swift assistant does. The SDK's renderers label an action with the control the person tapped, never with text the agent wrote for it, so the conversation shows only what the person chose. OpenUI sends only the values of the form the action belongs to, and never a password field's value.

When the person does something in a reply's UI that's for the agent, such as tapping a place, choosing a follow-up or submitting a form, the host calls its send with an AIUIAction. Ask the session with it, as AgentChat does above: action.label is the question, in the person's words, such as "Tell me more about St. Naum", and action.json is what they did, passed as uiAction to responses.create. See Send what the person did in a reply's UI.

The Android guide sends the action to the channel, with action.json in the message's custom data under AIUIAction.CUSTOM_DATA_KEY. The agent doesn't answer a message sent to the session's channel, so ask the session instead: the router writes the person's message into the channel itself, with the action on it.

The host calls send on the main thread and doesn't wait for it, so ask in a coroutine, as AgentChat does. responses.create throws an AgentsException when asking fails: catch it to show the person, as the Android assistant does. The SDK's renderers label an action with the control the person tapped, never with text the agent wrote for it, so the conversation shows only what the person chose. OpenUI sends only the values of the form the action belongs to, and never a password field's value. What the person types into an A2UI text field that's obscured is in none of the action's details.

The Swift assistant sample plans a trip as interactive UI. Its settings choose how the agent answers: in Markdown, or with UI in A2UI or OpenUI. Ask "Plan a weekend in Ohrid" and the reply is a map of the places, the person's own calendar for those days, a tab for each day and a form to change the plan. The map, the place cards and the calendar are the app's own views, described once in UIComponents.json and placed by both formats. The agent learns them from a2ui/instructions.md or openui/instructions.md, which are generated from that file with the renderers' own code. When the settings change, the sample's backend rewrites the agent's instructions: its instructions.md, followed by the chosen format's.

The React assistant sample shows the same replies in the browser, with the page's own components for the map, the place cards and the calendar. Tapping a place or sending the form asks the agent in the person's words, with what they did as its uiAction, and the agent answers with a new plan.

The Android assistant sample shows them in a Compose app. Its Settings screen chooses how the agent answers: in Markdown, or with UI in A2UI or OpenUI.

The Swift assistant sample plans a trip as interactive UI. Its settings choose how the agent answers: in Markdown, or with UI in A2UI or OpenUI. Ask "Plan a weekend in Ohrid" and the reply is a map of the places, the person's own calendar for those days, a tab for each day and a form to change the pace and the interests.

• AgentChat.swift keeps one AIUIHost with both renderers for each conversation, so a reply shows whichever UI it carries, and its send asks the session with responses.create(_:uiAction:).
• TripComponents.swift holds the map, the place cards and the calendar, the app's own views. They're described once in UIComponents.json, which the app decodes as [AIUIComponentDefinition], and both formats place them. The cards load their photos from Wikipedia and the calendar reads EventKit on the phone, so the agent never sees the person's calendar.
• prompts/generate.sh writes a2ui/instructions.md and openui/instructions.md from that file with the renderers' own code, from a stream-chat-swift checkout. Each prompt.json picks the components the agent may use and says when to write UI, and its examples have to render, or nothing is written.
• When the settings change, the sample's backend rewrites the agent's instructions: its instructions.md, followed by the chosen format's. A conversation keeps the instructions it started with, so the next new one is the first to answer the new way.

Tapping a place, a pin on the map, a follow-up or the form's button asks the agent in the person's words, such as "Update the plan", with the form's values in the action, and the agent answers with a new plan. The React assistant and Android assistant samples show the same replies in the browser and in a Compose app.

The Android assistant sample shows the replies of the Swift assistant's agent in a Compose app. Its Settings screen chooses how the agent answers: in Markdown, or with UI in A2UI or OpenUI. Ask "Plan a weekend in Ohrid" and the reply shows its UI in place of its words.

• ChatViewModel.kt keeps one AIUIHost with both renderers for the screen's life, whatever the format, so a reply shows whichever UI it carries, and earlier replies keep their UI and its state when the format changes.
• AgentChat.kt gives each message its uiDocument with AIUIDocument.from(message.extraData), and its answer asks the session with responses.create(action.label, uiAction = action.json), showing the person why when asking fails.
• ChatScreen.kt passes the host to AIConversation as ui.
• The setting is kept by the Swift assistant's backend (GET and PUT /reply-format), which rewrites the agent's instructions: its instructions.md, followed by the chosen format's. So every app answers the same way, from the next new conversation on.

The app gives its renderers no components of its own yet, so a reply shows the format's own components, without the map, the place cards and the calendar of the Swift assistant. Tapping a follow-up or sending a form asks the agent in the person's words, with what they did as its uiAction, and the agent answers again.