Generated UI

The model writes UI when the agent's instructions ask it to: add the format's own instructions, with the components of your app it may place, to the agent's instructions.md. stream-chat-react writes those instructions from the same components your app shows. The capabilities() of the app's AIUIHost hold, for each format it reads, the fence the model writes in and the instructions for the model:

// Each format the host reads, such as A2UI and OpenUI, with its `fence` and `instructions`.
const { renderers } = await ui.capabilities();
console.log(JSON.stringify(renderers, null, 2));

The page can't change the agent, so print them while you build the app and add them to instructions.md, or send them to your backend and have it add them. Write them again when your components change. You can also write them on your backend or at build time, from the same component definitions:

import { a2uiInstructions } from "stream-chat-react/a2ui";
import { openUIInstructions } from "stream-chat-react/openui";

const a2ui = await a2uiInstructions({ components: [placeCard] });
const openui = await openUIInstructions([placeCard]);

See Tell the agent what the app shows in the React guide.

The model writes UI when the agent's instructions ask it to: add the format's own instructions, with the components of your app it may place, to the agent's instructions.md. StreamChatAI writes those instructions from the same components your app shows. The capabilities of the app's AIUIHost hold, for each format it reads, the fence the model writes in and the instructions for the model: what you gave the renderer as its instructions, then the format's own, written from its components, yours and the host's tools:

// The instructions of each format the host reads, such as A2UI's and OpenUI's.
let instructions = (ui.capabilities["renderers"]?.arrayValue ?? [])
    .compactMap { $0.dictionaryValue?["instructions"]?.stringValue }

The app can't change the agent, so print them while you build the app and add them to instructions.md, or send them to your backend and have it add them. Write them again when your components change. A renderer's instructions parameter is where to say which answers to write as UI and when to use your components:

A2UIRenderer(instructions: "Write a trip plan as UI, with a place_card for each place. Answer everything else in Markdown.")

A2UI's capabilities also hold the catalog_id of the catalog with your components, getstream.io:app-catalog by default (the renderer's catalogId changes it). See Tell the agent what the app shows in the iOS guide.

AIConversation from stream-chat-react shows a reply's UI in place of its text, brings it in as the agent writes it, and sends what the person does in it back to the agent. Its renderers of A2UI and OpenUI are in stream-chat-react/a2ui and stream-chat-react/openui, from version 14.16.0. Each is built on its format's official packages, which your app installs for the formats your agent writes, at the versions the React guide's Installation names.

Keep an AIUIHost with the renderers of the formats your agent writes and your components for as long as the conversation shows, so each reply's UI keeps its state, such as what the person typed into a form. Its send asks the session, with what the person did as uiAction:

import type { Session } from "@stream-io/calypso";
import { useMemo } from "react";
import type { LocalMessage } from "stream-chat";
import {
  AIConversation,
  AIUIHost,
  aiUIActionJSON,
  fromStreamMessage,
} from "stream-chat-react/ai-components";
import { createA2UIRenderer } from "stream-chat-react/a2ui";
import { createOpenUIRenderer } from "stream-chat-react/openui";
import "stream-chat-react/css/index.css";
import "stream-chat-react/css/ai-components.css";
import "@openuidev/react-ui/components.css";

import { placeCard } from "./placeCard";

// `messages` are the messages of the channel session.chat() opened.
export const AgentConversation = ({
  session,
  messages,
}: {
  session: Session;
  messages: LocalMessage[];
}) => {
  const ui = useMemo(
    () =>
      new AIUIHost({
        components: [placeCard],
        renderers: [createA2UIRenderer(), createOpenUIRenderer()],
        // What the person did, in their words, with the action for the agent.
        send: (action) =>
          session.responses.create(action.label, {
            uiAction: aiUIActionJSON(action),
          }),
      }),
    [session],
  );

  return (
    <AIConversation
      messages={messages.map((message) => fromStreamMessage(message))}
      ui={ui}
    />
  );
};

A reply shows its UI when one of the host's renderers reads its format and version, the first one listed asked first. Otherwise, it shows its text.

A component is a view of your app the agent can place, described once with the parameters the agent fills in and the events it reports. Every renderer places it by its name, so it works whichever format the agent writes:

import {
  AIUIParameterType,
  defineAIUIComponent,
  requiredAIUIParameter,
} from "stream-chat-react/ai-components";

export const placeCard = defineAIUIComponent<{ title: string }>(
  {
    name: "place_card",
    description: "A photo and a short summary of a place.",
    parameters: [
      requiredAIUIParameter(
        "title",
        AIUIParameterType.string,
        "The place's Wikipedia title.",
      ),
    ],
    events: [
      {
        name: "ask_about",
        description: "The person asks about the place.",
        parameters: [],
      },
    ],
  },
  ({ args, context }) => (
    <PlaceCard
      title={args.title}
      onClick={() =>
        context.emit("ask_about", `Tell me more about ${args.title}`)
      }
    />
  ),
);

A component can load what it shows in the browser, such as a place's photo, which then never reaches the agent. The React generative UI guide covers the rest: the renderers' options, theming, running your app's tools from a reply's UI and handling errors. The iOS and Android guides do the same in the apps.

StreamChatAI's AIConversationView shows a reply's UI in place of its text, brings it in as the agent writes it, and sends what the person does in it back to the agent. Its renderers of A2UI and OpenUI are the StreamChatA2UI and StreamChatOpenUI libraries, from version 5.16.0 of the Stream Chat Swift SDK. Add the library of each format your agent writes to your app target, next to StreamChatAI:

The quickstart's chat UI shows the session's channel with AgentChat and AgentMessage. Give each message the UI the router moved out of its text, as its uiDocument:

struct AgentMessage: AIMessage {
    let message: ChatMessage

    // The id, role, text, isGenerating and parts, as in the quickstart.

    // The reply's UI, from the message's `custom.ai_ui`.
    var uiDocument: AIUIDocument? { AIUIDocument(customData: message.extraData) }
}

Keep an AIUIHost with the renderers of the formats your agent writes and your components in AgentChat, so each reply's UI keeps its state, such as what the person typed into a form, for as long as the conversation shows. Its send asks the session, as the composer does:

import StreamChat
import StreamChatA2UI
import StreamChatAI
import StreamChatOpenUI
import SwiftUI
import VisionAgentsCore

@MainActor
final class AgentChat: ObservableObject, ChatChannelControllerDelegate {
    let session: AgentSession
    /// The UI the agent writes for some replies, in A2UI or OpenUI, with the app's own views.
    let ui: AIUIHost<AIUIRendererPair<A2UIRenderer, OpenUIRenderer>>
    @Published private(set) var messages: [AgentMessage] = []
    @Published private(set) var isAnswering = false

    private let channel: ChatChannelController

    init(session: AgentSession, channel: ChatChannelController) {
        self.session = session
        self.channel = channel
        ui = AIUIHost(components: [PlaceCardComponent()]) {
            A2UIRenderer()
            OpenUIRenderer()
        } send: { [session] action in
            // What the person did, in their words, with the action for the agent.
            _ = try await session.responses.create(action.label, uiAction: action.json)
        }
        channel.delegate = self
        channel.synchronize()
    }

    // channelController(_:didUpdateMessages:) as in the quickstart.
}

Then pass the host to the conversation:

AIConversationView(messages: chat.messages, ui: chat.ui)

A reply shows its UI when one of the host's renderers reads its format and version, the first one listed asked first. Otherwise, it shows its text. The UI comes in as the agent writes it: each part fades in, rising into place, after the parts before it. A reply loaded from the channel's history shows at once. With an AIConversation, as the Swift assistant has, pass the host the same way: AIConversationView(conversation: conversation, ui: ui).

OpenUIRenderer needs iOS 17. With an earlier deployment target, keep a host of A2UIRenderer alone, an AIUIHost<A2UIRenderer>, before iOS 17.

A component is a view of your app the agent can place, described once with the parameters the agent fills in and the events it reports. Every renderer places it by its name, so it works whichever format the agent writes:

struct PlaceCardComponent: AIUIComponent {
    struct Arguments: Decodable {
        let title: String
    }

    let definition = AIUIComponentDefinition(
        name: "place_card",
        description: "A photo and a short summary of a place.",
        parameters: [.required("title", .string, description: "The place's Wikipedia title.")],
        events: [.init(name: "ask_about", description: "The person asks about the place.")]
    )

    func body(arguments: Arguments, context: AIUIComponentContext) -> some View {
        PlaceCard(title: arguments.title) {
            context.emit("ask_about", label: "Tell me more about \(arguments.title)")
        }
    }
}

A component can load what it shows on the device, such as a place's photo or the person's calendar, which then never reaches the agent. Its definition reads and writes JSON, so your apps and the agent's instructions can share one description of it, decoded with JSONDecoder as [AIUIComponentDefinition].

The renderers treat what the agent writes as untrusted: a reply's UI can't crash or freeze the app, and its links open through the host, which opens only web and email links by default. A reply's UI can also run your app's client side tools, passed to the host as tools, such as to load the person's calendar for the days of a plan. They are apart from the tools the agent calls, and the host runs one by itself only when its definition says it's read-only (readOnlyHint).

The iOS generative UI guide covers the rest: the renderers' options and theming, components, a renderer of your own for another format, and showing a reply's UI in your own views with AIUIView.

When the person does something in a reply's UI that's for the agent, such as tapping a place or submitting a form, the host calls its send with an AIUIAction. Ask the session with it, as AgentConversation does above: action.label is the question, in the person's words, such as "Tell me more about St. Naum", and aiUIActionJSON(action) is what they did, passed as uiAction to responses.create. See Send what the person did in a reply's UI.

The React guide sends the action to the channel as a message. The agent doesn't answer a message sent to the session's channel, so ask the session instead: the router writes the person's message into the channel itself, with the action on it.

When the person does something in a reply's UI that's for the agent, such as tapping a place, choosing a follow-up or submitting a form, the host calls its send with an AIUIAction. Ask the session with it, as AgentChat does above: action.label is the question, in the person's words, such as "Tell me more about St. Naum", and action.json is what they did, passed as uiAction to responses.create. See Send what the person did in a reply's UI.

The iOS guide sends the action to the channel, with action.customData as the message's extra data. The agent doesn't answer a message sent to the session's channel, so ask the session instead: the router writes the person's message into the channel itself, with the action on it.

The host sends a reply's actions one after another, in the order they happened, and logs a send that throws. To show the person that asking failed, catch the error in send, as the Swift assistant does. The SDK's renderers label an action with the control the person tapped, never with text the agent wrote for it, so the conversation shows only what the person chose. OpenUI sends only the values of the form the action belongs to, and never a password field's value.

The Swift assistant sample plans a trip as interactive UI. Its settings choose how the agent answers: in Markdown, or with UI in A2UI or OpenUI. Ask "Plan a weekend in Ohrid" and the reply is a map of the places, the person's own calendar for those days, a tab for each day and a form to change the plan. The map, the place cards and the calendar are the app's own views, described once in UIComponents.json and placed by both formats. The agent learns them from a2ui/instructions.md or openui/instructions.md, which are generated from that file with the renderers' own code. When the settings change, the sample's backend rewrites the agent's instructions: its instructions.md, followed by the chosen format's.

The React assistant sample shows the same replies in the browser, with the page's own components for the map, the place cards and the calendar. Tapping a place or sending the form asks the agent in the person's words, with what they did as its uiAction, and the agent answers with a new plan.

The Android assistant sample shows them in a Compose app. Its Settings screen chooses how the agent answers: in Markdown, or with UI in A2UI or OpenUI.

The Swift assistant sample plans a trip as interactive UI. Its settings choose how the agent answers: in Markdown, or with UI in A2UI or OpenUI. Ask "Plan a weekend in Ohrid" and the reply is a map of the places, the person's own calendar for those days, a tab for each day and a form to change the pace and the interests.

• AgentChat.swift keeps one AIUIHost with both renderers for each conversation, so a reply shows whichever UI it carries, and its send asks the session with responses.create(_:uiAction:).
• TripComponents.swift holds the map, the place cards and the calendar, the app's own views. They're described once in UIComponents.json, which the app decodes as [AIUIComponentDefinition], and both formats place them. The cards load their photos from Wikipedia and the calendar reads EventKit on the phone, so the agent never sees the person's calendar.
• prompts/generate.sh writes a2ui/instructions.md and openui/instructions.md from that file with the renderers' own code, from a stream-chat-swift checkout. Each prompt.json picks the components the agent may use and says when to write UI, and its examples have to render, or nothing is written.
• When the settings change, the sample's backend rewrites the agent's instructions: its instructions.md, followed by the chosen format's. A conversation keeps the instructions it started with, so the next new one is the first to answer the new way.

Tapping a place, a pin on the map, a follow-up or the form's button asks the agent in the person's words, such as "Update the plan", with the form's values in the action, and the agent answers with a new plan. The React assistant and Android assistant samples show the same replies in the browser and in a Compose app.