# Generated UI An agent can answer with UI rather than text: a weekend plan with a map of the places, a tab for each day and a form to change the pace. The agent writes the UI into its reply, the router carries it to Stream Chat beside the reply's words, and your app shows it in place of them. What the person does in it comes back to the agent with their next message. The router knows no UI format. The model writes the UI in a format your app has a renderer for, such as [A2UI](https://a2ui.org) or [OpenUI](https://github.com/thesysdev/openui), and the router passes it on as it is. ## How the agent writes UI The model writes the UI in a fenced block of its reply. The block's info string is `ui:`, or `ui:@` to name the format's version, such as `ui:a2ui@1.0`. OpenUI's own fence, `openui-lang`, is read as the `openui` format, since that's the fence OpenUI's prompts ask models for: ````md Here's a weekend in Ohrid, with a map and a plan for each day. ```openui-lang root = Card([title, map]) title = CardHeader("A weekend in Ohrid") ``` ```` The first such block is the reply's UI, and any later one is left out of its text. Only the agent writes UI: a block in a person's message stays part of their words. **JavaScript** The model writes UI when the agent's instructions ask it to: add the format's own instructions, with the components of your app it may place, to the agent's `instructions.md`. `stream-chat-react` writes those instructions from the same components your app shows. The `capabilities()` of the app's [`AIUIHost`](#show-it-in-your-app) hold, for each format it reads, the `fence` the model writes in and the `instructions` for the model: ```tsx // Each format the host reads, such as A2UI and OpenUI, with its `fence` and `instructions`. const { renderers } = await ui.capabilities(); console.log(JSON.stringify(renderers, null, 2)); ``` The page can't change the agent, so print them while you build the app and add them to `instructions.md`, or send them to your backend and have it add them. Write them again when your components change. You can also write them on your backend or at build time, from the same component definitions: ```tsx import { a2uiInstructions } from "stream-chat-react/a2ui"; import { openUIInstructions } from "stream-chat-react/openui"; const a2ui = await a2uiInstructions({ components: [placeCard] }); const openui = await openUIInstructions([placeCard]); ``` See [Tell the agent what the app shows](https://getstream.io/chat/docs/sdk/react/v14/guides/ai-integrations/generative-ui/#tell-the-agent-what-the-app-shows) in the React guide. **Swift** The model writes UI when the agent's instructions ask it to: add the format's own instructions, with the components of your app it may place, to the agent's `instructions.md`. StreamChatAI writes those instructions from the same components your app shows. The `capabilities` of the app's [`AIUIHost`](#show-it-in-your-app) hold, for each format it reads, the `fence` the model writes in and the `instructions` for the model: what you gave the renderer as its `instructions`, then the format's own, written from its components, yours and the host's tools: ```swift // The instructions of each format the host reads, such as A2UI's and OpenUI's. let instructions = (ui.capabilities["renderers"]?.arrayValue ?? []) .compactMap { $0.dictionaryValue?["instructions"]?.stringValue } ``` The app can't change the agent, so print them while you build the app and add them to `instructions.md`, or send them to your backend and have it add them. Write them again when your components change. A renderer's `instructions` parameter is where to say which answers to write as UI and when to use your components: ```swift A2UIRenderer(instructions: "Write a trip plan as UI, with a place_card for each place. Answer everything else in Markdown.") ``` A2UI's capabilities also hold the `catalog_id` of the catalog with your components, `getstream.io:app-catalog` by default (the renderer's `catalogId` changes it). See [Tell the agent what the app shows](https://getstream.io/chat/docs/sdk/ios/guides/ai-integrations/generative-ui/#tell-the-agent-what-the-app-shows) in the iOS guide. ## What reaches Stream Chat As the reply streams, the router moves the block out of its text into the message's custom data, as `custom.ai_ui`, and keeps the words around it as the message's text: ```json { "v": 1, "format": "openui", "status": "streaming", "content": "root = Card([title, map])\ntitle = CardHeader(\"A weekend in Ohrid\")" } ``` | Field | Description | | --------- | ---------------------------------------------------------------------------- | | `v` | The version of the field, `1`. | | `format` | The format from the block's info string, such as `a2ui` or `openui`. | | `version` | The format's version, such as `1.0`, when the info string names one. | | `status` | `streaming` while the agent is still writing the block, then `complete`. | | `content` | The block as the model wrote it, such as an OpenUI program or A2UI messages. | The text is what people read wherever the UI can't show: in a push notification, in an external thread and in an app without a renderer for the format. So have the model write a sentence saying what the UI shows before the block. Stream Chat holds at most 50 KB of custom data on a message. When the UI would take the message's custom data over 48 KB as JSON, the router leaves it out, and the reply keeps only its words. A UI that was showing while the reply streamed is taken off again, rather than left cut off midway. When the conversation goes on, the model reads its earlier replies with their UI, written back as the blocks it wrote. ## Show it in your app **JavaScript** `AIConversation` from `stream-chat-react` shows a reply's UI in place of its text, brings it in as the agent writes it, and sends what the person does in it back to the agent. Its renderers of A2UI and OpenUI are in `stream-chat-react/a2ui` and `stream-chat-react/openui`, from version 14.16.0. Each is built on its format's official packages, which your app installs for the formats your agent writes, at the versions the React guide's [Installation](https://getstream.io/chat/docs/sdk/react/v14/guides/ai-integrations/generative-ui/#installation) names. Keep an `AIUIHost` with the renderers of the formats your agent writes and your components for as long as the conversation shows, so each reply's UI keeps its state, such as what the person typed into a form. Its `send` asks the session, with what the person did as `uiAction`: ```tsx title="AgentConversation.tsx" import type { Session } from "@stream-io/calypso"; import { useMemo } from "react"; import type { LocalMessage } from "stream-chat"; import { AIConversation, AIUIHost, aiUIActionJSON, fromStreamMessage, } from "stream-chat-react/ai-components"; import { createA2UIRenderer } from "stream-chat-react/a2ui"; import { createOpenUIRenderer } from "stream-chat-react/openui"; import "stream-chat-react/css/index.css"; import "stream-chat-react/css/ai-components.css"; import "@openuidev/react-ui/components.css"; import { placeCard } from "./placeCard"; // `messages` are the messages of the channel session.chat() opened. export const AgentConversation = ({ session, messages, }: { session: Session; messages: LocalMessage[]; }) => { const ui = useMemo( () => new AIUIHost({ components: [placeCard], renderers: [createA2UIRenderer(), createOpenUIRenderer()], // What the person did, in their words, with the action for the agent. send: (action) => session.responses.create(action.label, { uiAction: aiUIActionJSON(action), }), }), [session], ); return ( fromStreamMessage(message))} ui={ui} /> ); }; ``` A reply shows its UI when one of the host's renderers reads its format and version, the first one listed asked first. Otherwise, it shows its text. A component is a view of your app the agent can place, described once with the parameters the agent fills in and the events it reports. Every renderer places it by its name, so it works whichever format the agent writes: ```tsx title="placeCard.tsx" import { AIUIParameterType, defineAIUIComponent, requiredAIUIParameter, } from "stream-chat-react/ai-components"; export const placeCard = defineAIUIComponent<{ title: string }>( { name: "place_card", description: "A photo and a short summary of a place.", parameters: [ requiredAIUIParameter( "title", AIUIParameterType.string, "The place's Wikipedia title.", ), ], events: [ { name: "ask_about", description: "The person asks about the place.", parameters: [], }, ], }, ({ args, context }) => ( context.emit("ask_about", `Tell me more about ${args.title}`) } /> ), ); ``` A component can load what it shows in the browser, such as a place's photo, which then never reaches the agent. The [React generative UI guide](https://getstream.io/chat/docs/sdk/react/v14/guides/ai-integrations/generative-ui/) covers the rest: the renderers' options, theming, running your app's tools from a reply's UI and handling errors. The [iOS](https://getstream.io/chat/docs/sdk/ios/guides/ai-integrations/generative-ui/) and [Android](https://getstream.io/chat/docs/sdk/android/guides/ai-integrations/generative-ui/) guides do the same in the apps. **Swift** StreamChatAI's `AIConversationView` shows a reply's UI in place of its text, brings it in as the agent writes it, and sends what the person does in it back to the agent. Its renderers of A2UI and OpenUI are the `StreamChatA2UI` and `StreamChatOpenUI` libraries, from version 5.16.0 of the Stream Chat Swift SDK. Add the library of each format your agent writes to your app target, next to StreamChatAI: | Library | Renderer | Format | Requires | | ------------------ | ---------------- | --------------- | -------- | | `StreamChatA2UI` | `A2UIRenderer` | A2UI 1.0 | iOS 15 | | `StreamChatOpenUI` | `OpenUIRenderer` | OpenUI Lang 1.0 | iOS 17 | The [quickstart's chat UI](https://getstream.io/agents/docs/dotnet-csharp/#agent-ui) shows the session's channel with `AgentChat` and `AgentMessage`. Give each message the UI the router moved out of its text, as its `uiDocument`: ```swift struct AgentMessage: AIMessage { let message: ChatMessage // The id, role, text, isGenerating and parts, as in the quickstart. // The reply's UI, from the message's `custom.ai_ui`. var uiDocument: AIUIDocument? { AIUIDocument(customData: message.extraData) } } ``` Keep an `AIUIHost` with the renderers of the formats your agent writes and your components in `AgentChat`, so each reply's UI keeps its state, such as what the person typed into a form, for as long as the conversation shows. Its `send` asks the session, as the composer does: ```swift import StreamChat import StreamChatA2UI import StreamChatAI import StreamChatOpenUI import SwiftUI import VisionAgentsCore @MainActor final class AgentChat: ObservableObject, ChatChannelControllerDelegate { let session: AgentSession /// The UI the agent writes for some replies, in A2UI or OpenUI, with the app's own views. let ui: AIUIHost> @Published private(set) var messages: [AgentMessage] = [] @Published private(set) var isAnswering = false private let channel: ChatChannelController init(session: AgentSession, channel: ChatChannelController) { self.session = session self.channel = channel ui = AIUIHost(components: [PlaceCardComponent()]) { A2UIRenderer() OpenUIRenderer() } send: { [session] action in // What the person did, in their words, with the action for the agent. _ = try await session.responses.create(action.label, uiAction: action.json) } channel.delegate = self channel.synchronize() } // channelController(_:didUpdateMessages:) as in the quickstart. } ``` Then pass the host to the conversation: ```swift AIConversationView(messages: chat.messages, ui: chat.ui) ``` A reply shows its UI when one of the host's renderers reads its format and version, the first one listed asked first. Otherwise, it shows its text. The UI comes in as the agent writes it: each part fades in, rising into place, after the parts before it. A reply loaded from the channel's history shows at once. With an [`AIConversation`](https://getstream.io/chat/docs/sdk/ios/guides/ai-integrations/conversation-state/#generative-ui), as the Swift assistant has, pass the host the same way: `AIConversationView(conversation: conversation, ui: ui)`. `OpenUIRenderer` needs iOS 17. With an earlier deployment target, keep a host of `A2UIRenderer` alone, an `AIUIHost`, before iOS 17. A component is a view of your app the agent can place, described once with the parameters the agent fills in and the events it reports. Every renderer places it by its name, so it works whichever format the agent writes: ```swift struct PlaceCardComponent: AIUIComponent { struct Arguments: Decodable { let title: String } let definition = AIUIComponentDefinition( name: "place_card", description: "A photo and a short summary of a place.", parameters: [.required("title", .string, description: "The place's Wikipedia title.")], events: [.init(name: "ask_about", description: "The person asks about the place.")] ) func body(arguments: Arguments, context: AIUIComponentContext) -> some View { PlaceCard(title: arguments.title) { context.emit("ask_about", label: "Tell me more about \(arguments.title)") } } } ``` A component can load what it shows on the device, such as a place's photo or the person's calendar, which then never reaches the agent. Its definition reads and writes JSON, so your apps and the agent's instructions can share one description of it, decoded with `JSONDecoder` as `[AIUIComponentDefinition]`. The renderers treat what the agent writes as untrusted: a reply's UI can't crash or freeze the app, and its links open through the host, which opens only web and email links by default. A reply's UI can also run your app's [client side tools](https://getstream.io/chat/docs/sdk/ios/guides/ai-integrations/client-side-tools/), passed to the host as `tools`, such as to load the person's calendar for the days of a plan. They are apart from the tools the agent calls, and the host runs one by itself only when its definition says it's read-only (`readOnlyHint`). The [iOS generative UI guide](https://getstream.io/chat/docs/sdk/ios/guides/ai-integrations/generative-ui/) covers the rest: the [renderers' options](https://getstream.io/chat/docs/sdk/ios/guides/ai-integrations/generative-ui/#renderers) and theming, [components](https://getstream.io/chat/docs/sdk/ios/guides/ai-integrations/generative-ui/#components), a [renderer of your own](https://getstream.io/chat/docs/sdk/ios/guides/ai-integrations/generative-ui/#write-a-renderer) for another format, and showing a reply's UI [in your own views](https://getstream.io/chat/docs/sdk/ios/guides/ai-integrations/generative-ui/#show-it-in-your-own-views) with `AIUIView`. ## What the person does **JavaScript** When the person does something in a reply's UI that's for the agent, such as tapping a place or submitting a form, the host calls its `send` with an `AIUIAction`. Ask the session with it, as `AgentConversation` does above: `action.label` is the question, in the person's words, such as "Tell me more about St. Naum", and `aiUIActionJSON(action)` is what they did, passed as `uiAction` to `responses.create`. See [Send what the person did in a reply's UI](https://getstream.io/agents/docs/dotnet-csharp/responses-create/#send-what-the-person-did-in-a-replys-ui). The React guide sends the action to the channel as a message. The agent doesn't answer a message sent to the session's channel, so ask the session instead: the router writes the person's message into the channel itself, with the action on it. **Swift** When the person does something in a reply's UI that's for the agent, such as tapping a place, choosing a follow-up or submitting a form, the host calls its `send` with an `AIUIAction`. Ask the session with it, as `AgentChat` does above: `action.label` is the question, in the person's words, such as "Tell me more about St. Naum", and `action.json` is what they did, passed as `uiAction` to `responses.create`. See [Send what the person did in a reply's UI](https://getstream.io/agents/docs/dotnet-csharp/responses-create/#send-what-the-person-did-in-a-replys-ui). | Property | Description | | ----------- | ------------------------------------------------------------------------------------------------------------------------------ | | `messageID` | The reply whose UI it came from. | | `format` | The format of that UI. | | `name` | What happened, such as a component's event (`ask_about`) or an action of the format, such as OpenUI's `continue_conversation`. | | `label` | What the person did, in their words. Use it as the question. | | `payload` | The details, in the format's own shape, such as what was chosen and a form's values. | The [iOS guide](https://getstream.io/chat/docs/sdk/ios/guides/ai-integrations/generative-ui/#what-the-user-does) sends the action to the channel, with `action.customData` as the message's extra data. The agent doesn't answer a message sent to the session's channel, so ask the session instead: the router writes the person's message into the channel itself, with the action on it. The host sends a reply's actions one after another, in the order they happened, and logs a `send` that throws. To show the person that asking failed, catch the error in `send`, as the Swift assistant does. The SDK's renderers label an action with the control the person tapped, never with text the agent wrote for it, so the conversation shows only what the person chose. OpenUI sends only the values of the form the action belongs to, and never a password field's value. > > **Info:** The Swift SDK takes `uiAction` on the `accelerate` branch of Vision-Agents, from tag `0.2.0`. > The action names the reply it came from (`message_id`, the id of the reply's Stream Chat message), the UI's `format` and the action (`name`), with what it carries (`payload`), in at most 8 KB as JSON. The router keeps it on the person's message as `custom.ai_ui_action`, and tells the model what they did after their words, with the opening words of the reply it was in. The model reads the action as data, not as instructions. ## Examples **JavaScript** The [Swift assistant](https://github.com/GetStream/Vision-Agents/tree/accelerate/examples/voice_agents/swift_assistant) sample plans a trip as interactive UI. Its settings choose how the agent answers: in Markdown, or with UI in A2UI or OpenUI. Ask "Plan a weekend in Ohrid" and the reply is a map of the places, the person's own calendar for those days, a tab for each day and a form to change the plan. The map, the place cards and the calendar are the app's own views, described once in `UIComponents.json` and placed by both formats. The agent learns them from `a2ui/instructions.md` or `openui/instructions.md`, which are generated from that file with the renderers' own code. When the settings change, the sample's backend rewrites the agent's instructions: its `instructions.md`, followed by the chosen format's. The [React assistant](https://github.com/GetStream/Vision-Agents/tree/accelerate/examples/voice_agents/react_assistant) sample shows the same replies in the browser, with the page's own components for the map, the place cards and the calendar. Tapping a place or sending the form asks the agent in the person's words, with what they did as its `uiAction`, and the agent answers with a new plan. The [Android assistant](https://github.com/GetStream/Vision-Agents/tree/accelerate/examples/voice_agents/android_assistant) sample shows them in a Compose app. Its Settings screen chooses how the agent answers: in Markdown, or with UI in A2UI or OpenUI. **Swift** The [Swift assistant](https://github.com/GetStream/Vision-Agents/tree/accelerate/examples/voice_agents/swift_assistant) sample plans a trip as interactive UI. Its settings choose how the agent answers: in Markdown, or with UI in A2UI or OpenUI. Ask "Plan a weekend in Ohrid" and the reply is a map of the places, the person's own calendar for those days, a tab for each day and a form to change the pace and the interests. - `AgentChat.swift` keeps one `AIUIHost` with both renderers for each conversation, so a reply shows whichever UI it carries, and its `send` asks the session with `responses.create(_:uiAction:)`. - `TripComponents.swift` holds the map, the place cards and the calendar, the app's own views. They're described once in `UIComponents.json`, which the app decodes as `[AIUIComponentDefinition]`, and both formats place them. The cards load their photos from Wikipedia and the calendar reads EventKit on the phone, so the agent never sees the person's calendar. - `prompts/generate.sh` writes `a2ui/instructions.md` and `openui/instructions.md` from that file with the renderers' own code, from a `stream-chat-swift` checkout. Each `prompt.json` picks the components the agent may use and says when to write UI, and its examples have to render, or nothing is written. - When the settings change, the sample's backend rewrites the agent's instructions: its `instructions.md`, followed by the chosen format's. A conversation keeps the instructions it started with, so the next new one is the first to answer the new way. Tapping a place, a pin on the map, a follow-up or the form's button asks the agent in the person's words, such as "Update the plan", with the form's values in the action, and the agent answers with a new plan. The [React assistant](https://github.com/GetStream/Vision-Agents/tree/accelerate/examples/voice_agents/react_assistant) and [Android assistant](https://github.com/GetStream/Vision-Agents/tree/accelerate/examples/voice_agents/android_assistant) samples show the same replies in the browser and in a Compose app. --- For the most recent version of this documentation, visit [https://getstream.io/agents/docs/dotnet-csharp/generated-ui/](https://getstream.io/agents/docs/dotnet-csharp/generated-ui/).