# Generative UI An agent can answer with UI rather than text: a weekend plan with a map of the places, the user's own calendar for those days and a form to change the pace. The AI components show a reply's UI in place of its text, bring it in as the agent writes it, and send what the user does in it back to the agent as their next message. The AI components know no UI format of their own. You give a conversation a renderer for each format your agent writes, and the composables of your app the agent can place, such as a map, a calendar or a card over your own data. The SDK ships renderers of [A2UI](https://a2ui.org) and [OpenUI](https://github.com/thesysdev/openui), and you can write one for any other format. A component works whichever format the agent writes. > > **Info:** Generative UI is part of the `stream-chat-android-ai-compose` artifact, and the A2UI and OpenUI renderers are the `stream-chat-android-ai-a2ui` and `stream-chat-android-ai-openui` artifacts, all available from version 7.18.0. See [Renderers](#renderers). > ## How a reply carries its UI The agent's backend writes a reply's UI in the message's `ai_ui` custom field. The reply's text stays readable on its own, for wherever the UI can't show, such as a push notification or an app without a renderer for the format: ```json { "text": "Here's a weekend in Ohrid, with a map and a plan for each day.", "ai_ui": { "v": 1, "format": "openui", "status": "streaming", "content": "root = Card([title, map])\ntitle = CardHeader(\"A weekend in Ohrid\")" } } ``` | Field | Description | | --------- | ----------------------------------------------------------------------------------------------- | | `v` | The version of the field, `1`. An app that doesn't know a newer version shows the reply's text. | | `format` | The format the UI is written in, such as `a2ui` or `openui`. | | `version` | The format's version, such as `1.0`, when the agent names one. Optional. | | `status` | `streaming` while the agent is still writing the UI, then `complete`. | | `content` | The UI in its format: text, such as an OpenUI program or A2UI messages one per line, or JSON. | A model usually writes the UI into its reply, for example in a fenced code block. Have your backend move it out of the text into `ai_ui` as the reply streams, so the text keeps only the words around it. A message's custom data holds at most 50 KB, so leave the UI out when it doesn't fit beside the rest of it. Read the field with `AIUIDocument.from`, as the `uiDocument` of your `AIMessage`, such as the `StreamAIMessage` from [Messages from Stream Chat](https://getstream.io/chat/docs/sdk/android/guides/ai-integrations/conversation-view/#messages-from-stream-chat): ```kotlin override val uiDocument: AIUIDocument? = AIUIDocument.from(message.extraData) ``` It returns `null` when the message carries no `ai_ui` field the SDK can read, and the reply shows its text. ## Show it in a conversation Make an `AIUIHost` with the renderers of the formats your agent writes and your components, and pass it to `AIConversation` as `ui`. Keep one host for as long as its conversation shows, for example in the `ViewModel` behind your screen, so each reply's UI keeps its state, such as what the user typed into a form, across scrolling and configuration changes: ```kotlin class AssistantViewModel(private val cid: String) : ViewModel() { val ui: AIUIHost = AIUIHost( renderers = listOf(A2UIRenderer(), OpenUIRenderer()), components = listOf(placeCard), ) { action -> ChatClient.instance().channel(cid) .sendMessage(Message(text = action.label, extraData = mapOf(AIUIAction.CUSTOM_DATA_KEY to action.json))) .enqueue() } } ``` ```kotlin AIConversation(messages = messages, ui = viewModel.ui) ``` `placeCard` is one of your [components](#components). A reply shows its UI in place of its text, after its [steps](https://getstream.io/chat/docs/sdk/android/guides/ai-integrations/reasoning-and-tool-calls/), when one of the host's renderers reads its format and version, the first one listed asked first. Otherwise, it shows its text. The UI comes in as the agent writes it: each part fades in, rising into place, after the parts before it. A reply written before it showed, such as one loaded from history, shows at once. A reply with UI the host can show counts as showing something, so the conversation stops showing the agent thinking as soon as its UI starts. The [reply actions](https://getstream.io/chat/docs/sdk/android/guides/ai-integrations/conversation-view/#reply-actions) slot shows under a reply once its UI is written, as it does under text. Its `text` is still the reply's text, which the reply doesn't show while it shows its UI. The host is confined to the main thread. ## Renderers A renderer reads one format, and shows each reply's UI written in it. Add the artifact of each format your agent writes next to `stream-chat-android-ai-compose`, which both depend on: ```kotlin dependencies { implementation("io.getstream:stream-chat-android-ai-a2ui:$stream_version") implementation("io.getstream:stream-chat-android-ai-openui:$stream_version") } ``` | Artifact | Renderer | Format | | ------------------------------- | ---------------- | --------------- | | `stream-chat-android-ai-a2ui` | `A2UIRenderer` | A2UI 1.0 | | `stream-chat-android-ai-openui` | `OpenUIRenderer` | OpenUI Lang 1.0 | Both require `minSdk` 23, like the AI components. Both renderers place your app's [components](#components) beside their own, run your app's [tools](#run-your-apps-tools), and draw their own components with Material 3, so a reply's UI follows your `MaterialTheme`. They take these options: | Parameter | Description | | -------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `instructions` | What the agent's model is told ahead of the format's own instructions, such as which answers to write as UI and when to use your components. See [Tell the agent what the app shows](#tell-the-agent-what-the-app-shows). | | `onError` | Gets the problems with a reply's UI, such as a component the format doesn't know or a tool that failed, with the reply's id. A chat has no channel for them, so forward them to your backend if the agent should hear about them. They are logged by default. See [Errors](#errors). | `A2UIRenderer` also takes a `catalog`, a `catalogId` and a `mediaUrl`, and `OpenUIRenderer` a `library`, as described below. The renderers treat what the agent writes as untrusted: links in a reply's UI open through the host, which opens only web and email links by default, and images load only from the web. ### A2UI `A2UIRenderer` reads [A2UI 1.0](https://a2ui.org), with every component and function of its basic catalog drawn in Compose. The agent writes with the basic catalog, which your components join under the `getstream.io:app-catalog` id (`catalogId` changes it). A surface that names the basic catalog's id shows them as well. Each reply has surfaces of its own, kept while the host is, so what the user enters in them survives scrolling. A button reaches the host as an `AIUIAction` named after its event, labeled with the button's text, or else with the event's name as words (`submit_form` is "Submit form"). Its payload holds the `surfaceId`, the `sourceComponentId`, the event's resolved `context`, the event's `userMessage` when it has one, and the surface's `dataModel` when the agent created the surface with `sendDataModel`. A function the catalog doesn't define runs as one of the host's [tools](#run-your-apps-tools), with the tool's output as the function's value. By default, only web links load as images, video and audio. Pass `mediaUrl` to return the URL an image, video or audio clip loads from, given the one the agent wrote, or `null` to not load it, such as to load media through your own CDN. The agent writes these URLs, so return only ones your app trusts. Images load only from web links, whatever it returns: ```kotlin val renderer: A2UIRenderer = A2UIRenderer( mediaUrl = { url -> url.takeIf { it.startsWith("https://media.example.com/") } }, ) ``` To change how one of the basic catalog's components looks, give the renderer a catalog in which your composable draws it. It reads the component through an `A2UIComponentScope`, with its bindings resolved, and draws its children with the same catalog: ```kotlin val catalog: A2UICatalog = A2UICatalog.Basic.with("Card") { scope -> BrandCard(modifier = scope.modifier) { scope.child()?.let { scope.Child(it) } } } val renderer: A2UIRenderer = A2UIRenderer(catalog = catalog) ``` Here `BrandCard` is a composable of your app. Apply `scope.modifier` to the outermost element: it carries what the parent asks of the component, such as its weight in a `Row`, and its accessibility label. The scope reads properties with `value`, `string`, `number`, `boolean` and `stringList`, writes an input's value to the surface's data model with `write`, and runs what the component does with `performAction`. `removing` leaves out components you don't want the agent to use. The agent isn't told about them, and a reply that still uses one shows nothing for it: ```kotlin val renderer: A2UIRenderer = A2UIRenderer(catalog = A2UICatalog.Basic.removing("Video", "AudioPlayer")) ``` ### OpenUI `OpenUIRenderer` reads [OpenUI Lang 1.0](https://github.com/thesysdev/openui). The agent writes with the chat library by default, `openuiChatLibrary`, where every reply is one card. Pass `library = openuiLibrary` for the standard library, with a `Stack` root, layouts and modals, for surfaces such as dashboards. Each reply keeps its own state while the host is kept, so its forms and `$variables` survive scrolling. A follow-up or a form's submit button reaches the host as an `AIUIAction` named after OpenUI's action, such as `continue_conversation`, labeled with the control the user tapped. Its payload holds the action's `context`, its other `params`, the `form` and the form's `values`. A `Query` runs one of the host's read-only [tools](#run-your-apps-tools) while the reply shows, and a `Mutation` runs any of them when the user taps. Images load only from public web hosts, at about the size they show at. To change how one of the library's components looks, give the renderer a library in which your composable draws it. It gets the component's `ComponentRenderProps`, with its `props`, `isStreaming`, `RenderProp` for a prop that holds components, and `DefaultView`, the library's own view of it, to wrap: ```kotlin val library: Library = openuiChatLibrary .replacingView("Button") { props -> Box(Modifier.padding(4.dp)) { props.DefaultView() } } .removing(setOf("Carousel")) val renderer: OpenUIRenderer = OpenUIRenderer(library = library) ``` `replacingView` keeps the component's props and what the agent is told about it. `removing` leaves out components you don't want the agent to use, also from every slot that takes several types. `componentNames` lists the components a library has. ### Write a renderer For any other format, write a renderer of your own. A renderer makes an engine for each conversation's host, which makes the content of each reply's UI: ```kotlin class CardsRenderer : AIUIRenderer { override val format: AIUIFormat = AIUIFormat("cards") override fun makeEngine(host: AIUIHost): AIUIEngine = CardsEngine() } class CardsEngine : AIUIEngine { override fun content(document: AIUIDocument, messageId: String): (@Composable () -> Unit)? { val cards = document.content as? List<*> ?: return null return { Cards(cards) } } } @Composable fun Cards(cards: List<*>) { Column { cards.forEach { card -> AIUIEntrance { Text(text = (card as? Map<*, *>)?.get("title") as? String ?: "") } } } } ``` List it in the host like any other renderer. A document's `content` is the agent's text, which `text` reads, or JSON as plain Kotlin values: maps, lists, strings, numbers, booleans and `null`. - `content(document, messageId)` is asked again each time the reply's document changes, such as while the agent writes it, while the conversation composes. Return `null` to show the reply's text instead. - The host makes one engine per renderer and keeps it for as long as it's kept, so an engine can hold each reply's state by its `messageId`. Keep what shows on screen in snapshot state, so the UI follows it. The host calls the engine on the main thread. - `receive(document, messageId)` gets each reply's document as it arrives or changes, oldest first, including replies that aren't on screen. Use it for a format whose later replies change an earlier one's UI. `remove(messageId)` tells the engine a reply, or its UI, left the conversation, so it can let go of what it kept for it. - `supports(version)` says which versions of the format it reads. It reads every version by default. - `capabilities(components, tools)` says what the agent's backend needs to write the format, such as the instructions for its model. See [Tell the agent what the app shows](#tell-the-agent-what-the-app-shows). A renderer places the app's components by name, with `host.component(name)`, among the ones its format has. A component's `Content(arguments, context)` draws it for the arguments the agent wrote. Give it an `AIUIComponentContext` whose `emit` sends its events the way your format reports what the user did. ## Components A component is a composable of your app the agent can place in a reply's UI. You describe it once, with the parameters the agent fills in and the events it reports, and say how it looks for the arguments the agent wrote: ```kotlin val placeCard: AIUIComponent = AIUIComponent( AIUIComponentDefinition( name = "place_card", description = "A photo and a short summary of a place.", parameters = listOf( AIUIParameter.required("title", AIUIParameterType.String, description = "The place's Wikipedia title."), AIUIParameter.optional("visit_date", AIUIParameterType.Date), ), events = listOf(AIUIComponentDefinition.Event("ask_about", "The user asks about the place.")), ), ) { arguments, context -> val title = arguments["title"] as? String ?: return@AIUIComponent PlaceCard( title = title, visitDate = arguments["visit_date"] as? String, onClick = { context.emit("ask_about", label = "Tell me more about $title") }, ) } ``` `PlaceCard` is a composable of your app. Every renderer places it by its name, so it works whichever format the agent writes. A2UI and OpenUI place it under its name in `PascalCase`, so `place_card` is `PlaceCard`. A component named like one of the format's own, such as `Card` in A2UI, would replace it, so the renderer leaves it out. The arguments are the JSON the agent wrote, as plain Kotlin values: maps, lists, strings, numbers, booleans and `null`. Read what you need, and show nothing while they don't fit, such as while the agent is still writing them. A date arrives as the text the agent wrote, such as `2026-11-14`. `context.isStreaming` tells whether the arguments may still change, and `context.messageId` which reply the component is in. An event's `payload` carries what it reports, and the renderer adds which component it came from as `component`. You can also implement `AIUIComponent` on a class of your own, with its `definition` and a `Content` composable. The parameter types are `AIUIParameterType.String`, `Number`, `Integer`, `Boolean`, `Date`, `DateTime` and `Url`, and `enumeration(values)`, `array(items)` and `obj(fields)`. `AIUIParameterType(schema)` takes any other JSON Schema. A component can load what it shows on the device, such as photos or the user's calendar, which then never reaches the agent. `context.openUrl(url)` opens a link through the host, which decides the kinds of link that open. ### One description for every platform A component's definition reads and writes JSON, so every platform your app runs on, and the agent's backend, can share one description of it. Its parameters are a JSON Schema object, as a tool's are, which keeps their order in `x-order`: ```json { "name": "place_card", "description": "A photo and a short summary of a place.", "parameters": { "type": "object", "properties": { "title": { "type": "string", "description": "The place's Wikipedia title." }, "visit_date": { "type": "string", "format": "date" } }, "required": ["title"], "x-order": ["title", "visit_date"] }, "events": [ { "name": "ask_about", "description": "The user asks about the place." } ] } ``` Read it with `AIUIComponentDefinition.fromJsonString(text)`, or `fromJson(map)`, to use the same file in your app. Both return `null` for a definition without a name. `toJson()` and `toJsonString()` write it. ## Run your app's tools A reply's UI can run your app's [client-side tools](https://getstream.io/chat/docs/sdk/android/guides/ai-integrations/client-side-tools/) on the device, such as to load the user's calendar for the days of a plan. Pass them to the host as `tools`. They are apart from the tools the agent calls, and they run without asking the user, so the host runs a tool by itself only when it says it's [read-only](https://getstream.io/chat/docs/sdk/android/guides/ai-integrations/client-side-tools/#read-only-tools): ```kotlin class CalendarEventsTool : AIClientTool { override val name: String = "calendar_events" override val description: String = "The user's calendar events between two days." override val inputSchema: Map = mapOf( "type" to "object", "properties" to mapOf( "from" to mapOf("type" to "string", "format" to "date"), "to" to mapOf("type" to "string", "format" to "date"), ), "required" to listOf("from", "to"), ) override val isReadOnly: Boolean = true override suspend fun run(call: AIToolCallPart): AIClientToolResult { val arguments = call.arguments?.let { runCatching { JSONObject(it) }.getOrNull() } ?: return AIClientToolResult.failed("The days are missing") val events = DeviceCalendar.eventsJson(arguments.optString("from"), arguments.optString("to")) return AIClientToolResult.completed(outputJson = events, summary = "Read the calendar") } } ``` Here `DeviceCalendar` stands for your app's own access to the calendar, and `sendToAgent` for sending an action as shown in [Show it in a conversation](#show-it-in-a-conversation). ```kotlin val ui: AIUIHost = AIUIHost( renderers = listOf(A2UIRenderer(), OpenUIRenderer()), components = listOf(placeCard), tools = listOf(CalendarEventsTool()), ) { action -> sendToAgent(action) } ``` A read-only tool runs whenever the UI asks, such as while it shows. Any other tool runs only when the user taps something that runs it, and fails otherwise. The renderers follow this: an OpenUI `Query` runs only read-only tools, and a `Mutation` runs from a tap, and an A2UI function runs a tool that isn't read-only only from a tap. The model's instructions list the host's tools, with their descriptions and arguments, and say which ones are read-only. A component of yours runs a tool through its context's host, and gets the tool's output, JSON, as plain Kotlin values. `runTool` suspends, so call it from a coroutine, such as a `LaunchedEffect`. Pass `userInitiated = true` when the user asked for it, such as with a tap, and `false` when the component loads data while it shows: ```kotlin var events by remember { mutableStateOf(null) } LaunchedEffect(context.messageId) { events = try { context.host.runTool( name = "calendar_events", arguments = mapOf("from" to "2026-11-14", "to" to "2026-11-15"), userInitiated = false, ) } catch (e: AIUIToolException) { null } } ``` `runTool` throws an `AIUIToolException` when the host has no such tool, the tool isn't read-only and the user didn't ask for it, it fails, or its output isn't JSON. The host's `linkSchemes` are the kinds of link a reply's UI may open: `http`, `https` and `mailto` by default, since the agent writes the links, and another kind could call someone or act in another app. Add another scheme only when your app wants it. ## What the user does When the user does something in a reply's UI that's for the agent, such as tapping a place, choosing a follow-up or submitting a form, the host calls your `send` function with an `AIUIAction`. Send it as the user's next message. | Property | Description | | ----------- | ------------------------------------------------------------------------------------------------------------------------------ | | `messageId` | The reply whose UI it came from. | | `format` | The format of that UI. | | `name` | What happened, such as a component's event (`ask_about`) or an action of the format, such as OpenUI's `continue_conversation`. | | `label` | What the user did, in their words, such as "Tell me more about St. Naum". Use it as the message's text. | | `payload` | The details, in the format's own shape, such as what was chosen and a form's values. | `json` is the action as the JSON object an agent's backend reads. Put it in the message's custom data under the `ai_ui_action` key (`AIUIAction.CUSTOM_DATA_KEY`), as above, or send it to a backend that takes it beside the question. `customData` is the same, as a map. Have your backend tell the model what the user did after their words, and which reply it was in, since the same UI can show in several replies, such as a plan for each of two trips. The SDK's renderers label an action with the control the user tapped, never with text the agent wrote for it, so the conversation shows only what the user chose. OpenUI sends only the values of the form the action belongs to, and never what a password field holds, which it also leaves out of the state it keeps for the reply. What the user types into an A2UI text field that's `obscured` is in none of the action's details. ## Tell the agent what the app shows The model has to know which formats and components your app shows, and which tools its UI can run. The host's `capabilities` describes them, as plain Kotlin values: each format, with what its renderer adds, and your components' definitions: ```kotlin val capabilities: Map = ui.capabilities // {"v": 1, "renderers": [{"format": "a2ui", "fence": ..., "catalog_id": ..., "instructions": ...}, ...], "components": [...]} ``` Each renderer adds the `fence` the model writes its UI in, `ui:a2ui@1.0` or `openui-lang`, and the `instructions` for the model: the `instructions` you gave the renderer, then the format's own, written from its components, yours and the host's tools. A2UI also adds the `catalog_id` of the catalog with your components. Send the capabilities to your backend, for example when a conversation starts, and add the instructions to what the model is told. You can also write the model's instructions on your backend, from the same component definitions. ## Theming The renderers' own components follow your `MaterialTheme`. To restyle one of them, replace its view, with [A2UI's catalog](#a2ui) or [OpenUI's library](#openui). OpenUI's components read their colors, spacing, corner radii and text styles from an `OpenUITheme`, which by default `OpenUITheme.fromMaterial()` makes from the `MaterialTheme`. To change it, provide a theme of your own around the conversation: ```kotlin val material: OpenUITheme = OpenUITheme.fromMaterial() val theme: OpenUITheme = material.copy( colors = material.colors.copy(accent = BrandOrange), radius = OpenUITheme.Radius(small = 4.dp, medium = 6.dp, large = 8.dp), ) ProvideOpenUITheme(theme) { AIConversation(messages = messages, ui = viewModel.ui) } ``` Here `BrandOrange` is a color of your app. The theme applies to the OpenUI components drawn inside `ProvideOpenUITheme`. The renderers' texts, such as form validation messages, are string resources whose names start with `stream_ai_a2ui_` and `stream_ai_openui_`, which you can override or translate in your app's `strings.xml`. ## Errors A reply's UI can't show everything the agent wrote, such as a component the format doesn't know, a message that doesn't read or a tool that failed. Each renderer passes these problems to its `onError`, with the reply's id. A chat has no channel for them, so forward them to your backend if the agent should hear about them, for example to have the model correct its UI. Without `onError`, they are logged: ```kotlin val renderer: OpenUIRenderer = OpenUIRenderer( onError = { error: OpenUIError, messageId: String -> agentBackend.reportUIError(messageId, error.toString()) }, ) ``` Here `agentBackend` stands for your app's connection to the agent's backend. - `A2UIRenderer` passes an `A2UIError`, the problems A2UI reports to the agent: its `code` in A2UI 1.0, such as `VALIDATION_FAILED`, the `message`, written for the agent, and the `surfaceId` and the `path`, a JSON Pointer to the part of the agent's message at fault, when they apply. - `OpenUIRenderer` passes an `OpenUIError` once the reply is complete: its `source` (`Parser`, `Runtime`, `Query` or `Mutation`), a `code`, such as `unknown-component` or `tool-not-found`, the `message`, and, when they apply, the `statementId`, `component`, `path` and `toolName`, a `hint` to fix it, and `isWarning` for a problem the reply still shows despite. ## Show it in your own views To show replies with views of your own rather than with `AIConversation`, show a reply's UI with `AIUIView`, and its text when `showsUI` is false: ```kotlin @Composable fun Reply(message: AIMessage, ui: AIUIHost) { val document = message.uiDocument if (document != null && ui.showsUI(document, message.id)) { AIUIView(host = ui, document = document, messageId = message.id) } else { StreamingText(text = message.text, animate = message.isGenerating) } } ``` `canShow(document)` tells whether one of the host's renderers reads a document, and `showsUI(document, messageId)` whether it also shows something of its own for the reply. Each time your messages change, pass them to `ui.receive(messages)`, oldest first, before the replies show, so the renderers get the UI of every reply, also of those that aren't on screen. `AIConversation` does it for you. ## Motion `AIUIEntrance` brings a part of a reply's UI in when it shows: the part fades in, rising into place and coming into focus, after the parts that came in just before it. Parts that show together come in together, so a reply written before it showed shows at once. With the system's animations off, parts show at once. The SDK's renderers give it to each part they place, and to every component. Use it on the parts of your own renderer's content, as `Cards` does above. Outside a reply's UI, it shows its content as it is. --- For the most recent version of this documentation, visit [https://getstream.io/chat/docs/sdk/android/guides/ai-integrations/generative-ui/](https://getstream.io/chat/docs/sdk/android/guides/ai-integrations/generative-ui/).