# Generative UI An agent can answer with UI rather than text: a weekend plan with a map of the places, a card for each day and a form to change the pace. `AIConversation` shows a reply's UI in place of its text, brings it in as the agent writes it, and sends what the user does in it back to the agent as their next message. The AI components know no UI format of their own. You give a conversation a renderer for each format your agent writes, and the components of your app the agent can place, such as a map, a calendar or a card over your own data. The SDK ships renderers of [A2UI](https://a2ui.org) and [OpenUI](https://github.com/thesysdev/openui), and you can write one for any other format. A component works whichever format the agent writes. > > **Info:** Generative UI is available from version 14.16.0 of `stream-chat-react`. The host and the components are in the `stream-chat-react/ai-components` entry point, and the renderers in `stream-chat-react/a2ui` and `stream-chat-react/openui`. > ## Installation Install `stream-chat-react` 14.16.0 or later, and load the AI components' stylesheet next to the SDK's, as in [Usage](https://getstream.io/chat/docs/sdk/react/guides/ai-integrations/#usage): ```bash npm install stream-chat-react@^14.16.0 stream-chat ``` ```tsx import "stream-chat-react/css/index.css"; import "stream-chat-react/css/ai-components.css"; ``` Each renderer is built on its format's official packages, which are optional peers of `stream-chat-react`. Install the ones for the formats your agent writes. `stream-chat-react` takes the patch releases of the versions it's tested with, since their 0.x releases change their API between minor versions. For [A2UI](#a2ui): ```bash npm install @a2ui/web_core@~0.15.0 @a2ui/react@~0.15.0 ``` For [OpenUI](#openui), and import OpenUI's styles once, which `ai-components.css` themes: ```bash npm install @openuidev/react-lang@~1.0.2 @openuidev/react-ui@~0.18.1 @openuidev/react-headless@~0.18.1 zustand@^4.5.5 ``` ```tsx import "@openuidev/react-ui/components.css"; ``` An app that installs neither still shows every reply: a reply in a format it has no renderer for shows its text. ## How a reply carries its UI The agent's backend writes a reply's UI in the message's `ai_ui` custom field. The reply's text stays readable on its own, for wherever the UI can't show, such as a push notification or an app without a renderer for the format: ```json { "text": "Here's a weekend in Ohrid, with a map and a plan for each day.", "ai_ui": { "v": 1, "format": "openui", "status": "streaming", "content": "root = Card([title, map])\ntitle = CardHeader(\"A weekend in Ohrid\")" } } ``` | Field | Description | | --------- | ----------------------------------------------------------------------------------------------- | | `v` | The version of the field, `1`. An app that doesn't know a newer version shows the reply's text. | | `format` | The format the UI is written in, such as `a2ui` or `openui`. | | `version` | The format's version, such as `1.0`, when the agent names one. Optional. | | `status` | `streaming` while the agent is still writing the UI, then `complete`. | | `content` | The UI in its format: text, such as an OpenUI program or A2UI messages one per line, or JSON. | A model usually writes the UI into its reply, for example in a fenced code block. Have your backend move it out of the text into `ai_ui` as the reply streams, so the text keeps only the words around it. Stream Agents does this for you. A message's custom data holds at most 50 KB, so leave the UI out when it doesn't fit beside the rest of it. `fromStreamMessage` reads the field into the `uiDocument` of the `AIMessage`, for the agent's replies only. A field of a newer `v` than the SDK reads is left out, so the reply shows its text. To read the field yourself, use `parseAIUIDocument(message.ai_ui)`, and type the fields on your messages: ```ts import type { AIUIActionJSON, AIUIDocumentJSON, } from "stream-chat-react/ai-components"; declare module "stream-chat" { interface CustomMessageData { ai_ui?: AIUIDocumentJSON; ai_ui_action?: AIUIActionJSON; } } ``` ## Show it in a conversation Make an `AIUIHost` with the renderers of the formats your agent writes and your components, and pass it to `AIConversation` as `ui`. Keep one host for as long as its conversation shows, for example with `useMemo`, so each reply's UI keeps its state, such as what the user typed into a form. With Stream Agents, the host's `send` asks the agent with the user's words, from `@stream-io/calypso`, and carries the action as `uiAction`. The agents router keeps it on the user's message as `ai_ui_action`: ```tsx import { useMemo } from "react"; import { AIConversation, AIUIHost, aiUIActionJSON, fromStreamMessage, } from "stream-chat-react/ai-components"; import { createA2UIRenderer } from "stream-chat-react/a2ui"; import { createOpenUIRenderer } from "stream-chat-react/openui"; export const Assistant = ({ session, messages }) => { const ui = useMemo( () => new AIUIHost({ components: [placeCard], renderers: [createA2UIRenderer(), createOpenUIRenderer()], send: (action) => session.responses.create(action.label, { uiAction: aiUIActionJSON(action), }), }), [session], ); return ( fromStreamMessage(message))} ui={ui} /> ); }; ``` With a backend of your own, send the action as a message on the channel, with the action in its `ai_ui_action` field: ```tsx import { AI_UI_ACTION_CUSTOM_DATA_KEY, aiUIActionJSON, } from "stream-chat-react/ai-components"; const send = (action) => channel.sendMessage({ text: action.label, [AI_UI_ACTION_CUSTOM_DATA_KEY]: aiUIActionJSON(action), }); ``` A reply shows its UI in place of its text when one of the host's renderers reads its format and version, the first one listed asked first. Otherwise, it shows its text. The UI comes in as the agent writes it: the agent's thinking gives way to it as soon as it starts, and each part fades in, rising into place, after the parts before it. A reply written before it showed, such as one loaded from history, shows at once. [Reply actions](https://getstream.io/chat/docs/sdk/react/guides/ai-integrations/conversation-view/#reply-actions) show under a reply once its UI is written, as they do under text. A reply's UI never takes the conversation down with it: when it can't show, the reply shows its text. See [Handle errors](#handle-errors). ## Renderers A renderer reads one format, and makes the views of each reply's UI written in it. Each one is its own entry point, built on the format's official packages, which your app [installs](#installation). They load in the browser the first time a reply shows the format, so an app whose agent never writes it never loads them, and server rendering shows a placeholder. | Entry point | Renderer | Format | Requires | | -------------------------- | ------------------------ | --------------- | ---------- | | `stream-chat-react/a2ui` | `createA2UIRenderer()` | A2UI 1.0 | React 18 | | `stream-chat-react/openui` | `createOpenUIRenderer()` | OpenUI Lang 1.0 | React 18.3 | Both renderers place your app's [components](#components) beside their own, run your app's [tools](#run-your-apps-tools), and follow the AI components' light and dark [themes](#theming). Both take an `onError(error, messageId?)` option for the problems with a reply's UI, such as a component the format doesn't know. See [Handle errors](#handle-errors). The renderers treat what the agent writes as untrusted: links open through the host, which opens only web and email links by default, and Markdown never renders raw HTML. ### A2UI `createA2UIRenderer` reads [A2UI 1.0](https://a2ui.org) with the basic catalog of the official `@a2ui/web_core` and `@a2ui/react` packages. The entry point is an ES module only. ```tsx import { createA2UIRenderer } from "stream-chat-react/a2ui"; const ui = new AIUIHost({ components: [placeCard, tripMap], renderers: [createA2UIRenderer()], send, }); ``` A reply's `content` is A2UI messages, one JSON object per line, and each one shows as it completes, while the agent writes it. The agent writes with the basic catalog, which your components join under their A2UI type (`place_card` is `PlaceCard`), in the `getstream.io:app-catalog` catalog. A parameter takes the value its schema describes, or a binding (`@path`) or function (`@call`), which your component gets worked out. Each reply's surfaces are its own, kept for as long as the host is, so what the user entered survives scrolling. The parts a user sees most, Button, Text, Card, TextField, ChoicePicker and Icon, draw with this SDK's components and follow its theme. The rest of the basic catalog, its layouts and media, draws with A2UI's own elements, themed with the AI components' tokens. A Modal is a native dialog, which keeps focus while open and closes on Escape or its backdrop. What the agent says about a part with its `accessibility` reaches assistive technology. Text renders with `AIMarkdown`. A button reaches the host as an `AIUIAction` named after its event, labeled with its `userMessage`, else the button's text, with `{ surfaceId, sourceComponentId, context }` as its payload. | Option | Description | | ----------- | ------------------------------------------------------------------------------------------------------------------------------------------ | | `catalogId` | The catalog your components join, `getstream.io:app-catalog` by default. | | `icons` | Your own icon components by A2UI icon name, such as `{ shoppingCart: CartIcon }`. The SDK draws about half of A2UI's names with its icons. | | `iconFont` | An icon font the page loads, such as `'Material Symbols Outlined'`, for the names neither draws. | | `onError` | Gets the problems with a reply's A2UI, with the A2UI error code. | ### OpenUI `createOpenUIRenderer` reads [OpenUI Lang 1.0](https://github.com/thesysdev/openui) with the official OpenUI packages. Import OpenUI's styles once, and `ai-components.css` themes them: ```tsx import "@openuidev/react-ui/components.css"; import { createOpenUIRenderer } from "stream-chat-react/openui"; const ui = new AIUIHost({ components: [placeCard, tripMap], renderers: [createOpenUIRenderer()], send, }); ``` A reply's `content` is an OpenUI Lang program, which shows as the agent writes it. A finished reply with no program, or one that draws nothing, such as words that aren't OpenUI, shows its text. The agent writes with OpenUI's chat library by default, where every reply is one card. Your components join it under their OpenUI name (`place_card` is `PlaceCard`), wherever the library's own content goes, such as a Card's children. What the user chose in a reply's forms is kept by the host, so it survives scrolling. A follow-up or a form's submit button reaches the host as an `AIUIAction` named `continue_conversation`, labeled with the control's words, with `{ context, params, form, values }` as its payload. `values` are the values of its form, without a field OpenUI marks secret. | Option | Description | | -------------- | ---------------------------------------------------------------------------------------------------- | | `library` | The library the agent writes with, before your components join it. OpenUI's chat library by default. | | `toolProvider` | Tools a program's `Query()` and `Mutation()` run, beside the host's `tools`. | | `devtools` | Mounts OpenUI's inspector on the page in development. Off by default. | | `onError` | Gets the problems with a finished reply's program, with OpenUI's error code, once per reply. | `@openuidev/lang-core`, which these packages install, reports the install to OpenUI's analytics anonymously, unless `OPENUI_TELEMETRY_DISABLED=1` or `DO_NOT_TRACK=1` is set. It sends nothing from a browser, and nothing from a server unless `OPENUI_RUNTIME_TELEMETRY_ENABLED=1` is set. ### Write a renderer For any other format, write a renderer of your own. A renderer makes an engine for each conversation, which makes the view of each reply's UI: ```tsx import { AIUIEntrance, type AIUIRenderer, } from "stream-chat-react/ai-components"; const cardsRenderer: AIUIRenderer = { format: "cards", createEngine: () => ({ view: (document) => { const cards = JSON.parse(document.content); if (!Array.isArray(cards)) return null; return (
{cards.map((card, index) => (

{card.title}

 ))}
); }, }), }; ``` List it in the host like any other renderer. - `view(document, messageId)` is asked again each time the conversation renders. Return `null` to show the reply's text instead. Keep what it makes for a reply and return it again. - The host keeps its engines for as long as it's kept, so an engine can hold each reply's state by its `messageId`, or with `host.state(messageId)` and `host.setState(messageId, state)`. - `receive(document, messageId)` gets each reply's document as it arrives, oldest first, including replies that aren't on screen. Use it for a format whose later replies change an earlier one's UI. `remove(messageId)` tells the engine a reply, or its UI, left the conversation. - When what a reply shows changes outside a render, such as once a package loads, call `host.refresh()`, and the conversation shows its replies again. `AIUIRuntimeLoader` and `AIUIRuntimeView` load a format's packages the first time a reply shows it, as the SDK's renderers do. - `supports(version)` says which versions of the format it reads. It reads every version by default. - `capabilities(components)` says what the agent's backend needs to write the format, such as the instructions for its model. It may return a promise. See [Tell the agent what the app shows](#tell-the-agent-what-the-app-shows). To place the app's components, find one with `host.component(name)` and show it with `AIUIComponentView`, giving it the arguments the agent wrote and an `emit` that reports its events the way your format does. ## Components A component is a view of your app the agent can place in a reply's UI. You describe it once, with the parameters the agent fills in and the events it reports, and say how it looks for the arguments the agent wrote: ```tsx import { AIUIParameterType, defineAIUIComponent, optionalAIUIParameter, requiredAIUIParameter, } from "stream-chat-react/ai-components"; type PlaceCardArgs = { title: string; visitDate?: Date }; export const placeCard = defineAIUIComponent( { name: "place_card", description: "A photo and a short summary of a place.", parameters: [ requiredAIUIParameter( "title", AIUIParameterType.string, "The place's Wikipedia title.", ), optionalAIUIParameter("visit_date", AIUIParameterType.date), ], events: [ { name: "ask_about", description: "The user asks about the place.", parameters: [], }, ], }, ({ args, context }) => ( context.emit("ask_about", `Tell me more about ${args.title}`, { title: args.title, }) } /> ), ); ``` `context.emit(event, label, payload?)` reports one of its events: the label is what the user did, in their words, which becomes their message. Every renderer places it by its name, so it works whichever format the agent writes. A component named like one of the format's own is left out and logged. The arguments arrive with camelCase keys, and dates as `Date`s, given as a day, such as `2026-11-14`, or a moment in RFC 3339. While the agent is still writing them and they don't fit the parameters yet, the component waits to show. `context.isStreaming` tells whether they may still change, and `context.messageId` which reply it's in. The parameter types are `string`, `number`, `integer`, `boolean`, `date`, `dateTime`, `url`, `enumeration(values)`, `array(items)` and `object(fields)` on `AIUIParameterType`. A parameter's type can be any other JSON Schema. A component can load what it shows itself, such as photos, which then never reaches the agent. ### One description for every platform A component's definition is JSON, so your web, iOS and Android apps, and the agent's backend, can share one description of it. Its parameters are a JSON Schema object, as a tool's are: ```json { "name": "place_card", "description": "A photo and a short summary of a place.", "parameters": { "type": "object", "properties": { "title": { "type": "string", "description": "The place's Wikipedia title." }, "visit_date": { "type": "string", "format": "date" } }, "required": ["title"], "x-order": ["title", "visit_date"] }, "events": [ { "name": "ask_about", "description": "The user asks about the place." } ] } ``` Read it with `parseAIUIComponentDefinition` to use the same file in your app, and write one back with `aiUIComponentDefinitionJSON`: ```tsx import definitions from "./UIComponents.json"; const placeCard = defineAIUIComponent( parseAIUIComponentDefinition(definitions[0])!, PlaceCardView, ); ``` ## Run your app's tools A reply's UI can run your app's tools in the browser, such as to load the user's saved places for the days of a plan. Pass them to the host as `tools`. They are apart from the tools the agent calls, and they run whenever the UI asks, without asking the user, so give the host only tools that are safe to run that way: ```tsx const ui = new AIUIHost({ renderers, send, tools: [ { name: "saved_places", run: async ({ city }) => loadSavedPlaces(String(city)), }, ], }); ``` An OpenUI program's `Query()` and `Mutation()` run them, and so does an A2UI function the catalog doesn't define. A component of yours runs one through its context's host, and gets the tool's output: ```tsx const places = await context.host.runTool("saved_places", { city: "Ohrid" }); ``` The host's `open` option opens the links of a reply's UI. By default, web and email links open, in a new tab, and others don't, since the agent writes the links, and another kind could call someone or act in another app. ## What the user does When the user does something in a reply's UI that's for the agent, such as tapping a place, choosing a follow-up or submitting a form, the host calls your `send` with an `AIUIAction`. Send it as the user's next message. The host sends a reply's actions one after another, in the order they happened. | Property | Description | | ----------- | ------------------------------------------------------------------------------------------------------------------------------ | | `messageId` | The reply whose UI it came from. | | `format` | The format of that UI. | | `name` | What happened, such as a component's event (`ask_about`) or an action of the format, such as OpenUI's `continue_conversation`. | | `label` | What the user did, in their words, such as "Tell me more about St. Naum". Use it as the message's text. | | `payload` | The details, in the format's own shape, such as what was chosen and a form's values. A component's event adds `component`. | `aiUIActionJSON(action)` is the action as the message's `ai_ui_action` field, or as Stream Agents' `uiAction`. Have your backend tell the model what the user did after their words, and which reply it was in, since the same UI can show in several replies. An action the agents router would refuse, one without a reply, format or name, or over 8 KB as JSON, isn't sent. It goes to the host's `onError(error, action)`, as does one that fails to send, and neither holds up the actions after it. See [Handle errors](#handle-errors). ## Tell the agent what the app shows The model has to know which formats and components your app shows. The host's `capabilities()` describes them: each format, with what its renderer adds, and your components' definitions. It resolves once the renderers have written their instructions, which loads their packages: ```tsx const capabilities = await ui.capabilities(); // { v: 1, renderers: [{ format: "a2ui", fence, catalog_id, instructions }, ...], components: [...] } ``` Each renderer adds the `fence` the model writes its UI in and the `instructions` for the model, written from the format's components and yours. A2UI also adds the `catalog_id` of the catalog with your components. Send the capabilities to your backend, for example when a conversation starts, and add the instructions to what the model is told. You can also write the instructions on your backend or at build time, from the same component definitions: - `a2uiInstructions(config)`, from `stream-chat-react/a2ui`, writes A2UI's instructions with your components, examples and rules. It rejects with an `A2UIInstructionsError` when an example wouldn't render, since a broken example teaches the model broken UI. `a2uiProblems(text, config)` lists what's wrong with A2UI messages for the catalog, for checking what a model writes. - `openUIInstructions(components)`, from `stream-chat-react/openui`, writes OpenUI's instructions with your components, with OpenUI's own prompt generator. ```tsx import { a2uiInstructions } from "stream-chat-react/a2ui"; import { openUIInstructions } from "stream-chat-react/openui"; const a2ui = await a2uiInstructions({ components: [placeCard, tripMap] }); const openui = await openUIInstructions([placeCard, tripMap]); ``` Both load their format's packages, so ask for them where that's fine, such as once at build time or when a conversation starts. ## Handle errors A reply's UI never takes the conversation down with it. The host and the renderers each take a callback for what goes wrong, and log it by default: ```tsx const ui = new AIUIHost({ components: [placeCard], renderers: [ createA2UIRenderer({ onError: (error, messageId) => report("a2ui", error, messageId), }), createOpenUIRenderer({ onError: (error, messageId) => report("openui", error, messageId), }), ], send, onError: (error, action) => setUnsent(`"${action.label}" wasn't sent.`), onRenderError: (error, messageId) => report("render", error, messageId), }); ``` | Callback | Gets | | -------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Renderer `onError(error, messageId?)` | The problems with a reply's UI, such as a message that isn't JSON or a component the format doesn't know, with the format's error code and the reply's id. The rest of the reply still shows. OpenUI reports a finished reply's problems once, and nothing while the agent still writes it. A problem with the renderer rather than a reply, such as packages that couldn't load, comes without an id. | | Host `onRenderError(error, messageId)` | A reply's UI that threw while it rendered, once the reply is written. | | Host `onError(error, action)` | An action that wasn't sent: one the agents router would refuse, or one your `send` failed to send, by throwing or rejecting. The actions after it are still sent. | A reply shows its text in place of its UI when: - No renderer reads its format or version, or its `ai_ui` field is of a newer `v` than the SDK reads. - The renderer's packages can't load, such as after a deploy replaced the file they are in. The next reply the agent writes in that format tries again. - A finished reply draws nothing, such as an A2UI reply that made no surface or a program with no UI in it. - Its UI throws while it renders, outside your components. A reply the agent is still writing tries again once it's written. One of your components that throws, such as when given what it didn't expect, leaves only itself out of the reply's UI, and is tried again when what it's given changes. A chat has no channel for these problems, so forward them to your backend if the agent should hear about them. To tell the user that what they did wasn't sent, throw from `send`, as when the agent is still answering, and show the error from `onError` where the composer shows a failed question. ## Theming A reply's UI follows the AI components' theme, and with it the `stream-chat-react` light and dark themes: inside `.str-chat`, the AI components read the SDK's theme tokens. To change its colors, set the `--str-chat__ai-*` variables described in [Theming](https://getstream.io/chat/docs/sdk/react/guides/ai-integrations/#theming). The ones a reply's UI reads most are: | Variable (`--str-chat__ai-...`) | Used for | | ------------------------------------------------- | ------------------------------------- | | `accent` | Links, chosen chips and follow-ups | | `card-bg`, `card-border` | The reply's card and the panels in it | | `text-primary`, `text-secondary`, `text-tertiary` | Text | | `border`, `border-subtle` | Fields and hairlines | | `button-primary-bg`, `button-primary-text` | Primary buttons | | `button-secondary-bg`, `button-secondary-text` | Secondary buttons | | `surface-hover`, `elevated-bg` | Hovered controls, menus and popovers | ```css :root { --str-chat__ai-accent: #6d28d9; --str-chat__ai-button-primary-bg: #6d28d9; } ``` Each format's own variables are set from these on the element its renderer draws in: `--a2ui-*` on `.str-chat__a2ui`, and `--openui-*` on `.str-chat__openui`. To change one format only, set its variables there: ```css .str-chat__openui { --openui-font-body: "Inter", sans-serif; } ``` The rest of OpenUI's look, such as its chart palette, stays its own. A2UI's `Icon` draws the SDK's icons, which you can replace like any SDK icon with the `icons` of `WithComponents`, and the renderer's `icons` and `iconFont` options draw the names the SDK doesn't. ## Show it in your own views To show replies in views of your own rather than with `AIConversation`, show a reply's UI with `AIUIView`, and its text when the host shows nothing for it: ```tsx import { AIUIView, StreamingMessage, useAIUIHostRefresh, } from "stream-chat-react/ai-components"; const Reply = ({ message, ui }) => { useAIUIHostRefresh(ui); const document = message.uiDocument; return document && ui.content(document, message.id) !== null ? ( ) : ( ); }; ``` Each time your messages change, pass them to `ui.receive(messages)` before the replies render, so the renderers get the UI of every reply, also of those that aren't on screen. `useAIUIHostRefresh` renders again when the host's answer changes, such as once a renderer's packages load. ## Motion `AIUIEntrance` brings a part of a reply's UI in when it shows: the part fades in, rising into place and coming into focus, after the parts that came in just before it, 0.1 seconds apart and at most 0.4 seconds later. Parts that show together come in together, so a reply written before it showed shows at once. With reduced motion, parts fade in without moving. The SDK's renderers give it to each part they place, and every component has it. Use it on the parts of your own renderer's views, as the cards renderer does above. --- For the most recent version of this documentation, visit [https://getstream.io/chat/docs/sdk/react/guides/ai-integrations/generative-ui/](https://getstream.io/chat/docs/sdk/react/guides/ai-integrations/generative-ui/).