Generative UI

An agent can answer with UI rather than text: a weekend plan with a map of the places, the user's own calendar for those days and a form to change the pace. The AI components show a reply's UI in place of its text, bring it in as the agent writes it, and send what the user does in it back to the agent as their next message.

The AI components know no UI format of their own. You give a conversation a renderer for each format your agent writes, and the widgets of your app the agent can place, such as a map, a calendar or a card over your own data. The SDK ships a renderer of A2UI, and you can write one for any other format. A component works whichever format the agent writes.

How a reply carries its UI

The agent's backend writes a reply's UI in the message's ai_ui custom field. The reply's text stays readable on its own, for wherever the UI can't show, such as a push notification or an app without a renderer for the format:

{
  "text": "Here's a weekend in Ohrid, with a map and a plan for each day.",
  "ai_ui": {
    "v": 1,
    "format": "a2ui",
    "status": "streaming",
    "content": "{\"version\":\"1.0\",\"createSurface\":{\"surfaceId\":\"plan\",\"catalogId\":\"getstream.io:app-catalog\"}}\n{\"version\":\"1.0\",\"updateComponents\":{\"surfaceId\":\"plan\",\"components\":[{\"id\":\"root\",\"component\":\"Text\",\"text\":\"A weekend in Ohrid\"}]}}"
  }
}

A model usually writes the UI into its reply, for example in a fenced code block. Have your backend move it out of the text into ai_ui as the reply streams, so the text keeps only the words around it. A message's custom data holds at most 50 KB in a channel of the agent type, and 5 KB in the other types, so leave the UI out when it doesn't fit beside the rest of it.

Read the field with AIUIDocument.fromCustomData, as the uiDocument of your AIMessage, such as the StreamAIMessage from Messages from Stream Chat:

class StreamAIMessage with AIMessage {
  StreamAIMessage(this.message) : uiDocument = AIUIDocument.fromCustomData(message.extraData);

  final Message message;

  // The id, role, text and the rest, as in the Conversation View guide.

  @override
  final AIUIDocument? uiDocument;
}

It returns null when the message carries no ai_ui field the SDK can read, and the reply shows its text.

Show it in a conversation

Make an AIUIHost with the renderers of the formats your agent writes and your components, and pass it to AIConversationView as ui. Keep one host for as long as its conversation shows, for example in the State behind your screen, so each reply's UI keeps its state, such as what the user typed into a form, across scrolling and rebuilds. Call dispose when the conversation closes:

class _AssistantScreenState extends State<AssistantScreen> {
  late final AIUIHost _ui = AIUIHost(
    renderers: [A2UIRenderer()],
    components: const [PlaceCardComponent()],
    send: (action) => widget.channel.sendMessage(
      Message(text: action.label, extraData: action.toCustomData()),
    ),
  );

  @override
  void dispose() {
    _ui.dispose();
    super.dispose();
  }

  @override
  Widget build(BuildContext context) {
    return StreamBuilder<List<Message>>(
      stream: widget.channel.state!.messagesStream,
      initialData: widget.channel.state!.messages,
      builder: (context, snapshot) => AIConversationView(
        messages: [
          for (final message in snapshot.data!)
            if (message.type == MessageType.regular) StreamAIMessage(message),
        ],
        ui: _ui,
      ),
    );
  }
}

PlaceCardComponent is one of your components. A reply shows its UI in place of its text, after its steps, when one of the host's renderers reads its format and version, the first one listed asked first, and builds something for it. Otherwise, it shows its text. The UI comes in as the agent writes it: each part fades in, rising into place, after the parts before it. A reply written before it showed, such as one loaded from history, shows at once.

A reply with UI the host can show counts as showing something, so the conversation stops showing the agent thinking as soon as its UI starts. The reply actions show under a reply once its UI is written, as they do under text. Their message's text is still the reply's text, which the reply doesn't show while it shows its UI. A reply that only shows UI and has no text offers no copying or sharing.

With an AIConversation, pass the host the same way:

AIConversationView.conversation(conversation: conversation, ui: ui)

Renderers

A renderer reads one format, and shows each reply's UI written in it. Add the package of each format your agent writes next to stream_chat_flutter_ai, which each depends on:

dependencies:
  stream_chat_flutter_ai: ^0.2.0
  stream_chat_flutter_ai_a2ui: ^0.2.0

It requires Dart 3.12 or later and Flutter 3.44 or later, like the AI components.

The SDK's renderers place your app's components beside their own, run your app's tools, and style their widgets with the app's StreamTheme, so a reply's UI looks like the rest of the conversation. They take these options:

A2UIRenderer also takes a catalog and a catalogId, as described below.

The renderers treat what the agent writes as untrusted: links in a reply's UI open through the host, which opens only web and email links by default, and images load only from the web.

A2UI

A2UIRenderer reads A2UI 1.0, with every component and function of its basic catalog drawn in Flutter. The agent writes with the basic catalog, which your components join under the getstream.io:app-catalog id (catalogId changes it). A surface that names the basic catalog's id shows them as well. Each reply has surfaces of its own, kept while the host is, so what the user enters in them survives scrolling.

A button reaches the host as an AIUIAction named after its event, labeled with the button's text, or else with the event's name as words (submit_form is "Submit form"). Its payload holds the surfaceId, the sourceComponentId, the event's resolved context, the event's userMessage when it has one, and the surface's dataModel when the agent created the surface with sendDataModel. A function the catalog doesn't define runs as one of the host's tools, with the tool's output as the function's value.

By default, only web links load as images, video and audio. Pass mediaUrl to return the URL an image, video or audio clip loads from, given the one the agent wrote, or null to not load it, such as to load media through your own CDN. The agent writes these URLs, so return only ones your app trusts:

final renderer = A2UIRenderer(
  mediaUrl: (url) => url.scheme == 'https' && url.host == 'media.example.com' ? url : null,
);

To change how one of the basic catalog's components looks, give the renderer a catalog in which your widget draws it. The builder reads the component through an A2UIComponentScope, with its bindings resolved, and draws its children with the same catalog:

// In a `static final` field: a catalog made in `build` is a new one each time,
// so everything drawn with it is built again.
static final catalog = A2UICatalog.basic.replacing('Card', (context, scope) {
  final child = scope.child();
  return BrandCard(child: child == null ? const SizedBox.shrink() : scope.build(child));
});

final renderer = A2UIRenderer(catalog: catalog);

Here BrandCard is a widget of your app.

The scope reads properties with value, string, number, boolean and stringList, writes an input's value to the surface's data model with write, and runs what the component does with performAction. A builder draws with the surface's theme through A2UITheme.of(context), whose fields are null where nothing overrides them, so fall back to the StreamTheme, such as A2UITheme.of(context).border ?? context.streamColorScheme.borderDefault. A Row or Column measures its children before laying them out, so a child can't be a widget that can't measure itself, such as a LayoutBuilder.

removing leaves out components you don't want the agent to use. The agent isn't told about them, and a reply that still uses one shows nothing for it:

final renderer = A2UIRenderer(catalog: A2UICatalog.basic.removing(['Video', 'AudioPlayer']));

Write a renderer

For any other format, write a renderer of your own. A renderer makes an engine for each conversation's host, which makes the widget of each reply's UI:

class CardsRenderer extends AIUIRenderer {
  const CardsRenderer();

  @override
  AIUIFormat get format => const AIUIFormat('cards');

  @override
  AIUIEngine createEngine(AIUIHosting host) => CardsEngine();
}

class CardsEngine extends AIUIEngine {
  @override
  Widget? view(AIUIDocument document, String messageId) {
    final cards = document.content;
    if (cards is! List<Object?>) return null;
    return Cards(cards: cards);
  }
}

class Cards extends StatelessWidget {
  const Cards({super.key, required this.cards});

  final List<Object?> cards;

  @override
  Widget build(BuildContext context) {
    return Column(
      crossAxisAlignment: CrossAxisAlignment.start,
      children: [
        for (final card in cards)
          AIUIEntrance(child: Text(card is Map<String, Object?> ? '${card['title'] ?? ''}' : '')),
      ],
    );
  }
}

List it in the host like any other renderer. A document's content is the agent's text, which text reads, or JSON as plain Dart values: maps, lists, strings, numbers, booleans and null.

• view(document, messageId) is called for each new version of a reply's document, such as while the agent writes it. Return null to show the reply's text instead. It isn't called again when other replies change, so a widget that other replies can update should listen to the engine's own state, such as a Listenable.
• The host makes one engine per renderer and keeps it until AIUIHost.dispose, so an engine can hold each reply's state by its messageId. The host reports what an engine throws to FlutterError.onError, and the reply shows its text.
• receive(document, messageId) gets each reply's document as it arrives or changes, oldest first, including replies that aren't on screen. Use it for a format whose later replies change an earlier one's UI. remove(messageId) tells the engine a reply, or its UI, left the conversation, so it can let go of what it kept for it. dispose() releases the engine's resources.
• supports(version) on the renderer says which versions of the format it reads. It reads every version by default.
• capabilities(components, tools) on the renderer says what the agent's backend needs to write the format, such as the instructions for its model. See Tell the agent what the app shows.

Extend AIUIRenderer and AIUIEngine rather than implementing them, so members added in later versions don't break your code.

A renderer places the app's components by name, with host.component(name), among the ones its format has. A component's view(arguments, context) makes its widget for the arguments the agent wrote, or returns null while they don't fit its parameters yet. Give it an AIUIComponentContext whose emit sends its events the way your format reports what the user did.

Components

A component is a widget of your app the agent can place in a reply's UI. You describe it once, with the parameters the agent fills in and the events it reports, say how to read the arguments the agent wrote, and how it looks for them:

class PlaceCardComponent extends AIUIComponent<({String title, DateTime? visitDate})> {
  const PlaceCardComponent();

  @override
  AIUIComponentDefinition get definition => const AIUIComponentDefinition(
    name: 'place_card',
    description: 'A photo and a short summary of a place.',
    parameters: [
      AIUIParameter.required('title', AIUIParameterType.string, description: "The place's Wikipedia title."),
      AIUIParameter.optional('visit_date', AIUIParameterType.date),
    ],
    events: [AIUIComponentEvent(name: 'ask_about', description: 'The user asks about the place.')],
  );

  @override
  ({String title, DateTime? visitDate}) readArguments(AIUIArguments arguments) => (
    title: arguments.string('title'),
    visitDate: arguments.optional('visit_date', arguments.date),
  );

  @override
  Widget build(BuildContext context, ({String title, DateTime? visitDate}) arguments, AIUIComponentContext ui) {
    return PlaceCard(
      title: arguments.title,
      date: arguments.visitDate,
      onTap: () => ui.emit('ask_about', label: 'Tell me more about ${arguments.title}'),
    );
  }
}

PlaceCard is a widget of your app. Every renderer places it by its name, so it works whichever format the agent writes. The SDK's renderers place it under its name in PascalCase, so place_card is PlaceCard. A component named like one of the format's own, such as Card in A2UI, would replace it, so the renderer leaves it out and reports it in debug builds. A2UI also leaves out a component with a parameter named id, component, weight or accessibility, or whose name isn't an identifier, such as visit-date, and an app component holds no other components and no action there.

The arguments are the JSON the agent wrote, which readArguments reads through the getters of AIUIArguments, such as string, number, integer, boolean, date (a day or a date and time), url, object, their lists (strings, numbers, integers, booleans, dates, urls and objects) and optional. A getter throws a FormatException when a value is missing or doesn't fit, and the component shows nothing, as it does while the agent is still writing the arguments. A date arrives as the text the agent wrote, such as 2026-11-14, which date reads. ui.isStreaming tells whether the arguments may still change, and ui.messageId which reply the component is in. An event's payload carries what it reports, and the renderer adds which component it came from as component, and A2UI also surfaceId and sourceComponentId. An event can describe what it carries with parameters, as a component does.

Extend AIUIComponent rather than implementing it, so members added in later versions don't break your code.

The parameter types are AIUIParameterType.string, number, integer, boolean, date, dateTime and url, and AIUIParameterType.enumeration(values), array(items) and object(fields). Those three aren't constants, so a definition that uses one isn't const. AIUIParameterType(schema) takes any other JSON Schema.

A component can load what it shows on the device, such as photos or the user's calendar, which then never reaches the agent. ui.host.open(url) opens a link through the host, which decides the kinds of link that open.

One description for every platform

A component's definition reads and writes JSON, so every platform your app runs on, and the agent's backend, can share one description of it. Its parameters are a JSON Schema object, as a tool's are, which keeps their order in x-order:

{
  "name": "place_card",
  "description": "A photo and a short summary of a place.",
  "parameters": {
    "type": "object",
    "properties": {
      "title": {
        "type": "string",
        "description": "The place's Wikipedia title."
      },
      "visit_date": { "type": "string", "format": "date" }
    },
    "required": ["title"],
    "x-order": ["title", "visit_date"]
  },
  "events": [
    { "name": "ask_about", "description": "The user asks about the place." }
  ]
}

Read it with AIUIComponentDefinition.fromJson to use the same file in your app. It throws a FormatException for a definition without a name. toJson() writes it:

final json = jsonDecode(await rootBundle.loadString('assets/place_card.json')) as Map<String, Object?>;
final definition = AIUIComponentDefinition.fromJson(json);

Run your app's tools

A reply's UI can run your app's tools on the device, such as to load the user's calendar for the days of a plan. Pass them to the host as tools, as AIResultClientTools; when two share a name, the first is used. They are apart from the tools the agent calls, and they run without asking the user, so the host runs a tool by itself only when its definition says it's read-only, with isReadOnly:

class CalendarEventsTool extends AIResultClientTool {
  const CalendarEventsTool();

  @override
  AIToolDefinition get definition => const AIToolDefinition(
    name: 'calendar_events',
    description: "The user's calendar events between two days.",
    parameters: {
      'type': 'object',
      'properties': {
        'from': {'type': 'string', 'format': 'date'},
        'to': {'type': 'string', 'format': 'date'},
      },
      'required': ['from', 'to'],
    },
    isReadOnly: true,
  );

  @override
  Future<AIClientToolResult> run(AIToolCallPart call) async {
    final from = call.arguments?['from'];
    final to = call.arguments?['to'];
    if (from is! String || to is! String) return const AIClientToolResult.failed('The days are missing');
    final events = await DeviceCalendar.eventsJson(from, to);
    return AIClientToolResult.completed(events, summary: 'Read the calendar');
  }
}

Here DeviceCalendar stands for your app's own access to the calendar, and sendToAgent for sending an action as shown in Show it in a conversation.

final ui = AIUIHost(
  renderers: [A2UIRenderer()],
  components: const [PlaceCardComponent()],
  tools: const [CalendarEventsTool()],
  send: sendToAgent,
);

A read-only tool runs whenever the UI asks, such as while it shows. Any other tool runs only when the user taps something that runs it, and fails otherwise. The renderers follow this: an A2UI function runs a tool that isn't read-only only from a tap. The model's instructions list the host's tools, with their descriptions and arguments, and say which ones are read-only.

A component of yours runs a tool through its context's host, and gets the tool's output, JSON, as plain Dart values. runTool is asynchronous, so call it from a State, such as in initState. Pass userInitiated: true when the user asked for it, such as with a tap, and false when the component loads data while it shows:

class _CalendarStripState extends State<CalendarStrip> {
  late final Future<Object?> _events;

  @override
  void initState() {
    super.initState();
    _events = _load();
  }

  Future<Object?> _load() async {
    try {
      return await widget.ui.host.runTool(
        'calendar_events',
        arguments: {'from': '2026-11-14', 'to': '2026-11-15'},
        userInitiated: false,
      );
    } on AIUIToolException {
      return null;
    }
  }

  @override
  Widget build(BuildContext context) {
    return FutureBuilder<Object?>(
      future: _events,
      builder: (context, snapshot) => EventsList(events: snapshot.data),
    );
  }
}

Here CalendarStrip takes the AIUIComponentContext its component's build got, as ui, and EventsList is a widget of your app. runTool throws an AIUIToolException when the host has no such tool, the tool isn't read-only and the user didn't ask for it, or it fails.

The host's open callback opens the links of a reply's UI. By default, web and email links open and others don't, since the agent writes the links, and another kind could call someone or act in another app. Pass your own open only when your app wants another kind of link.

What the user does

When the user does something in a reply's UI that's for the agent, such as tapping a place, choosing a follow-up or submitting a form, the host calls your send function with an AIUIAction. Send it as the user's next message. The host sends a reply's actions one after another, in the order they happened.

toCustomData() holds the action under the ai_ui_action key (AIUIAction.customDataKey), for the message's custom data, as above. toJson() is the action alone, for a backend that takes it beside the question. Have your backend tell the model what the user did after their words, and which reply it was in, since the same UI can show in several replies, such as a plan for each of two trips.

The SDK's renderers label an action with the control the user tapped, never with text the agent wrote for it, so the conversation shows only what the user chose. An A2UI text field that's obscured only hides its text on screen. What the user types into it is data like any other, so it reaches the agent when the action asks for it, but it never becomes a button's label.

Tell the agent what the app shows

The model has to know which formats and components your app shows, and which tools its UI can run. The host's capabilities describes them, as plain Dart values: each format, with what its renderer adds, and your components' definitions:

final capabilities = ui.capabilities;
// {"v": 1, "renderers": [{"format": "a2ui", "fence": ..., "catalog_id": ..., "instructions": ...}, ...], "components": [...]}

Each renderer adds the fence the model writes its UI in, such as ui:a2ui@1.0, and the instructions for the model: the instructions you gave the renderer, then the format's own, written from its components, yours and the host's tools. A2UI also adds the catalog_id of the catalog with your components.

Send the capabilities to your backend, for example when a conversation starts, and add the instructions to what the model is told. You can also write the model's instructions on your backend, from the same component definitions.

Write the instructions from a test

An agent on Stream Agents is told what to do by the instructions.md in its folder, which you sync with the Stream CLI, so the app can't send it the capabilities. Write the instructions into a file from a test of your app instead. Run it with flutter test from your app's folder, which holds the agent's, so the file lands in your project:

import 'dart:io';

import 'package:flutter_test/flutter_test.dart';
import 'package:stream_chat_flutter_ai/stream_chat_flutter_ai.dart';
import 'package:stream_chat_flutter_ai_a2ui/stream_chat_flutter_ai_a2ui.dart';
import 'package:your_app/place_card_component.dart';

void main() {
  // Writes what the agent is told about the app's UI into the agent's folder.
  test('writes the UI instructions', () {
    // The same components and renderers as the app's host.
    final ui = AIUIHost(
      components: const [PlaceCardComponent()],
      renderers: const [
        A2UIRenderer(
          instructions:
              'Write a trip plan as UI, with a place card for each place. Answer everything else in Markdown.',
        ),
      ],
      send: (_) async {},
    );
    addTearDown(ui.dispose);
    final renderers = ui.capabilities['renderers']! as List<Object?>;
    final instructions = [
      for (final renderer in renderers.cast<Map<String, Object?>>())
        if (renderer['instructions'] case final String text) text,
    ].join('\n\n');

    File('agents/my-agent/ui-instructions.md').writeAsStringSync(instructions);
  });
}

Add the file's text to the agent's instructions.md, after your own, and sync the agent with getstream agents sync my-agent. Run the test again whenever a component changes. Build the host in one function that the app and the test both call, so the two never drift apart.

A2UI names your components in its own style, PlaceCard for place_card, so use that name when your renderer's instructions mention one.

The instructions describe every component of the renderer's catalog, with its properties, so with the whole basic catalog they run to tens of kilobytes, which the model reads on every turn. Give the renderer a catalog of only the components your agent uses, with removing as in A2UI, in the app and in the test alike.

Theming

The renderers' own components follow the app's StreamTheme, in light and dark mode. To restyle one of them, replace its builder, with A2UI's catalog.

A2UI's components read their colors, corner radii and text styles from an A2UIThemeData. Every field is nullable, and one left unset comes from the StreamTheme. To change it, give the renderer a theme:

final a2ui = A2UIRenderer(
  theme: A2UIThemeData(accent: brandOrange, cornerRadius: 8),
);

Here brandOrange is a color of your app. To theme the replies of a whole subtree instead, wrap it in an A2UITheme. The renderer's theme is layered over the nearest one, and A2UITheme.merge adds to the enclosing theme rather than replace it.

The renderers' texts that have no text of their own in the reply, such as the buttons that play and pause a video or an audio clip, are members of AITranslations (playMedia and pauseMedia), which you translate as described in Localization.

Buttons, check boxes, fields and images in a reply are the Stream design system's own, so the builders your app registers with StreamComponentFactory draw them there too.

Errors

A reply's UI can't show everything the agent wrote, such as a component the format doesn't know, a message that doesn't read or a tool that failed. Each renderer passes these problems to its onError, with the reply's id. A chat has no channel for them, so forward them to your backend if the agent should hear about them, for example to have the model correct its UI. Without onError, they are reported in debug builds only:

final renderer = A2UIRenderer(
  onError: (A2UIError error, String messageId) {
    agentBackend.reportUIError(messageId, error.toString());
  },
);

Here agentBackend stands for your app's connection to the agent's backend.

• A2UIRenderer passes an A2UIError, the problems A2UI reports to the agent: its code in A2UI 1.0, such as A2UIErrorCode.validationFailed, the message, written for the developer of the agent, and the surfaceId and the path, a JSON Pointer to the part of the agent's message at fault, when they apply.

Show it in your own views

To show replies with views of your own rather than with AIConversationView, show a reply's UI with AIUIView, below an AIAssistantMessageView whose text is empty, and the reply's text when canShow is false:

Widget reply(AIMessage message, AIUIHost ui) {
  final document = message.uiDocument;
  final view = document == null ? null : AIUIView(host: ui, document: document, messageId: message.id);
  final showsUI = view != null && view.canShow;
  return Column(
    children: [
      AIAssistantMessageView(text: showsUI ? '' : message.text, parts: message.parts),
      if (view != null && showsUI) view,
    ],
  );
}

AIUIHost.canShow(document) tells whether one of the host's renderers reads a document's format and version, and AIUIView.canShow whether it also built something to show for the reply. Each time your messages change, pass them to ui.receive(messages), oldest first, before the replies show, so the renderers get the UI of every reply, also of those that aren't on screen. AIConversationView does it for you.

Motion

AIUIEntrance brings a part of a reply's UI in when it shows: the part fades in, rising into place, growing to full size and coming into focus, after the parts that came in just before it. Parts that show together come in together, so a reply written before it showed shows at once. With the system's animations off, parts only fade in.

The SDK's renderers give it to each part they place, and to every component. Use it on the parts of your own renderer's widgets, as Cards does above. Outside a reply's UI, it shows its child as it is.