Client-Side Tools

The AI components let the agent run tools on the user's device. When the agent decides a tool should be called, it can hand that call to the app instead of running it on the backend.

This lets your app use the device's data (calendar, location, health data and so on), show its own UI, such as screens, forms or dialogs, or start other apps' actions with intents.

Running client tools takes three steps:

1. Implement AIClientTool for each tool.
2. Give your tools to an AIClientToolRunner, and register them with your backend, so the agent knows it can call them.
3. Run the calls the agent sends to the device. An agent that writes its tool calls on the reply as ai_tool_call steps reaches the runner directly. The Stream Chat AI SDK and the Stream Chat LangChain SDK send a custom_client_tool_invocation event instead.

Create your own client tool

A client tool has a name, the one the agent declared it with, and a run function:

public interface AIClientTool {
    public val name: String

    public suspend fun run(call: AIToolCallPart): AIClientToolResult
}

When the agent calls the tool, run does the work and returns an AIClientToolResult. If run throws, the runner reports the call as failed ("The tool failed on this device."), so the agent always gets an answer.

Greet tool

This example tool greets the user when they ask to be greeted. It publishes the greeting as state, which your UI shows, for example as an AlertDialog:

class GreetTool : AIClientTool {

    private val _greeting = MutableStateFlow<String?>(null)

    // Show it in your UI, for example as an AlertDialog, and reset it when dismissed.
    val greeting: StateFlow<String?> = _greeting.asStateFlow()

    override val name: String = "greetUser"

    override suspend fun run(call: AIToolCallPart): AIClientToolResult {
        _greeting.value = "Hello there! The assistant asked me to greet you."
        return AIClientToolResult.completed(
            outputJson = """{"greeted":true}""",
            summary = "Greeted the user",
        )
    }

    fun dismiss() {
        _greeting.value = null
    }
}

run runs in the runner's scope. With MainScope(), that's the main thread, so the tool can update your UI state directly. Different tools can expose other state the same way, such as banners or navigation triggers.

Tool results

AIClientToolResult carries the output for the model (a JSON object, as text), a short summary of the outcome, or a failure, such as "Location not shared". Create one with AIClientToolResult.completed(outputJson, summary) or with AIClientToolResult.failed(reason).

For tools that take arguments, read the call's arguments, the JSON text of the object the agent sent:

class OpenScreenTool : AIClientTool {

    override val name: String = "openScreen"

    override suspend fun run(call: AIToolCallPart): AIClientToolResult {
        val arguments = call.arguments?.let { runCatching { JSONObject(it) }.getOrNull() }
        val screen = arguments?.optString("screen").orEmpty()
        if (screen.isEmpty()) {
            return AIClientToolResult.failed("The screen to open is missing")
        }
        AppRouter.open(screen)
        return AIClientToolResult.completed(
            outputJson = JSONObject().put("opened", screen).toString(),
            summary = "Opened $screen",
        )
    }
}

Arguments and summaries can be visible to every channel member, so keep private data out of them: a summary like "Shared approximate location" rather than the coordinates.

Register the tools

Give your tools to an AIClientToolRunner. It knows who is signed in and which install it runs on, so it only runs the calls addressed to this device. Create one runner when the user signs in, in an object that lives as long as they stay signed in, and hand it to the view models that need it:

// Created when the user signs in, and closed when they sign out.
class SignedInSession(userId: String, clientId: String) {

    private val scope = MainScope()

    val greetTool = GreetTool()

    val toolRunner = AIClientToolRunner(
        userId = userId,
        clientId = clientId,
        tools = listOf(greetTool, OpenScreenTool()),
        scope = scope,
    )

    fun close() {
        scope.cancel()
    }
}

class ToolsViewModel(
    private val runner: AIClientToolRunner,
    private val backend: AgentBackend,
) : ViewModel()

clientId is AIClientIdentity.installId(context), a stable identifier for this install, created on first use and kept until the app's data is cleared. It isn't backed up, so a device restored from a backup gets its own. It reads and writes a small file, so call it off the main thread.

Keep the runner while the user is signed in: it remembers which calls it already ran, so a new runner could run a call again. The runner is confined to the main thread, so pass a scope that runs on it, such as MainScope(). When two tools share a name, the first is used.

Registering the tools server-side

Your agent also needs to know about the tools: their names, what they do, and the JSON schema of their input. The runner only knows the names (runner.toolNames), so keep each tool's description and input schema next to it, and send them to your backend when the conversation opens, in whatever format your backend registers tools with. For example, the greet and open-screen tools above could register as:

{
  "channel_id": "ai-assistant-123",
  "tools": [
    {
      "name": "greetUser",
      "description": "Display a native greeting to the user",
      "instructions": "Use the greetUser tool when the user asks to be greeted.",
      "parameters": { "type": "object", "properties": {} }
    },
    {
      "name": "openScreen",
      "description": "Open a screen of the app",
      "parameters": {
        "type": "object",
        "properties": {
          "screen": { "type": "string", "enum": ["settings", "profile"] }
        },
        "required": ["screen"]
      }
    }
  ]
}

Run tool calls from agent steps

An agent can write each tool call on its reply, as an ai_tool_call step (see Reasoning and Tool Calls). A call with executor: client and status: awaiting_client asks a user's device to run a tool. It names the user (target_user_id) and the install (target_client_id) that should run it, which your backend copies from the custom data (client_id) of that user's message. Send the install's identifier with every message:

val clientId = withContext(Dispatchers.IO) { AIClientIdentity.installId(context) }
ChatClient.instance().channel(cid)
    .sendMessage(Message(text = text, extraData = mapOf("client_id" to clientId)))
    .enqueue()

Then hand the runner each AI reply's steps whenever they change, on the main thread:

// In ToolsViewModel
fun onReplyChanged(message: Message) {
    runner.run(message.aiParts()) { call, result ->
        backend.sendToolResult(messageId = message.id, toolCallId = call.id, result = result)
    }
}

aiParts() decodes the reply's steps, as shown in Show the steps of a reply.

The runner runs a call only when the call has its own id, awaits this user and this install, and a tool has its name, and it runs each call only once. Your send function delivers the result to your backend, which updates the step and gives the output to the model; the step then shows the summary or the failure. If send throws, the result is offered again on a later update that still shows the call waiting (up to maxAttempts, 3 by default), without running the tool again. Your backend should still accept a result only from the targeted user and install, only while the call is waiting, and only once.

The steps, with their arguments and summaries, are visible to every channel member.

A call whose tool asks the user first reaches the device only once they allowed it, as described in Tool approvals.

Run tool calls from events

The Stream Chat AI SDK and the Stream Chat LangChain SDK ask the app to run a client tool with a custom_client_tool_invocation event on the channel, carrying the tool's name (under tool), its args and the message_id of the reply. The Android client has no type for this event, so it arrives as an UnknownEvent, with the event's fields in rawData.

Listen for it on the client, and run the call through the runner. The runner only runs calls addressed to this user and install, so build the step with those. The event names no user or install, and the step below addresses each call to the runner's own, so every device of the user that watches the channel runs the tool:

// Started with the runner when the user signs in, and disposed when they sign out.
class ClientToolEvents(private val runner: AIClientToolRunner, private val scope: CoroutineScope) {

    private val subscription = ChatClient.instance().subscribeFor("custom_client_tool_invocation") { event ->
        val data = (event as? UnknownEvent)?.rawData ?: return@subscribeFor
        val name = (data["tool"] as? Map<*, *>)?.get("name") as? String ?: return@subscribeFor
        // The runner runs calls addressed to this person and install, so address the call to them.
        val call = AIMessagePart.from(
            type = AIMessagePart.Kind.ToolCall.rawValue,
            payload = mapOf(
                "id" to UUID.randomUUID().toString(),
                "name" to name,
                "executor" to "client",
                "status" to "awaiting_client",
                "target_user_id" to runner.userId,
                "target_client_id" to runner.clientId,
                "arguments" to (data["args"] as? Map<*, *>).orEmpty(),
            ),
        ) ?: return@subscribeFor
        // Events arrive on a background thread, and the runner is confined to the main thread.
        scope.launch {
            runner.run(listOf(call)) { _, _ ->
                // These agents don't wait for a result, so there is nothing to send.
            }
        }
    }

    fun dispose() {
        subscription.dispose()
    }
}

Create it in SignedInSession next to the runner, with the same main-thread scope, and call dispose() from close(), so it listens for as long as the user is signed in. These SDKs don't wait for a result: the tool does its work on the device, and the agent goes on with its reply.