# Personal Assistant

A personal assistant knows what's on the user's plate. It reads their calendar, adds their reminders, searches their email, keeps every conversation they've had with it, and listens when they'd rather talk than type.

>
> **Note:** **What you'll build**
>
> - A side menu with the user's conversations, titled and searchable
> - The user's calendar and reminders, as tools the agent runs on the phone
> - A question before the agent adds a reminder
> - The user's Gmail, which they connect from the conversation
> - Photos the user attaches to a question
> - A voice mode in the same conversation
>

**Starts from:** the [Quickstart](https://getstream.io/chat/docs/sdk/android/ai-components/), with `agents`, `AgentChat` and `AgentMessage` in your project. The calendar tools use Android's `CalendarContract`.

## Best practices

- Run on the phone what only the phone has, such as the calendar, reminders or location. Only a tool's answer leaves the phone.
- Ask for each permission when a tool first needs it, not when the app starts.
- Ask before a tool that changes something, such as adding a reminder.
- Let users connect their own accounts, such as Gmail, from the conversation. The agent never sees their password.

## Define the assistant

Make the agent from the voice template, so it can talk as well as write, and give it each user's Gmail as a connector:

```sh
getstream agents init assistant --template voice
```

```yaml title="agents/assistant/agent.yaml"
name: assistant
mode: voice
llm: llm-fast
connectors:
  - name: gmail # each user signs in to their own Gmail
    tools: [search_threads, get_thread, get_message]
```

Write what the assistant is for in `instructions.md`, then sync it with `getstream agents sync assistant`. [Each user's account](https://getstream.io/agents/docs/connectors/each-user-account/?lang=android) lists the other connectors, such as Google Calendar and Notion.

## Give the agent the calendar and reminders

A tool the app gives the agent runs on the phone. When the model calls it, its lambda runs and returns the answer the model sees.

A tool runs on the session's connection, which outlives any one screen, but only an activity can ask for a permission. `Permissions` lends the activity on screen to the tools, so a tool asks the first time it needs the calendar:

```kotlin
import android.content.Context
import android.content.pm.PackageManager
import androidx.activity.ComponentActivity
import androidx.activity.result.ActivityResultLauncher
import androidx.activity.result.contract.ActivityResultContracts
import kotlinx.coroutines.CompletableDeferred
import kotlinx.coroutines.Dispatchers
import kotlinx.coroutines.withContext

/** Asks for permissions on the activity on screen, the first time a tool needs them. */
object Permissions {
    private lateinit var app: Context
    private var launcher: ActivityResultLauncher<Array<String>>? = null
    private var answer: CompletableDeferred<Map<String, Boolean>>? = null

    /** Call it from your activity's `onCreate`. */
    fun attach(activity: ComponentActivity) {
        app = activity.applicationContext
        launcher = activity.registerForActivityResult(ActivityResultContracts.RequestMultiplePermissions()) {
            answer?.complete(it)
        }
    }

    /** Whether the app holds all of [permissions], asking for those it doesn't first. */
    suspend fun request(vararg permissions: String): Boolean {
        val missing = permissions.filter { app.checkSelfPermission(it) != PackageManager.PERMISSION_GRANTED }
        if (missing.isEmpty()) return true
        val launch = launcher ?: return false
        val pending = CompletableDeferred<Map<String, Boolean>>().also { answer = it }
        withContext(Dispatchers.Main) { launch.launch(missing.toTypedArray()) }
        val granted = pending.await()
        return missing.all { granted[it] == true }
    }
}
```

Android has no reminders app of its own for an app to add to, so `add_reminder` adds an event to the user's calendar, with an alert when it's due. It declares an approval, so the user is asked first, with the reminder's title:

```kotlin
import android.Manifest
import android.content.ContentUris
import android.content.ContentValues
import android.content.Context
import android.provider.CalendarContract
import io.getstream.agents.android.core.AgentTool
import java.text.DateFormat
import java.text.SimpleDateFormat
import java.util.Date
import java.util.Locale
import java.util.TimeZone
import kotlinx.coroutines.Dispatchers
import kotlinx.coroutines.withContext
import kotlinx.serialization.json.JsonObject
import kotlinx.serialization.json.contentOrNull
import kotlinx.serialization.json.jsonPrimitive

private fun JsonObject.string(name: String): String = this[name]?.jsonPrimitive?.contentOrNull.orEmpty()

fun assistantTools(context: Context): List<AgentTool> {
    val resolver = context.contentResolver

    val listEvents = AgentTool(
        name = "list_events",
        description = "The events on the user's calendar for one day.",
        parameters = AgentTool.strings(mapOf("day" to "the day, as YYYY-MM-DD"), required = listOf("day")),
    ) { arguments ->
        if (!Permissions.request(Manifest.permission.READ_CALENDAR)) return@AgentTool "No access to the calendar."
        val start = SimpleDateFormat("yyyy-MM-dd", Locale.US).parse(arguments.string("day"))!!.time
        val time = DateFormat.getTimeInstance(DateFormat.SHORT)
        val columns = arrayOf(CalendarContract.Instances.BEGIN, CalendarContract.Instances.TITLE)
        withContext(Dispatchers.IO) {
            CalendarContract.Instances.query(resolver, columns, start, start + 86_400_000)?.use { rows ->
                buildList {
                    while (rows.moveToNext()) add("${time.format(Date(rows.getLong(0)))} ${rows.getString(1).orEmpty()}")
                }
            }.orEmpty().joinToString("\n")
        }
    }

    val addReminder = AgentTool(
        name = "add_reminder",
        description = "Adds a reminder to the user's calendar, with an alert when it's due.",
        parameters = AgentTool.strings(
            mapOf("title" to "what to remind them of", "due" to "when, in ISO 8601 with the UTC offset"),
            required = listOf("title", "due"),
        ),
        approval = AgentTool.Approval(title = "Add this reminder?", reasonArgument = "title"),
    ) { arguments ->
        val permissions = arrayOf(Manifest.permission.READ_CALENDAR, Manifest.permission.WRITE_CALENDAR)
        if (!Permissions.request(*permissions)) return@AgentTool "No access to the calendar."
        val due = SimpleDateFormat("yyyy-MM-dd'T'HH:mm:ssXXX", Locale.US).parse(arguments.string("due"))!!.time
        withContext(Dispatchers.IO) {
            // The primary calendar the user can write to.
            val calendar = resolver.query(
                CalendarContract.Calendars.CONTENT_URI,
                arrayOf(CalendarContract.Calendars._ID),
                "${CalendarContract.Calendars.CALENDAR_ACCESS_LEVEL} >= ${CalendarContract.Calendars.CAL_ACCESS_CONTRIBUTOR}",
                null,
                "${CalendarContract.Calendars.IS_PRIMARY} DESC",
            )?.use { rows -> if (rows.moveToFirst()) rows.getLong(0) else null }
                ?: return@withContext "There's no calendar to add the reminder to."
            val event = resolver.insert(
                CalendarContract.Events.CONTENT_URI,
                ContentValues().apply {
                    put(CalendarContract.Events.CALENDAR_ID, calendar)
                    put(CalendarContract.Events.TITLE, arguments.string("title"))
                    put(CalendarContract.Events.DTSTART, due)
                    put(CalendarContract.Events.DTEND, due + 15 * 60_000)
                    put(CalendarContract.Events.EVENT_TIMEZONE, TimeZone.getDefault().id)
                    put(CalendarContract.Events.HAS_ALARM, 1)
                },
            ) ?: return@withContext "The calendar didn't take the reminder."
            // An alert as the event starts.
            resolver.insert(
                CalendarContract.Reminders.CONTENT_URI,
                ContentValues().apply {
                    put(CalendarContract.Reminders.EVENT_ID, ContentUris.parseId(event))
                    put(CalendarContract.Reminders.MINUTES, 0)
                    put(CalendarContract.Reminders.METHOD, CalendarContract.Reminders.METHOD_ALERT)
                },
            )
            "Added the reminder."
        }
    }

    return listOf(listEvents, addReminder)
}
```

A tool that throws, such as on a day it can't read, tells the model it failed and why. Add `READ_CALENDAR` and `WRITE_CALENDAR` to your `AndroidManifest.xml`, and call `Permissions.attach(this)` in your activity's `onCreate`.

## Open a conversation

Pass the tools every time a conversation opens, new or carried on. A new one also names where a connector's sign-in comes back to, in your app's own URL scheme:

```kotlin
import android.content.Context
import io.getstream.agents.android.core.AgentSession
import io.getstream.agents.android.core.Session
import io.getstream.agents.android.core.SessionOptions

suspend fun openConversation(context: Context, session: Session?): AgentSession {
    val tools = assistantTools(context)
    if (session != null) {
        return agents.attach(session.id, tools)
    }
    val options = SessionOptions(tools = tools, returnUrl = "com.example.assistant:/connected")
    return agents.agent("assistant").chat(options)
}
```

## List the user's conversations

Each conversation is a session, which the router titles after its first answers. `ConversationList` lists the user's sessions with the assistant, most recent first, and searches them:

```kotlin
import androidx.compose.foundation.layout.Column
import androidx.compose.foundation.layout.fillMaxWidth
import androidx.compose.foundation.layout.padding
import androidx.compose.foundation.lazy.LazyColumn
import androidx.compose.foundation.lazy.items
import androidx.compose.material.icons.Icons
import androidx.compose.material.icons.filled.Edit
import androidx.compose.material3.Icon
import androidx.compose.material3.NavigationDrawerItem
import androidx.compose.material3.OutlinedTextField
import androidx.compose.material3.Text
import androidx.compose.runtime.Composable
import androidx.compose.runtime.LaunchedEffect
import androidx.compose.runtime.getValue
import androidx.compose.runtime.mutableStateOf
import androidx.compose.runtime.remember
import androidx.compose.runtime.setValue
import androidx.compose.ui.Modifier
import androidx.compose.ui.unit.dp
import io.getstream.agents.android.core.AgentsException
import io.getstream.agents.android.core.Session

@Composable
fun ConversationList(onSelect: (Session?) -> Unit) {
    var sessions by remember { mutableStateOf(emptyList<Session>()) }
    var search by remember { mutableStateOf("") }

    LaunchedEffect(search) {
        val all = agents.agent("assistant").sessions
        sessions = try {
            val page = if (search.isEmpty()) all.query() else all.search(search)
            // A session that was only a call has no written conversation to open.
            page.items.filter { it.conversationId.isNotEmpty() }
        } catch (_: AgentsException) {
            emptyList()
        }
    }

    Column {
        OutlinedTextField(
            value = search,
            onValueChange = { search = it },
            placeholder = { Text("Search") },
            singleLine = true,
            modifier = Modifier.fillMaxWidth().padding(16.dp),
        )
        NavigationDrawerItem(
            label = { Text("New conversation") },
            icon = { Icon(Icons.Default.Edit, contentDescription = null) },
            selected = false,
            onClick = { onSelect(null) },
        )
        LazyColumn {
            items(sessions, key = { it.id }) { session ->
                NavigationDrawerItem(
                    label = { Text(session.title.ifEmpty { "New conversation" }) },
                    selected = false,
                    onClick = { onSelect(session) },
                )
            }
        }
    }
}
```

## Put it together

A view model holds the open conversation. Each conversation gets a scope of its own, so switching to another stops following the one before. Closing a session keeps what it recorded, so it opens again from the list:

```kotlin
import android.app.Application
import androidx.lifecycle.AndroidViewModel
import androidx.lifecycle.viewModelScope
import io.getstream.agents.android.core.AgentsException
import io.getstream.agents.android.core.Session
import kotlinx.coroutines.CoroutineScope
import kotlinx.coroutines.Job
import kotlinx.coroutines.SupervisorJob
import kotlinx.coroutines.cancel
import kotlinx.coroutines.flow.MutableStateFlow
import kotlinx.coroutines.flow.StateFlow
import kotlinx.coroutines.flow.asStateFlow
import kotlinx.coroutines.launch

class AssistantHomeViewModel(application: Application) : AndroidViewModel(application) {
    private val _chat = MutableStateFlow<AgentChat?>(null)
    val chat: StateFlow<AgentChat?> = _chat.asStateFlow()
    private var chatScope: CoroutineScope? = null

    init {
        open(null)
    }

    /** Opens [session], or a new conversation, and closes the one before. */
    fun open(session: Session?) {
        val previous = _chat.value
        chatScope?.cancel()
        _chat.value = null
        val scope = CoroutineScope(viewModelScope.coroutineContext + SupervisorJob(viewModelScope.coroutineContext[Job]))
        chatScope = scope
        viewModelScope.launch {
            // What the previous conversation recorded is kept, to carry on later.
            previous?.session?.close()
            try {
                val context = getApplication<Application>()
                _chat.value = AgentChat.open(context, openConversation(context, session), scope)
            } catch (_: AgentsException) {
                // Not opened: the user can pick it again.
            }
        }
    }
}
```

A `ModalNavigationDrawer` puts the list beside the open conversation, and the user opens it from the menu button or by dragging from the leading edge. The conversation's approver gets an `AIToolAuthorizer`, so when the agent first needs Gmail, the reply shows a **Connect Gmail** card that signs the user in:

```kotlin
import androidx.compose.foundation.layout.Box
import androidx.compose.foundation.layout.Column
import androidx.compose.foundation.layout.fillMaxSize
import androidx.compose.foundation.layout.padding
import androidx.compose.material.icons.Icons
import androidx.compose.material.icons.filled.GraphicEq
import androidx.compose.material.icons.filled.Menu
import androidx.compose.material3.CircularProgressIndicator
import androidx.compose.material3.DrawerValue
import androidx.compose.material3.ExperimentalMaterial3Api
import androidx.compose.material3.Icon
import androidx.compose.material3.IconButton
import androidx.compose.material3.ModalDrawerSheet
import androidx.compose.material3.ModalNavigationDrawer
import androidx.compose.material3.Scaffold
import androidx.compose.material3.Text
import androidx.compose.material3.TopAppBar
import androidx.compose.material3.rememberDrawerState
import androidx.compose.runtime.Composable
import androidx.compose.runtime.getValue
import androidx.compose.runtime.mutableStateOf
import androidx.compose.runtime.remember
import androidx.compose.runtime.rememberCoroutineScope
import androidx.compose.runtime.setValue
import androidx.compose.ui.Alignment
import androidx.compose.ui.Modifier
import androidx.compose.ui.platform.LocalContext
import androidx.lifecycle.compose.collectAsStateWithLifecycle
import androidx.lifecycle.viewmodel.compose.viewModel
import io.getstream.chat.android.ai.compose.conversation.AIMessageRole
import io.getstream.chat.android.ai.compose.parts.AIClientIdentity
import io.getstream.chat.android.ai.compose.ui.component.AIConversation
import io.getstream.chat.android.ai.compose.ui.component.AIToolApprover
import io.getstream.chat.android.ai.compose.ui.component.AIToolAuthorizer
import io.getstream.chat.android.ai.compose.ui.component.ChatComposer
import kotlinx.coroutines.launch

@OptIn(ExperimentalMaterial3Api::class)
@Composable
fun AssistantHome(viewModel: AssistantHomeViewModel = viewModel()) {
    val chat by viewModel.chat.collectAsStateWithLifecycle()
    val drawer = rememberDrawerState(DrawerValue.Closed)
    val scope = rememberCoroutineScope()
    var isTalking by remember { mutableStateOf(false) }

    val current = chat
    if (isTalking && current != null) {
        VoiceModeScreen(current.session, onClose = { isTalking = false })
        return
    }
    ModalNavigationDrawer(
        drawerState = drawer,
        drawerContent = {
            ModalDrawerSheet {
                ConversationList { session ->
                    scope.launch { drawer.close() }
                    viewModel.open(session)
                }
            }
        },
    ) {
        Scaffold(
            topBar = {
                TopAppBar(
                    title = { Text("Assistant") },
                    navigationIcon = {
                        IconButton(onClick = { scope.launch { drawer.open() } }) {
                            Icon(Icons.Default.Menu, contentDescription = "Conversations")
                        }
                    },
                    actions = {
                        IconButton(onClick = { isTalking = true }, enabled = current != null) {
                            Icon(Icons.Default.GraphicEq, contentDescription = "Talk")
                        }
                    },
                )
            },
        ) { padding ->
            if (current == null) {
                Box(Modifier.fillMaxSize().padding(padding), contentAlignment = Alignment.Center) {
                    CircularProgressIndicator()
                }
            } else {
                AssistantConversation(current, Modifier.padding(padding))
            }
        }
    }
}

@Composable
fun AssistantConversation(chat: AgentChat, modifier: Modifier = Modifier) {
    val context = LocalContext.current
    val scope = rememberCoroutineScope()
    val signIns = remember { SignIns(context) }
    val messages by chat.messages.collectAsStateWithLifecycle()
    val answering by chat.isAnswering.collectAsStateWithLifecycle()

    // The quickstart's approver, with an authorizer for the sign-ins of connectors such as Gmail.
    val approver = remember(chat) {
        val authorizer = AIToolAuthorizer(
            connect = { _, url -> signIns.open(url) },
            decline = { call -> chat.session.declineLogin(call.id) },
        )
        val userId = agents.backend.user?.id.orEmpty()
        AIToolApprover(userId, AIClientIdentity.installId(context), scope, authorizer) { call, allowed ->
            chat.session.decide(call.id, allowed)
        }
    }

    Column(modifier) {
        AIConversation(
            messages = messages,
            modifier = Modifier.weight(1f),
            isThinking = answering && messages.lastOrNull()?.role != AIMessageRole.Assistant,
            approver = approver,
        )
        ChatComposer(
            onSendClick = { chat.send(it.text) },
            onStopClick = chat::stop,
            isGenerating = answering,
        )
    }
}
```

`SignIns` opens the sign-in page in a Custom Tab and resumes once the browser comes back to `com.example.assistant:/connected`, as described in [Tool sign-ins](https://getstream.io/chat/docs/sdk/android/ai-components/reasoning-and-tool-calls/#tool-sign-ins). Register that scheme on the activity that hands the return URL to it. The reply shows each tool call as the agent makes it, and the question of `add_reminder` under its call.

## Send photos

The quickstart's `AgentChat.send` sends only the text. To send the photos the user attaches in `ChatComposer`, shrink each one and send it inline. Add a `send` that takes them to `AgentChat`, next to the one for text:

```kotlin
import android.content.Context
import android.graphics.Bitmap
import android.graphics.BitmapFactory
import android.graphics.ImageDecoder
import android.net.Uri
import android.os.Build
import android.util.Base64
import io.getstream.agents.android.core.AgentsException
import io.getstream.agents.android.core.ImageSource
import java.io.ByteArrayOutputStream
import kotlinx.coroutines.Dispatchers
import kotlinx.coroutines.launch
import kotlinx.coroutines.withContext

fun send(context: Context, text: String, photos: Set<Uri>) {
    scope.launch {
        try {
            session.responses.create(text, images = photoSources(context, photos))
        } catch (_: AgentsException) {
            // Not sent: the user can send it again.
        }
    }
}

/** Each photo, at most 1280 pixels across and turned the way the camera held it, sent inline as a JPEG. */
suspend fun photoSources(context: Context, photos: Collection<Uri>): List<ImageSource> = withContext(Dispatchers.IO) {
    photos.mapNotNull { photo ->
        val bitmap = decodePhoto(context, photo) ?: return@mapNotNull null
        val jpeg = ByteArrayOutputStream().use {
            bitmap.compress(Bitmap.CompressFormat.JPEG, 70, it)
            it.toByteArray()
        }
        ImageSource("data:image/jpeg;base64," + Base64.encodeToString(jpeg, Base64.NO_WRAP))
    }
}

private fun decodePhoto(context: Context, photo: Uri): Bitmap? = runCatching {
    if (Build.VERSION.SDK_INT >= 28) {
        // ImageDecoder turns the photo the way the camera held it.
        ImageDecoder.decodeBitmap(ImageDecoder.createSource(context.contentResolver, photo)) { decoder, info, _ ->
            val scale = minOf(1f, 1280f / maxOf(info.size.width, info.size.height))
            decoder.setTargetSize((info.size.width * scale).toInt(), (info.size.height * scale).toInt())
            // A hardware bitmap can't be compressed.
            decoder.allocator = ImageDecoder.ALLOCATOR_SOFTWARE
        }
    } else {
        // Before Android 9, the photo is decoded as stored, without its rotation.
        val bitmap = context.contentResolver.openInputStream(photo)?.use { BitmapFactory.decodeStream(it) }
            ?: return null
        val scale = minOf(1f, 1280f / maxOf(bitmap.width, bitmap.height))
        Bitmap.createScaledBitmap(bitmap, (bitmap.width * scale).toInt(), (bitmap.height * scale).toInt(), true)
    }
}.getOrNull()
```

Then send the composer's attachments with the text:

```kotlin
ChatComposer(
    onSendClick = { chat.send(context, it.text, it.attachments) },
    onStopClick = chat::stop,
    isGenerating = answering,
)
```

Pick a model that reads images in `agent.yaml`. `responses.create` also takes a `reasoningEffort`, for an option such as "Think harder" in the composer.

## Talk instead of typing

The Talk button switches the open conversation to voice. The agent joins a call with everything said so far, and what's said on the call is written into the same conversation:

```kotlin
import android.Manifest
import androidx.activity.compose.rememberLauncherForActivityResult
import androidx.activity.result.contract.ActivityResultContracts
import androidx.compose.foundation.layout.Box
import androidx.compose.foundation.layout.fillMaxSize
import androidx.compose.material3.CircularProgressIndicator
import androidx.compose.runtime.Composable
import androidx.compose.runtime.LaunchedEffect
import androidx.compose.runtime.getValue
import androidx.compose.runtime.mutableStateOf
import androidx.compose.runtime.remember
import androidx.compose.runtime.rememberCoroutineScope
import androidx.compose.runtime.setValue
import androidx.compose.ui.Alignment
import androidx.compose.ui.Modifier
import androidx.compose.ui.platform.LocalContext
import io.getstream.agents.android.core.AgentSession
import io.getstream.agents.android.core.AgentsException
import io.getstream.agents.android.core.VoiceSession
import kotlinx.coroutines.launch

@Composable
fun VoiceModeScreen(session: AgentSession, onClose: () -> Unit) {
    val context = LocalContext.current
    val scope = rememberCoroutineScope()
    var voice by remember { mutableStateOf<VoiceSession?>(null) }
    val microphone = rememberLauncherForActivityResult(ActivityResultContracts.RequestPermission()) { granted ->
        if (!granted) onClose()
    }
    LaunchedEffect(Unit) { microphone.launch(Manifest.permission.RECORD_AUDIO) }

    val joined = voice
    if (joined != null) {
        VoiceCallScreen(joined) {
            // Takes the agent off the call, and the conversation goes on in writing.
            scope.launch {
                try {
                    session.stopVoice()
                } catch (_: AgentsException) {
                    // The call already ended.
                }
                onClose()
            }
        }
    } else {
        Box(Modifier.fillMaxSize(), contentAlignment = Alignment.Center) { CircularProgressIndicator() }
        LaunchedEffect(session) {
            try {
                // The agent joins the call with everything said so far.
                session.startVoice()
                voice = VoiceSession.attach(context, agents, session.id)
            } catch (_: AgentsException) {
                onClose()
            }
        }
    }
}
```

`VoiceCallScreen` is the call screen of the [Voice agent](https://getstream.io/chat/docs/sdk/android/ai-components/use-cases/voice-agent/) use case, which joins the call and leaves it when the user hangs up. Then `stopVoice()` takes the agent off the call, and the conversation goes on in writing.

## Next steps

- [Voice agent](https://getstream.io/chat/docs/sdk/android/ai-components/use-cases/voice-agent/): the call screen in full.
- [Sessions](https://getstream.io/agents/docs/sessions/?lang=android): listing, searching, forking and rewinding conversations.
- [Tool calling & Dispatch](https://getstream.io/agents/docs/tool-calling-and-dispatch/?lang=android): tools, approvals, timeouts and tools that return images.
- [Generative UI](https://getstream.io/chat/docs/sdk/android/ai-components/generative-ui/): answers that are UI, such as a day plan with a form to change it.

---

For the most recent version of this documentation, visit [https://getstream.io/chat/docs/sdk/android/ai-components/use-cases/personal-assistant/](https://getstream.io/chat/docs/sdk/android/ai-components/use-cases/personal-assistant/).