# TTS & Voices TTS keeps advancing at a rapid pace. Most providers can clone a voice from 30 seconds of audio, so the same voice can be taught to several providers and [follow the router](#voices) when it fails over. Some models focus on speed for voice AI, and others on how good they sound. ## TTS router yaml Here's an example of setting up your first router. It tries ElevenLabs Flash first, then any Cartesia voice, and ends with the `en-low-latency` group: ```yaml title="routers/switchboard.yaml" name: switchboard tts: providers: [elevenlabs/eleven_flash_v2_5, cartesia, en-low-latency] ``` You can sync your new router like this: ```sh curl -fsSL https://getstream.io/cli/beta/install.sh | bash # the agents commands are in the beta CLI getstream agents sync --router=switchboard ``` And then use it in your code as shown below. ## Speak live ```ts label="JavaScript" import { Socket, text } from "@stream-io/calypso"; const voice = await Socket.open( client.backend, await client.backend.socketURL("/v1/tts/stream"), ); try { voice.send({ type: "start", config_id: "switchboard", tts: {} }); voice.send({ type: "speak", text: "Thanks for calling. How can I help?", final: true, }); for await (const chunk of voice.messages()) { if (chunk instanceof Uint8Array) { process.stdout.write(chunk.subarray(8)); // 16-bit PCM, after an 8-byte header naming the sample rate continue; } if (chunk.type === "synthesis_complete") break; if (chunk.type === "error") throw new Error(`speaking: ${text(chunk, "error")}`); } } finally { voice.close(); } ``` ```python label="Python" import sys from vision_agents.plugins import stream router = stream.Client().router("switchboard") async with router.tts.realtime() as voice: async for chunk in await voice.stream_audio("Thanks for calling. How can I help?"): sys.stdout.buffer.write(chunk.samples.tobytes()) # 16-bit PCM at chunk.sample_rate ``` ```ruby label="Ruby" router = api.router("switchboard") router.tts.realtime do |voice| voice.speak("Thanks for calling. How can I help?") do |chunk| $stdout.write(chunk.pcm) # 16-bit PCM at chunk.sample_rate end end ``` ```php label="PHP" $voice = $client->router('switchboard')->tts->realtime(); try { $voice->speak('Thanks for calling. How can I help?'); foreach ($voice->frames() as $frame) { if (is_string($frame)) { fwrite(STDOUT, substr($frame, 8)); // 16-bit PCM after an 8-byte header naming the sample rate continue; } if ($frame['type'] === 'synthesis_complete') { break; } } } finally { $voice->close(); } ``` ```go label="Go" voice, err := client.Router("switchboard").TTS().Realtime(ctx, nil) if err != nil { return err } defer voice.Close() if err := voice.Speak("Thanks for calling. How can I help?"); err != nil { return err } for chunk := range voice.Audio() { if chunk.Done { break } if chunk.Error != "" { return fmt.Errorf("speaking: %s", chunk.Error) } os.Stdout.Write(chunk.Samples) // 16-bit PCM at chunk.SampleRate } ``` ```csharp label="C#" await using var voice = await client.Router("switchboard").Tts.RealtimeAsync(); using var stdout = Console.OpenStandardOutput(); await voice.SpeakAsync("Thanks for calling. How can I help?"); await foreach (var chunk in voice.ReadAllAsync()) { if (chunk.Done) { break; } if (chunk.Error != "") { throw new InvalidOperationException($"speaking: {chunk.Error}"); } await stdout.WriteAsync(chunk.Samples); // 16-bit PCM at chunk.SampleRate } ``` ```rust label="Rust" use std::io::Write; use vision_agents::types::TtsOptions; let mut voice = client .router("switchboard") .voice(TtsOptions::default()) .await?; let mut out = std::io::stdout(); voice .speak("Thanks for calling. How can I help?", |audio| { let _ = out.write_all(&audio.pcm); // 16-bit PCM at audio.sample_rate }) .await?; voice.close().await; ``` Interrupting the voice stops the current utterance, which is what barge-in sends when the caller starts talking over the agent. ## Make a recording To check that speech works on an app, ask for the recording `inline`. The audio comes back, base64, in the 202 response, and nothing is stored: ```ts label="JavaScript" const file = await client.post("/v1/tts/recordings", { body: { config_id: "switchboard", text: "Testing, one, two.", inline: true }, }); console.log(file.status, file.voice, file.error ?? ""); ``` ```python label="Python" import base64 from vision_agents.plugins import stream file = await stream.Client().router("switchboard").tts.recording( "Testing, one, two.", inline=True ) print(file.voice, len(base64.b64decode(file.audio)), "bytes") ``` An inline recording takes at most 16000 characters and no `callback`, and it must finish within 90 seconds. A synthesis that fails is still a 202, with `status: failed` and the reason in `error`. Python raises it as a `RuntimeError`. `voice` is the voice that spoke, the provider's default included, when the provider can say. Without `inline`, a recording is a job that stores the file. Poll it until it is done: ```ts label="JavaScript" let file = await client.post("/v1/tts/recordings", { body: { config_id: "switchboard", text: "Chapter one.", options: { format: "mp3_44100_128" }, }, }); while (file.status === "queued" || file.status === "running") { await new Promise((resolve) => setTimeout(resolve, 1000)); file = await client.get("/v1/tts/recordings/{id}", { path: { id: file.id } }); } console.log(file.url); ``` ```python label="Python" from vision_agents.plugins import stream file = await stream.Client().router("switchboard").tts.recording( "Chapter one.", format="mp3_44100_128" ) print(file.url) ``` ```ruby label="Ruby" router = api.router("switchboard") file = router.tts.recording("Chapter one.", format: "mp3_44100_128") puts file["url"] ``` ```php label="PHP" use GetStream\VisionAgents\Generated\TtsOptions; $file = $client->router('switchboard')->tts->recording('Chapter one.', new TtsOptions(format: 'mp3_44100_128')); echo $file->url, "\n"; ``` ```go label="Go" format := "mp3_44100_128" file, err := client.Router("switchboard").TTS().Recording(ctx, "Chapter one.", &acceleration.TtsOptions{Format: &format}) if err != nil { return err } fmt.Println(*file.Url) ``` ```csharp label="C#" var file = await client.Router("switchboard").Tts.RecordingAsync("Chapter one.", new TtsOptions { Format = "mp3_44100_128" }); Console.WriteLine(file.Url); ``` ```rust label="Rust" use vision_agents::types::TtsOptions; let file = client .router("switchboard") .record( "Chapter one.", TtsOptions { format: Some("mp3_44100_128".into()), ..Default::default() }, None, ) .await?; println!("{}", file.url.unwrap_or_default()); ``` Use `mp3_44100_128` for a file somebody downloads and `ulaw_8000` for telephony. Only ElevenLabs models record. The other models only stream. A recording on models that only stream, such as a pinned `cartesia/sonic-preview`, is refused with a 400 that names the models that record. A route that reaches ElevenLabs, such as `tts-quality`, records through ElevenLabs. `can_record` on each candidate of `GET /v1/tts/routes` and `GET /v1/tts/routes/{target}` says which models can record. ## Voices An agent speaks in one of two kinds of voice: a stock voice from a provider's library, or a voice of your own, cloned from a recording. A cloned voice is stored once in Stream and taught to every text-to-speech provider that can clone, so the router can still fail over between providers in the middle of a call without the agent changing voice. ### Pick a stock voice `GET /v1/agents/voices/library` lists the voices each provider publishes, by provider and then name, with whatever the vendor says about each: gender, accent, language and tags. Add `?provider=elevenlabs` for one provider's. `GET /v1/agents/voices/library/{provider}/{voice}/preview` plays the sample the vendor published; nothing is synthesised, so browsing costs nothing. Put the voice's `id` in the agent's `voice` setting. A provider that cannot be reached is listed under `unavailable` rather than emptying the list. The library has ElevenLabs, Cartesia and Inworld. Cartesia's voices are all listed. Fish Audio voices cannot be browsed through the router: find one in the fish.audio Voice Library and put its model id, which Fish's API calls `reference_id`, in `voice`. `voice` names a voice from the voice catalogue, which resolves it to the provider that has it. A name no provider has is a 404, not a routing failure. To give one vendor a voice of its own, name it in that vendor's `overwrites`, such as `{"cartesia": {"voice_id": ""}}`. A voice named only there is spoken only by that vendor, in that voice. If that vendor cannot serve the request, the request is refused. Another vendor never speaks it in its own default voice. A top-level `voice` does not tie the request to one vendor. Name a voice in one of the two places, not both: with both, which one is spoken depends on the vendor. The voice that spoke, the provider's default included, is `voice` on the `started` frame of `/v1/tts/stream` and on a recording, when the provider can say. ### Clone a voice A voice of your own takes four steps: 1. **Create** it with a name. It starts empty. 2. **Add a recording.** Thirty seconds of clean speech from one speaker is plenty. Add a transcript of what is said if you have one; the providers that use it clone more faithfully. 3. **Prepare** it. Each provider is sent the recordings and hands back its own copy of the voice. 4. **Preview** it through a provider, to hear what a caller will hear. These calls are backend-only: they need your API secret, so a phone or browser cannot make them. No SDK wraps them yet, so each example uses its SDK's plain HTTP client. ```ts label="JavaScript" import { readFile, writeFile } from "node:fs/promises"; import { Client } from "@stream-io/calypso"; const api = new Client(); const voice = await api.post("/v1/agents/voices", { body: { name: "founder", description: "Jo, for the welcome line" }, }); await api.post("/v1/agents/voices/{id}/samples", { path: { id: voice.id }, body: { audio: (await readFile("founder.wav")).toString("base64"), filename: "founder.wav", transcript: "Hi, I'm Jo, and I started this company to make support calls less painful.", }, }); const prepared = await api.post("/v1/agents/voices/{id}/prepare", { path: { id: voice.id }, body: {}, }); for (const binding of prepared.bindings ?? []) { console.log(binding.provider, binding.state, binding.error ?? ""); } const preview = await api.post("/v1/agents/voices/{id}/preview", { path: { id: voice.id }, body: { provider: "elevenlabs", text: "Thanks for calling. How can I help?" }, }); await writeFile("preview.mp3", Buffer.from(preview.audio, "base64")); ``` ```python label="Python" import base64 from pathlib import Path import httpx from vision_agents.plugins import stream backend = stream.Client().backend async with httpx.AsyncClient(base_url=backend.url, headers=backend.headers) as http: voice = ( await http.post( "/v1/agents/voices", json={"name": "founder", "description": "Jo, for the welcome line"}, ) ).raise_for_status().json() ( await http.post( f"/v1/agents/voices/{voice['id']}/samples", json={ "audio": base64.b64encode(Path("founder.wav").read_bytes()).decode(), "filename": "founder.wav", "transcript": "Hi, I'm Jo, and I started this company to make support calls less painful.", }, ) ).raise_for_status() prepared = ( await http.post(f"/v1/agents/voices/{voice['id']}/prepare", json={}) ).raise_for_status().json() for binding in prepared["bindings"]: print(binding["provider"], binding["state"], binding.get("error", "")) preview = ( await http.post( f"/v1/agents/voices/{voice['id']}/preview", json={"provider": "elevenlabs", "text": "Thanks for calling. How can I help?"}, ) ).raise_for_status().json() Path("preview.mp3").write_bytes(base64.b64decode(preview["audio"])) ``` ```ruby label="Ruby" require "base64" require "getstream/vision_agents" api = GetStream::VisionAgents::Client.new(api_key: ENV["STREAM_API_KEY"], api_secret: ENV["STREAM_API_SECRET"]) voice = api.post("/v1/agents/voices", body: { name: "founder", description: "Jo, for the welcome line" }) api.post("/v1/agents/voices/{id}/samples", path: { id: voice["id"] }, body: { audio: Base64.strict_encode64(File.binread("founder.wav")), filename: "founder.wav", transcript: "Hi, I'm Jo, and I started this company to make support calls less painful." }) prepared = api.post("/v1/agents/voices/{id}/prepare", path: { id: voice["id"] }, body: {}) prepared["bindings"].each { |binding| puts "#{binding["provider"]} #{binding["state"]} #{binding["error"]}" } preview = api.post("/v1/agents/voices/{id}/preview", path: { id: voice["id"] }, body: { provider: "elevenlabs", text: "Thanks for calling. How can I help?" }) File.binwrite("preview.mp3", Base64.decode64(preview["audio"])) ``` ```php label="PHP" use GetStream\VisionAgents\Backend; use GetStream\VisionAgents\Client; $client = new Client(new Backend(apiKey: $key, apiSecret: $secret, authenticate: true)); $voice = $client->post('/v1/agents/voices', body: ['name' => 'founder', 'description' => 'Jo, for the welcome line']); $id = ['id' => $voice['id']]; $client->post('/v1/agents/voices/{id}/samples', $id, body: [ 'audio' => base64_encode(file_get_contents('founder.wav')), 'filename' => 'founder.wav', 'transcript' => "Hi, I'm Jo, and I started this company to make support calls less painful.", ]); $prepared = $client->post('/v1/agents/voices/{id}/prepare', $id, body: []); foreach ($prepared['bindings'] as $binding) { echo $binding['provider'], ' ', $binding['state'], ' ', $binding['error'] ?? '', "\n"; } $preview = $client->post('/v1/agents/voices/{id}/preview', $id, body: [ 'provider' => 'elevenlabs', 'text' => 'Thanks for calling. How can I help?', ]); file_put_contents('preview.mp3', base64_decode($preview['audio'])); ``` ```go label="Go" api, err := stream.Backend{}.Client() if err != nil { log.Fatal(err) } description := "Jo, for the welcome line" created, err := api.CreateVoiceWithResponse(ctx, acceleration.VoiceRequest{Name: "founder", Description: &description}) if err != nil { log.Fatal(err) } if created.JSON201 == nil { log.Fatalf("creating the voice: %s", created.Status()) } voice := created.JSON201 audio, err := os.ReadFile("founder.wav") if err != nil { log.Fatal(err) } filename := "founder.wav" transcript := "Hi, I'm Jo, and I started this company to make support calls less painful." if _, err := api.AddVoiceSampleWithResponse(ctx, voice.Id, acceleration.VoiceSampleRequest{ Audio: audio, Filename: &filename, Transcript: &transcript, }); err != nil { log.Fatal(err) } prepared, err := api.PrepareVoiceWithResponse(ctx, voice.Id, acceleration.PrepareVoiceRequest{}) if err != nil { log.Fatal(err) } if prepared.JSON200 == nil { log.Fatalf("preparing the voice: %s", prepared.Status()) } for _, binding := range *prepared.JSON200.Bindings { fmt.Println(binding.Provider, binding.State) } line := "Thanks for calling. How can I help?" preview, err := api.PreviewVoiceWithResponse(ctx, voice.Id, acceleration.VoicePreviewRequest{Provider: "elevenlabs", Text: &line}) if err != nil { log.Fatal(err) } if preview.JSON200 == nil { log.Fatalf("previewing the voice: %s", preview.Status()) } if err := os.WriteFile("preview.mp3", preview.JSON200.Audio, 0o644); err != nil { log.Fatal(err) } ``` ```csharp label="C#" using GetStream.VisionAgents; using GetStream.VisionAgents.Models; using Voice = GetStream.VisionAgents.Models.Voice; using var api = new VisionAgentsClient(new VisionAgentsOptions { ApiKey = key, ApiSecret = secret, Authenticate = true }); var voice = await api.PostAsync("/v1/agents/voices", new VoiceRequest { Name = "founder", Description = "Jo, for the welcome line" }); await api.PostAsync($"/v1/agents/voices/{voice.Id}/samples", new VoiceSampleRequest { Audio = await File.ReadAllBytesAsync("founder.wav"), Filename = "founder.wav", Transcript = "Hi, I'm Jo, and I started this company to make support calls less painful.", }); var prepared = await api.PostAsync($"/v1/agents/voices/{voice.Id}/prepare", new PrepareVoiceRequest()); foreach (var binding in prepared.Bindings ?? []) { Console.WriteLine($"{binding.Provider} {binding.State} {binding.Error}"); } var preview = await api.PostAsync($"/v1/agents/voices/{voice.Id}/preview", new VoicePreviewRequest { Provider = "elevenlabs", Text = "Thanks for calling. How can I help?" }); await File.WriteAllBytesAsync("preview.mp3", preview.Audio); ``` ```rust label="Rust" use base64::{Engine, engine::general_purpose::STANDARD}; use vision_agents::Client; use vision_agents::types::{PrepareVoiceRequest, VoicePreviewRequest, VoiceRequest, VoiceSampleRequest}; let api = Client::from_env()?; let voice = api .create_voice(&VoiceRequest { name: "founder".into(), description: Some("Jo, for the welcome line".into()), }) .await?; api.add_voice_sample( &voice.id, &VoiceSampleRequest { audio: STANDARD.encode(std::fs::read("founder.wav")?), filename: Some("founder.wav".into()), transcript: Some("Hi, I'm Jo, and I started this company to make support calls less painful.".into()), ..Default::default() }, ) .await?; let prepared = api.prepare_voice(&voice.id, &PrepareVoiceRequest::default()).await?; for binding in prepared.bindings.iter().flatten() { println!("{} {:?} {}", binding.provider, binding.state, binding.error.clone().unwrap_or_default()); } let preview = api .preview_voice( &voice.id, &VoicePreviewRequest { provider: "elevenlabs".into(), text: Some("Thanks for calling. How can I help?".into()), }, ) .await?; std::fs::write("preview.mp3", STANDARD.decode(preview.audio)?)?; ``` The name is how the agent will ask for the voice, and it is unique among your voices. Send the recording's filename with its extension: that is how a provider knows what format it was given. Preparing reports one binding per provider: `ready` when that provider can speak in the voice, `failed` with the reason when it refused the recordings. One provider refusing leaves the others usable, and the router simply does not pick that provider for this voice. `prepare` teaches every provider this deployment can clone with. Pass `providers` to choose, for example `{"providers": ["elevenlabs", "cartesia"]}`; `GET /v1/agents/voices/providers` lists the ones available. Today they are Cartesia, ElevenLabs, Fish Audio and Inworld. From a terminal, the CLI makes the same calls. A recording's base64 can be too long for the command line, so `--request` reads the JSON from a file with `@`, or from stdin with `-`: ```sh base64 < founder.wav | tr -d '\n' | jq -Rs '{audio: ., filename: "founder.wav"}' | getstream agents api AddVoiceSample --id --request - ``` Only clone a voice you have the speaker's permission to use. Each provider's terms apply to its copy. ### Speak in it Name the voice in the agent's `voice` setting with the `custom:` prefix: ```yaml # agents/myagent/agent.yaml tts: elevenlabs/eleven_v3_conversational voice: custom:founder ``` The prefix says the voice is one of yours. Without it, a name is looked for among your voices first and otherwise passed to the provider as one of its stock voices. With it, a name that is not yours is refused, so a typo cannot quietly become a stranger's voice from a provider's catalogue. For each call the router asks the provider it routes to for that provider's copy of the voice. If the provider it would pick was never taught the voice, it moves on to one that was, so a voice prepared with two providers can still fail over. The same name works wherever a voice is named: in the `voice` option when a session is opened, and in the [session settings](https://getstream.io/agents/docs/python/sessions/#update-a-running-session) that swap the voice of a running call. ### Change or remove a voice | Call | What it does | | ------------------------------------- | ---------------------------------------------------------------------------------------------------------------- | | `GET /v1/agents/voices` | Your voices, newest first | | `GET /v1/agents/voices/{id}` | One voice, with its recordings and each provider's binding | | `PUT /v1/agents/voices/{id}` | Renames it. What it sounds like does not change | | `POST /v1/agents/voices/{id}/samples` | Adds another recording. It is not used until you prepare the voice again | | `POST /v1/agents/voices/{id}/prepare` | Teaches the providers again, from every recording. Each provider's previous copy is replaced and removed from it | | `DELETE /v1/agents/voices/{id}` | Removes the voice from every provider, then its recordings. Past calls still name it | Recordings are kept, so a provider added later can be taught the voice without asking the speaker to record again. On a self-hosted router they live in the bucket named by `voices_bucket_url`; without one, voices of your own are unavailable. See [Self-hosting](https://getstream.io/agents/docs/python/self-hosting/). ## Options | Option | What it asks for | | ------------------ | -------------------------------------------------------------------------------------------------- | | `voice` | The voice to speak with. | | `languages` | The language to speak. Some providers infer it from the text when it is absent. | | `speed` | Speaking rate, 1 being normal. Each provider has its own range, and a speed outside it is refused. | | `volume` | Loudness, 1 being the voice's own. | | `emotion`, `style` | Affect or delivery style, for the providers that take one. | | `stability` | How much the voice may vary between chunks. Higher is flatter and more consistent. | | `similarity` | How closely a cloned voice tracks its reference. | | `format` | The audio encoding of a recording. Live sockets are always PCM. | | `pronunciations` | How to say words the voice gets wrong, keyed by the word. | | `chunk_schedule` | How much text to buffer before speaking. | ## What varies between providers - **The best voices have the fewest settings.** ElevenLabs v3 conversational takes one setting, `stability`, and only on the first message. Everything else it can do is said in the text, as audio tags such as `[whispers]`. - **Speed ranges do not overlap.** Cartesia takes 0.6 to 1.5, Inworld 0.5 to 1.5 and ElevenLabs flash 0.7 to 1.2. `speed` is not one number every voice accepts. - **"Sound urgent" has three shapes.** A fixed list of emotions, English only (Cartesia); free prose in brackets that persists until reset (Inworld); tags the model acts out as it reads them (ElevenLabs v3). An `emotion` string only reaches the first. - **Live voices take few options.** Today the live models serve a target, a voice and a language. To steer one further, use its `overwrites`, for example `inworld: {delivery_mode: STABLE}`. --- For the most recent version of this documentation, visit [https://getstream.io/agents/docs/python/tts/](https://getstream.io/agents/docs/python/tts/).