TTS & Voices

TTS keeps advancing at a rapid pace.

Most providers can clone a voice from 30 seconds of audio, so the same voice can be taught to several providers and follow the router when it fails over.

Some models focus on speed for voice AI, and others on how good they sound.

TTS router yaml

Here's an example of setting up your first router. It tries ElevenLabs Flash first, then any Cartesia voice, and ends with the en-low-latency group:

routers/switchboard.yaml
name: switchboard

tts:
  providers: [elevenlabs/eleven_flash_v2_5, cartesia, en-low-latency]

You can sync your new router like this:

curl -fsSL https://getstream.io/cli/beta/install.sh | bash # the agents commands are in the beta CLI
getstream agents sync --router=switchboard

And then use it in your code as shown below.

Speak live

import { Socket, text } from "@stream-io/calypso";

const voice = await Socket.open(
  client.backend,
  await client.backend.socketURL("/v1/tts/stream"),
);
try {
  voice.send({ type: "start", config_id: "switchboard", tts: {} });

  voice.send({
    type: "speak",
    text: "Thanks for calling. How can I help?",
    final: true,
  });
  for await (const chunk of voice.messages()) {
    if (chunk instanceof Uint8Array) {
      process.stdout.write(chunk.subarray(8)); // 16-bit PCM, after an 8-byte header naming the sample rate
      continue;
    }
    if (chunk.type === "synthesis_complete") break;
    if (chunk.type === "error")
      throw new Error(`speaking: ${text(chunk, "error")}`);
  }
} finally {
  voice.close();
}

Interrupting the voice stops the current utterance, which is what barge-in sends when the caller starts talking over the agent.

Make a recording

To check that speech works on an app, ask for the recording inline. The audio comes back, base64, in the 202 response, and nothing is stored:

const file = await client.post("/v1/tts/recordings", {
  body: { config_id: "switchboard", text: "Testing, one, two.", inline: true },
});
console.log(file.status, file.voice, file.error ?? "");

An inline recording takes at most 16000 characters and no callback, and it must finish within 90 seconds. A synthesis that fails is still a 202, with status: failed and the reason in error. Python raises it as a RuntimeError. voice is the voice that spoke, the provider's default included, when the provider can say.

Without inline, a recording is a job that stores the file. Poll it until it is done:

let file = await client.post("/v1/tts/recordings", {
  body: {
    config_id: "switchboard",
    text: "Chapter one.",
    options: { format: "mp3_44100_128" },
  },
});
while (file.status === "queued" || file.status === "running") {
  await new Promise((resolve) => setTimeout(resolve, 1000));
  file = await client.get("/v1/tts/recordings/{id}", { path: { id: file.id } });
}
console.log(file.url);

Use mp3_44100_128 for a file somebody downloads and ulaw_8000 for telephony.

Only ElevenLabs models record. The other models only stream. A recording on models that only stream, such as a pinned cartesia/sonic-preview, is refused with a 400 that names the models that record. A route that reaches ElevenLabs, such as tts-quality, records through ElevenLabs. can_record on each candidate of GET /v1/tts/routes and GET /v1/tts/routes/{target} says which models can record.

Voices

An agent speaks in one of two kinds of voice: a stock voice from a provider's library, or a voice of your own, cloned from a recording. A cloned voice is stored once in Stream and taught to every text-to-speech provider that can clone, so the router can still fail over between providers in the middle of a call without the agent changing voice.

Pick a stock voice

GET /v1/agents/voices/library lists the voices each provider publishes, by provider and then name, with whatever the vendor says about each: gender, accent, language and tags. Add ?provider=elevenlabs for one provider's. GET /v1/agents/voices/library/{provider}/{voice}/preview plays the sample the vendor published; nothing is synthesised, so browsing costs nothing.

Put the voice's id in the agent's voice setting. A provider that cannot be reached is listed under unavailable rather than emptying the list.

The library has ElevenLabs, Cartesia and Inworld. Cartesia's voices are all listed. Fish Audio voices cannot be browsed through the router: find one in the fish.audio Voice Library and put its model id, which Fish's API calls reference_id, in voice.

voice names a voice from the voice catalogue, which resolves it to the provider that has it. A name no provider has is a 404, not a routing failure.

To give one vendor a voice of its own, name it in that vendor's overwrites, such as {"cartesia": {"voice_id": "<id>"}}. A voice named only there is spoken only by that vendor, in that voice. If that vendor cannot serve the request, the request is refused. Another vendor never speaks it in its own default voice. A top-level voice does not tie the request to one vendor. Name a voice in one of the two places, not both: with both, which one is spoken depends on the vendor.

The voice that spoke, the provider's default included, is voice on the started frame of /v1/tts/stream and on a recording, when the provider can say.

Clone a voice

A voice of your own takes four steps:

  1. Create it with a name. It starts empty.
  2. Add a recording. Thirty seconds of clean speech from one speaker is plenty. Add a transcript of what is said if you have one; the providers that use it clone more faithfully.
  3. Prepare it. Each provider is sent the recordings and hands back its own copy of the voice.
  4. Preview it through a provider, to hear what a caller will hear.

These calls are backend-only: they need your API secret, so a phone or browser cannot make them. No SDK wraps them yet, so each example uses its SDK's plain HTTP client.

import { readFile, writeFile } from "node:fs/promises";
import { Client } from "@stream-io/calypso";

const api = new Client();

const voice = await api.post("/v1/agents/voices", {
  body: { name: "founder", description: "Jo, for the welcome line" },
});

await api.post("/v1/agents/voices/{id}/samples", {
  path: { id: voice.id },
  body: {
    audio: (await readFile("founder.wav")).toString("base64"),
    filename: "founder.wav",
    transcript:
      "Hi, I'm Jo, and I started this company to make support calls less painful.",
  },
});

const prepared = await api.post("/v1/agents/voices/{id}/prepare", {
  path: { id: voice.id },
  body: {},
});
for (const binding of prepared.bindings ?? []) {
  console.log(binding.provider, binding.state, binding.error ?? "");
}

const preview = await api.post("/v1/agents/voices/{id}/preview", {
  path: { id: voice.id },
  body: { provider: "elevenlabs", text: "Thanks for calling. How can I help?" },
});
await writeFile("preview.mp3", Buffer.from(preview.audio, "base64"));

The name is how the agent will ask for the voice, and it is unique among your voices. Send the recording's filename with its extension: that is how a provider knows what format it was given.

Preparing reports one binding per provider: ready when that provider can speak in the voice, failed with the reason when it refused the recordings. One provider refusing leaves the others usable, and the router simply does not pick that provider for this voice.

prepare teaches every provider this deployment can clone with. Pass providers to choose, for example {"providers": ["elevenlabs", "cartesia"]}; GET /v1/agents/voices/providers lists the ones available. Today they are Cartesia, ElevenLabs, Fish Audio and Inworld.

From a terminal, the CLI makes the same calls. A recording's base64 can be too long for the command line, so --request reads the JSON from a file with @<file>, or from stdin with -:

base64 < founder.wav | tr -d '\n' | jq -Rs '{audio: ., filename: "founder.wav"}' |
  getstream agents api AddVoiceSample --id <id> --request -

Only clone a voice you have the speaker's permission to use. Each provider's terms apply to its copy.

Speak in it

Name the voice in the agent's voice setting with the custom: prefix:

# agents/myagent/agent.yaml
tts: elevenlabs/eleven_v3_conversational
voice: custom:founder

The prefix says the voice is one of yours. Without it, a name is looked for among your voices first and otherwise passed to the provider as one of its stock voices. With it, a name that is not yours is refused, so a typo cannot quietly become a stranger's voice from a provider's catalogue.

For each call the router asks the provider it routes to for that provider's copy of the voice. If the provider it would pick was never taught the voice, it moves on to one that was, so a voice prepared with two providers can still fail over.

The same name works wherever a voice is named: in the voice option when a session is opened, and in the session settings that swap the voice of a running call.

Change or remove a voice

Call What it does
GET /v1/agents/voices Your voices, newest first
GET /v1/agents/voices/{id} One voice, with its recordings and each provider's binding
PUT /v1/agents/voices/{id} Renames it. What it sounds like does not change
POST /v1/agents/voices/{id}/samples Adds another recording. It is not used until you prepare the voice again
POST /v1/agents/voices/{id}/prepare Teaches the providers again, from every recording. Each provider's previous copy is replaced and removed from it
DELETE /v1/agents/voices/{id} Removes the voice from every provider, then its recordings. Past calls still name it

Recordings are kept, so a provider added later can be taught the voice without asking the speaker to record again. On a self-hosted router they live in the bucket named by voices_bucket_url; without one, voices of your own are unavailable. See Self-hosting.

Options

Option What it asks for
voice The voice to speak with.
languages The language to speak. Some providers infer it from the text when it is absent.
speed Speaking rate, 1 being normal. Each provider has its own range, and a speed outside it is refused.
volume Loudness, 1 being the voice's own.
emotion, style Affect or delivery style, for the providers that take one.
stability How much the voice may vary between chunks. Higher is flatter and more consistent.
similarity How closely a cloned voice tracks its reference.
format The audio encoding of a recording. Live sockets are always PCM.
pronunciations How to say words the voice gets wrong, keyed by the word.
chunk_schedule How much text to buffer before speaking.

What varies between providers

  • The best voices have the fewest settings. ElevenLabs v3 conversational takes one setting, stability, and only on the first message. Everything else it can do is said in the text, as audio tags such as [whispers].
  • Speed ranges do not overlap. Cartesia takes 0.6 to 1.5, Inworld 0.5 to 1.5 and ElevenLabs flash 0.7 to 1.2. speed is not one number every voice accepts.
  • "Sound urgent" has three shapes. A fixed list of emotions, English only (Cartesia); free prose in brackets that persists until reset (Inworld); tags the model acts out as it reads them (ElevenLabs v3). An emotion string only reaches the first.
  • Live voices take few options. Today the live models serve a target, a voice and a language. To steer one further, use its overwrites, for example inworld: {delivery_mode: STABLE}.