STT

Speech to text models are improving daily. It's hard to keep up with the new model launches. That's why it's generally best to use a router.

STT models fall into two groups. Models are either optimized for speed or accuracy. For voice AI you want to use the fast models. For adding transcriptions to movies, clips etc it's typically better to use the slower and more accurate models.

Fast speech to text for voice AI

en-low-latency is the fast, English optimized router. It routes to whatever is currently state of the art for speed and English.

Name it as the target of your router:

routers/my-router-name.yaml
name: my-router-name

stt:
  target: en-low-latency

Then open the router from your server:

Node.js
import { Socket, flag, text } from "@stream-io/stream-agents";

const live = await Socket.open(
  client.backend,
  await client.backend.socketURL("/v1/stt/stream"),
);
try {
  live.send({ type: "start", config_id: "my-router-name", stt: {} });

  live.sendAudio(pcm); // 16-bit PCM at 16 kHz, sent chunk by chunk as it is captured
  for await (const transcript of live.messages()) {
    if (transcript instanceof Uint8Array) continue;
    if (transcript.type === "transcript" && flag(transcript, "final")) {
      console.log(text(transcript, "text"));
    }
  }
} finally {
  live.close();
}
Python
from vision_agents.core.stt import Transcript
from vision_agents.plugins import stream

client = stream.Client()
router = client.router("my-router-name")

async with router.stt.realtime() as live:
    await live.process_audio(pcm, participant)  # resampled to 16 kHz for you
    async for transcript in live.output:
        if isinstance(transcript, Transcript) and transcript.final:
            print(transcript.text)
Ruby
router = api.router("my-router-name")

router.stt.realtime do |live|
  live.send_audio(pcm) # 16-bit PCM at 16 kHz, sent chunk by chunk as it is captured
  live.each do |frame|
    puts frame["text"] if frame["type"] == "transcript" && frame["final"]
  end
end
PHP
$live = $client->router('my-router-name')->stt->realtime();
try {
    $live->sendAudio($pcm); // 16-bit PCM at 16 kHz, sent chunk by chunk as it is captured
    foreach ($live->frames() as $frame) {
        if (is_array($frame) && $frame['type'] === 'transcript' && ($frame['final'] ?? false) === true) {
            echo $frame['text'], "\n";
        }
    }
} finally {
    $live->close();
}
Go
live, err := client.Router("my-router-name").STT().Realtime(ctx, nil)
if err != nil {
	return err
}
defer live.Close()

if err := live.Send(pcm); err != nil { // 16-bit PCM at 16 kHz, sent chunk by chunk as it is captured
	return err
}
for transcript := range live.Transcripts() {
	if transcript.Final {
		fmt.Println(transcript.Text)
	}
}
.NET
await using var live = await client.Router("my-router-name").Stt.RealtimeAsync();

await live.SendAsync(pcm); // 16-bit PCM at 16 kHz, sent chunk by chunk as it is captured
await foreach (var transcript in live.ReadAllAsync())
{
    if (transcript.Final)
    {
        Console.WriteLine(transcript.Text);
    }
}
Rust
use vision_agents::types::SttOptions;

let mut live = client
    .router("my-router-name")
    .transcriber(SttOptions::default())
    .await?;

live.send_audio(&pcm).await?; // 16-bit PCM at 16 kHz, sent chunk by chunk as it is captured
while let Some(frame) = live.next().await {
    let frame = frame?;
    if frame.kind() == "transcript" && frame.flag("final") {
        println!("{}", frame.text("text"));
    }
}
live.close().await;

For multi language use multilingual-low-latency.

Accurate, slower speech to text

en-recorded is the accurate, English optimized route for files. Give it its own router:

routers/my-recordings.yaml
name: my-recordings

stt:
  target: en-recorded

Hand over a whole recording, audio or video, and get one transcript back:

Node.js
import { readFile } from "node:fs/promises";

let transcript = await client.post("/v1/stt/recordings", {
  body: {
    config_id: "my-recordings",
    source: { audio: (await readFile("interview.mp4")).toString("base64") },
  },
});
while (transcript.status === "queued" || transcript.status === "running") {
  await new Promise((resolve) => setTimeout(resolve, 1000));
  transcript = await client.get("/v1/stt/recordings/{id}", {
    path: { id: transcript.id },
  });
}
console.log(transcript.text);
Python
from vision_agents.plugins import stream

router = stream.Client().router("my-recordings")

transcript = await router.stt.recording("interview.mp4")
print(transcript.text)
Ruby
router = api.router("my-recordings")

transcript = router.stt.recording("interview.mp4")
puts transcript["text"]
PHP
use GetStream\VisionAgents\Generated\RecordingSource;

$transcript = $client->router('my-recordings')->stt->recording(
    new RecordingSource(audio: base64_encode(file_get_contents('interview.mp4'))),
);
echo $transcript->text, "\n";
Go
recording, err := stream.File("interview.mp4")
if err != nil {
	return err
}
transcript, err := client.Router("my-recordings").STT().Recording(ctx, recording, nil)
if err != nil {
	return err
}
fmt.Println(*transcript.Text)
.NET
var transcript = await client.Router("my-recordings").Stt.RecordingAsync(Recorded.File("interview.mp4"));
Console.WriteLine(transcript.Text);
Rust
use vision_agents::Recording;
use vision_agents::types::SttOptions;

let transcript = client
    .router("my-recordings")
    .transcribe(
        Recording::Audio(std::fs::read("interview.mp4")?),
        SttOptions::default(),
        None,
    )
    .await?;
println!("{}", transcript.text.unwrap_or_default());

There is also a multi language optimized router, multilingual-recorded.

Custom STT router

You can also create your own router with a custom set of models. This config tries Deepgram Flux first, then Cartesia Ink 2, and ends with the en-low-latency group.

name: live-support

stt:
  providers: [deepgram/flux-general-en, cartesia/ink-2, en-low-latency]

The model used will automatically switch from deepgram to cartesia when deepgram has an outage.

Options

Option Path What it asks for
target, providers both A route, provider or provider/model, or a priority list of them.
languages both ISO codes the model must cover.
mode both verbatim keeps the ums and false starts; smart removes them and tidies the grammar.
detect_language both Let the provider identify the language instead of being told it.
sample_rate realtime Rate of the PCM you send. Zero means 16 kHz.
interim realtime Partial transcripts as they firm up, not only final ones.
endpointing realtime How a turn is judged over: silence for a pause, or semantic for a model reading the words.
silence_ms, utterance_end_ms realtime How long a pause ends a turn, for pause endpointing.
diarize both Label each stretch of speech with who said it.
max_speakers both The most speakers to tell apart.
keyterms both Words to listen for: names, drugs, product terms.
format both Punctuation, capitalisation and smart formatting of numbers and dates.
redact both Remove personally identifying information.
profanity_filter both Mask profanity.
events both Tag non-speech audio such as laughter or music.
words recording Word-level timestamps.
summary, entities recording A summary, and the named entities in the recording.
channels recording Transcribe each channel of a multichannel file separately.
output recording srt or vtt subtitles, rendered by the router from the word timings.

What varies between providers

Not every provider can do every option, and the router refuses rather than drops one it cannot serve:

  • Diarization. Live, only Grok STT and Muse label speakers; for a recording, Deepgram Nova 3 does. No model can yet be told how many speakers there are, so max_speakers is refused.
  • Endpointing. Deepgram Flux, Cartesia Ink 2, Inworld and ElevenLabs Scribe take endpointing settings. The rest decide where a turn ends themselves.
  • Vocabulary. Every live model takes keyterms except Parakeet and Nemotron. A call full of drug names is routed away from those two.
  • Recording-only features. Word timings, summaries, entities, redaction, formatting, channels and language detection are served by the batch model, Deepgram Nova 3, so asking for them live is refused.
  • Not served yet. No model in the catalogue offers interim or events yet, so both are refused for now. mode: verbatim reaches Gemini and Grok, and mode: smart only Gemini.