Router Intro

Routers are important when working with AI. You can use our routers when setting up an agent. You can also use them independently of the agent, which is what these docs show.

We recommend using a router and naming it after the use case in your app. Names like healthcare_transcriptions, ivy_voice_ai or other project specific names are good. There are a few reasons why you want routers with use case based names like this:

• The cost and performance of models change rapidly. In the past at Stream we've spent tens of thousands of additional dollars just because updating to a new model took time.
• AI providers don't have perfect uptime. They are all dealing with rapid growth and limited GPU availability, so when a vendor goes down you want a fallback.
• The router supports model specific overwrites, so you don't lose much flexibility.

Note that your named router can also point to any of our predefined routes, such as llm-fast or en-low-latency.

Router example

One YAML file sets up transcription, voice and language for a dental office's voice agent:

name: dentist_ai

stt:
  # Deepgram Flux first, then whatever is healthiest in the English low latency group.
  providers: [deepgram/flux-general-en, en-low-latency]
  keyterms: [crown, root canal, periodontal, Invisalign]
  # Patient audio only goes to providers that neither train on it nor keep it.
  data_policy:
    allow_training: false
    retention: none

tts:
  target: en-low-latency
  voice: custom:front_desk

llm:
  providers: [openai/gpt-5-mini, llm-fast]
  reasoning_effort: low

Sync it. The sync command picks up the YAML file in the routers folder, and configs are written by name, so running it twice edits rather than duplicates:

curl -fsSL https://getstream.io/cli/beta/install.sh | bash # the agents commands are in the beta CLI
getstream agents sync --router=dentist_ai

Then open it by name. Each modality uses the defaults from the file:

from vision_agents.plugins import stream

router = stream.Client().router("dentist_ai")

async with router.stt.realtime() as live:
    ...

When a new model is better or cheaper, change this file and sync again. Every call that uses dentist_ai follows, without a code change.

Predefined routes

Name any of these as a target, or inside a providers list. A route tries its preferred model first while it is healthy, and the rest of the group catches it when it is not.

Stream's picks, one set per modality:

Groups by language and speed, with the same names in every modality:

GET /v1/{modality}/routes lists the routes a deployment offers, and GET /v1/{modality}/routes/{target} shows which models a route resolves to right now.

Data policy

Every provider in the catalogue declares what happens to what it is sent: trains_on_data (yes, no or unknown) and retention (none, a duration such as 30d, unspecified or unknown). A config can require a policy:

stt:
  data_policy:
    allow_training: false # only providers that said they do not train on it
    retention: none # only providers that keep nothing

A request is then only routed to providers that meet it. unknown and unspecified meet nothing, because a vendor who has said nothing is not a vendor who has said no. If no provider qualifies, the request is refused rather than sent somewhere that does not qualify.

Daily limits

An app caps what one end user may spend in a UTC day with quota in its policy. Send it with a server-side token:

{
  "quota": {
    "messages_per_user_per_day": 500,
    "tokens_per_user_per_day": 10000000
  }
}

• quota is for an app. An organization policy that sets it is refused with a 400.
• 0 is no cap. A number you leave out takes the deployment's default.
• The PUT replaces the whole policy, so send the rest of the policy too. A PUT without quota clears it, and the deployment's default applies again.

An end user with nothing left gets a 429 rate_limited with a Retry-After header: the seconds until the limit resets. On a socket that is already open, the refusal is an error frame.

An HTTP response to an end user's request carries three headers while a limit applies. A turn on an open socket or call carries none.

The router counts each user id and each address against the limit. The one nearest to being spent decides. End users behind one proxy share an address.

The router counts an end user's own turns, also in a session a backend opened for them. It never counts a backend's own turns, or answers from mock/* models.

Learn more