# LLM

The language model router speaks the shape of OpenAI's Responses API: instructions separate from the input, one stream per response, and tool calls as events. Providers that only offer chat completions are adapted, so every model behind a route is asked the same way.

There is one path. A completion is whole by the time it is returned, so there is no recording; what the socket buys is the answer arriving as it is written.

## Ask a question

```ts label="JavaScript"
import { Socket, text, type Schemas } from "@stream-io/calypso";

const model = await Socket.open(
  client.backend,
  await client.backend.socketURL("/v1/llm/stream"),
);
try {
  model.send({
    type: "start",
    config_id: "support",
    llm: { reasoning_effort: "low" } satisfies Schemas["LlmOptions"],
  });

  model.send({
    type: "respond",
    instructions: "You are a friendly support agent. Keep replies short.",
    messages: [{ role: "user", content: "Can I change my plan mid-month?" }],
  });
  for await (const answer of model.messages()) {
    if (answer instanceof Uint8Array) continue;
    if (answer.type === "error")
      throw new Error(`answering: ${text(answer, "error")}`);
    if (answer.type === "delta") process.stdout.write(text(answer, "text"));
    if (answer.type === "complete") break;
  }
} finally {
  model.close();
}
```

```python label="Python"
from vision_agents.core.llm.llm import LLMResponseDelta
from vision_agents.plugins import stream

router = stream.Client().router("support")

async with router.llm.realtime(reasoning_effort="low") as model:
    model.set_instructions("You are a friendly support agent. Keep replies short.")
    async for answer in model.simple_response("Can I change my plan mid-month?"):
        if isinstance(answer, LLMResponseDelta):
            print(answer.delta, end="", flush=True)
```

```ruby label="Ruby"
router = api.router("support")

router.llm.realtime(reasoning_effort: "low") do |model|
  model.respond("Can I change my plan mid-month?",
                instructions: "You are a friendly support agent. Keep replies short.") do |delta|
    print delta
  end
end
```

```php label="PHP"
use GetStream\VisionAgents\Generated\LlmOptions;

$model = $client->router('support')->llm->realtime(new LlmOptions(reasoningEffort: 'low'));
try {
    $model->respond('plan', [
        'instructions' => 'You are a friendly support agent. Keep replies short.',
        'messages' => [['role' => 'user', 'content' => 'Can I change my plan mid-month?']],
    ]);
    foreach ($model->frames() as $frame) {
        if (!is_array($frame)) {
            continue;
        }
        if ($frame['type'] === 'delta') {
            echo $frame['text'];
        }
        if ($frame['type'] === 'complete') {
            break;
        }
    }
} finally {
    $model->close();
}
```

```go label="Go"
effort := acceleration.LlmOptionsReasoningEffortLow
model, err := client.Router("support").LLM().Realtime(ctx, &acceleration.LlmOptions{
	ReasoningEffort: &effort,
})
if err != nil {
	return err
}
defer model.Close()

if err := model.Ask(stream.Question{
	Instructions: "You are a friendly support agent. Keep replies short.",
	Messages:     []stream.Said{{Role: "user", Content: "Can I change my plan mid-month?"}},
}); err != nil {
	return err
}
for answer := range model.Answers() {
	if answer.Error != "" {
		return fmt.Errorf("answering: %s", answer.Error)
	}
	fmt.Print(answer.Delta)
	if answer.Done {
		break
	}
}
```

```csharp label="C#"
await using var model = await client.Router("support").Llm.RealtimeAsync(new LlmOptions { ReasoningEffort = "low" });

await model.AskAsync(new Question
{
    Instructions = "You are a friendly support agent. Keep replies short.",
    Messages = [new Said("user", "Can I change my plan mid-month?")],
});
await foreach (var answer in model.ReadAllAsync())
{
    if (answer.Error != "")
    {
        throw new InvalidOperationException($"answering: {answer.Error}");
    }
    Console.Write(answer.Delta);
    if (answer.Done)
    {
        break;
    }
}
```

```rust label="Rust"
use serde_json::json;
use vision_agents::types::{LlmOptions, LlmOptionsReasoningEffort};
use vision_agents::Ask;

let mut model = client
    .router("support")
    .completions(LlmOptions {
        reasoning_effort: Some(LlmOptionsReasoningEffort::Low),
        ..Default::default()
    })
    .await?;

let ask = Ask {
    instructions: "You are a friendly support agent. Keep replies short.".into(),
    messages: vec![json!({"role": "user", "content": "Can I change my plan mid-month?"})],
    ..Default::default()
};
model.respond(&ask, |delta| print!("{delta}")).await?;
model.close().await;
```

A message can carry images alongside its text, for the vision routes such as `vlm`.

## Options

| Option              | What it asks for                                                                 |
| ------------------- | -------------------------------------------------------------------------------- |
| `max_output_tokens` | The longest answer, in tokens.                                                   |
| `temperature`       | Sampling randomness. Leave it unset: most current models ignore it or refuse it. |
| `reasoning_effort`  | How long the model may think before answering, on the models that think.         |
| `format`            | Whether the answer is prose or a JSON object.                                    |
| `verbosity`         | How long-winded the answer is, on the models that take it.                       |
| `tool_choice`       | Whether the model may, must or must not call a tool.                             |
| `store`             | Keep the response on the provider so a later one can continue from it.           |
| `prompt_cache_key`  | Groups requests that share a prefix, so the provider can cache it.               |
| `metadata`          | Passed to the provider untouched, for the providers that store it.               |

## What varies between providers

- **Thinking is not one scale.** The effort ladders run from three rungs to seven, and some models cannot be told not to think at all. Each model declares the efforts it accepts, and one it does not accept is refused rather than passed on to fail halfway through a call.
- **Temperature is being withdrawn.** Anthropic rejects anything but 1.0 on recent models, OpenAI removes it on reasoning models, and DeepSeek ignores it while thinking. Saying nothing is the only portable choice.
- **Caching is four unrelated mechanisms.** A key you choose, breakpoints you place, a prefix match you get for free, or a cache created as a resource. `prompt_cache_key` is the first; a cache lifetime a provider does not offer is dropped, since it changes only the bill.
- **The conversation lives with the caller.** OpenAI, Grok and Muse can hold a conversation for you; Anthropic and DeepSeek cannot. Sending the conversation whole every time is what lets a turn fail over to any provider, and what lets consecutive turns be answered by different ones.

## Fallback

A failed request is retried on the next candidate when the provider answers with:

| Response                          | Falls back? | Why                                                                    |
| --------------------------------- | ----------- | ---------------------------------------------------------------------- |
| 5xx, network error, timeout (408) | Yes         | The provider is down or slow.                                          |
| 429 rate limited                  | Yes         | Another provider has capacity.                                         |
| 401, 403, 404                     | Yes         | That provider's credentials or model are unavailable here.             |
| Other 4xx, such as 400            | No          | The request itself is wrong, and another provider would reject it too. |
| Request cancelled by the caller   | No          | Nobody is waiting for the answer.                                      |

Two more rules keep fallback honest:

- **A stream that has started is not replayed.** Once tokens have reached the caller, a failure ends that response rather than restarting it on another model, so a caller never hears an answer twice.
- **Continuing a stored response stays on its provider.** A request that continues a previous response cannot move, because the other providers do not hold that state.

---

For the most recent version of this documentation, visit [https://getstream.io/agents/docs/javascript/llm/](https://getstream.io/agents/docs/javascript/llm/).