# Open models and fine-tuning Open-weight models are now close to the closed frontier, cost a fraction of it per token, and can be fine-tuned on your own conversations. The router serves the strongest of them out of the box, and anything else you can reach over an OpenAI-compatible endpoint, a fine-tune included, plugs in as [your own model](#bring-your-own-model). ## Use an open model Name it as `provider/model` wherever an agent takes a model: ```yaml title="agent.yaml" llm: together/zai-org/GLM-5.3-Flash ``` When several hosts serve the same weights, list them and the router fails over between them in that order: ```yaml title="routers/open.yaml" llm: providers: - baseten/zai-org/GLM-5.3-Flash - together/zai-org/GLM-5.3-Flash ``` Thinking is off by default for open models, because a voice agent cannot wait for it. See the [LLM router](https://getstream.io/agents/docs/ios/llm/) for options and fallback, and [Inference Hardware & Hosting](https://getstream.io/agents/docs/ios/inference-hardware-hosting/) for what each host is good for. ## Top open models The open-weight leaders on the [Artificial Analysis Intelligence Index](https://artificialanalysis.ai/models/open-source) (v4.3.2, October 2026). Scores are measured with each model's maximum reasoning setting, so expect a little less with thinking off. | Model | Index | Parameters (active) | Built-in target | | --------------------------- | ----- | ------------------- | ----------------------------------------------------------------- | | MiMo-V2.6-Pro (Xiaomi) | 46 | 1.0T (42B) | [Your own model](#bring-your-own-model) | | GLM-5.3 (Z AI) | 45 | 753B (40B) | `baseten/zai-org/GLM-5.3`, `together/zai-org/GLM-5.3` | | Kimi K3 (Kimi) | 44 | 2.8T (104B) | `baseten/moonshotai/Kimi-K3`, `together/moonshotai/Kimi-K3` | | GLM-5.3-Flash (Z AI) | 42 | 320B (18B) | `baseten/zai-org/GLM-5.3-Flash`, `together/zai-org/GLM-5.3-Flash` | | Qwen3.8 Flash (Alibaba) | 40 | 125B (6B) | `together/Qwen/Qwen3.8-Flash` | | DeepSeek V4.1 Flash | 39 | 552B (16B) | `together/deepseek-ai/DeepSeek-V4.1-Flash` | | Qwen3.8 27B (Alibaba) | 34 | 27B (dense) | [Your own model](#bring-your-own-model) | | K2 Horizon (IFM) | 31 | 375B (23B) | [Your own model](#bring-your-own-model) | | MiniMax-M3 | 29 | 428B (23B) | `together/MiniMaxAI/MiniMax-M3` | | Inkling (Thinking Machines) | 25 | 975B (41B) | [Your own model](#bring-your-own-model) | | Nemotron 3 Ultra (NVIDIA) | 23 | 550B (55B) | `baseten/nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B` | The router also serves DeepSeek V4 Pro, Qwen3.8 2.4T and Gemma 4. Models marked "your own model" are served by hosts such as DeepInfra and Novita, or by your own GPUs; register the endpoint as below and they route like the rest. ## Fine-tuning Fine-tuning pays off when prompting has stopped helping: a fixed output format, a domain vocabulary, a tone, or a small model that has to behave like a large one on your task. Start from a strong small model (Qwen3.8 27B, Gemma 4, GLM-5.3-Flash) and a LoRA adapter, which trains in hours and serves next to the base model. Your data is usually already here: [sessions](https://getstream.io/agents/docs/ios/sessions/) keep every turn, so the conversations that went well are your first training set. Tutorials: - [Unsloth fine-tuning guide](https://docs.unsloth.ai/get-started/fine-tuning-llms-guide): LoRA and QLoRA on one GPU, the fastest way to a first adapter. Its [RL guide](https://docs.unsloth.ai/basics/reinforcement-learning-rl-guide) covers GRPO - [Hugging Face TRL SFT trainer](https://huggingface.co/docs/trl/sft_trainer) and [LoRA Without Regret](https://huggingface.co/docs/trl/main/en/lora_without_regret), on when LoRA matches full fine-tuning - [Axolotl](https://docs.axolotl.ai/): config-driven training for multi-GPU runs - [Tinker](https://tinker-docs.thinkingmachines.ai/): a training API from Thinking Machines; you write the loop, they run the GPUs - Managed fine-tuning, which also serves the result: [Together](https://docs.together.ai/docs/fine-tuning-quickstart) ([deploying it](https://docs.together.ai/docs/deploying-a-fine-tuned-model)), [Fireworks](https://docs.fireworks.ai/fine-tuning/fine-tuning-models), [Baseten Training](https://docs.baseten.co/training/overview) and [Nebius Token Factory](https://docs.tokenfactory.nebius.com/post-training/overview) - Serving it yourself: [vLLM's OpenAI-compatible server](https://docs.vllm.ai/en/latest/serving/openai_compatible_server.html), with [LoRA adapters](https://docs.vllm.ai/en/latest/features/lora.html), or [SGLang](https://docs.sglang.ai/) Whichever you pick, the result is an OpenAI-compatible endpoint, which is all the router needs. ## Bring your own model Register any OpenAI-compatible chat completions endpoint once, then name it as `custom/`: ```bash label="cURL" curl -sS -X POST "$STREAM_ACCELERATION_URL/v1/agents/models" \ -H "api_key: $STREAM_API_KEY" -H "stream-auth-type: jwt" \ -H "Authorization: Bearer $SERVER_TOKEN" \ -H "Content-Type: application/json" \ -d '{ "name": "support-qwen", "base_url": "https://model-abc123.api.baseten.co/environments/production/sync/v1", "model": "Qwen/Qwen3.8-27B", "api_key": "'"$BASETEN_API_KEY"'", "context_window": 262144, "per_million_input_tokens": 0.2, "per_million_output_tokens": 0.8, "trains_on_data": "no", "retention": "none" }' ``` ```go label="Go" key := os.Getenv("BASETEN_API_KEY") window := int64(262144) _, err := client.DefineModel(ctx, acceleration.CustomModelRequest{ Name: "support-qwen", BaseUrl: "https://model-abc123.api.baseten.co/environments/production/sync/v1", Model: "Qwen/Qwen3.8-27B", ApiKey: &key, ContextWindow: &window, }) ``` ```yaml title="agent.yaml" llm: custom/support-qwen ``` | Field | Meaning | | ------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------- | | `name` | What you call it, unique in your app. Lowercase letters, digits, `.`, `_` and `-` | | `base_url` | The endpoint root, up to and including `/v1` | | `model` | The id the endpoint serves the weights under | | `api_key` | Sent as a bearer token. Encrypted, never returned. Left out of an update keeps it; empty removes it | | `context_window` | Tokens it accepts, so a long conversation is compacted before it overflows | | `input_modalities` | Inputs beyond text, such as `image` | | `per_million_input_tokens`, `per_million_output_tokens` | What your host bills, in USD, so usage reports what a conversation cost | | `trains_on_data`, `retention` | The host's data handling. Without both, a session with a [data policy](https://getstream.io/agents/docs/ios/router-intro/#data-policy) never routes here | A custom model works everywhere a model does: in `agent.yaml`, a session's `llm`, a router's `providers` list (put a built-in model after it as the fallback), and a policy's `allowed_models`. Only your app can name its models. `GET`, `PUT` and `DELETE /v1/agents/models/{id}` read, replace and remove one. A shared router only calls public `https` endpoints. On a [self-hosted router](https://getstream.io/agents/docs/ios/self-hosting/) you can also reach a model on your own network, such as vLLM on the next machine, by allowing private endpoints: ```yaml title="/etc/router.yaml" models: private_endpoints: true # or ROUTER_MODELS_PRIVATE_ENDPOINTS=true ``` Only turn this on for a router no one else's apps share, since it lets any app point the router at your network. Keys are encrypted with `ROUTER_AUTH_KEK`, so a model with an `api_key` needs one set. The register calls above, sent to a router you run yourself, take `X-Api-Key` and `Stream-Auth-Type: server` in place of `api_key` and `stream-auth-type: jwt`. --- For the most recent version of this documentation, visit [https://getstream.io/agents/docs/ios/open-models-finetuning/](https://getstream.io/agents/docs/ios/open-models-finetuning/).