Open models and fine-tuning

Open-weight models are now close to the closed frontier, cost a fraction of it per token, and can be fine-tuned on your own conversations. The router serves the strongest of them out of the box, and anything else you can reach over an OpenAI-compatible endpoint, a fine-tune included, plugs in as your own model.

Use an open model

Name it as provider/model wherever an agent takes a model:

llm: together/zai-org/GLM-5.3-Flash

When several hosts serve the same weights, list them and the router fails over between them in that order:

llm:
  providers:
    - baseten/zai-org/GLM-5.3-Flash
    - together/zai-org/GLM-5.3-Flash

Thinking is off by default for open models, because a voice agent cannot wait for it. See the LLM router for options and fallback, and Inference Hardware & Hosting for what each host is good for.

Top open models

The open-weight leaders on the Artificial Analysis Intelligence Index (v4.3.2, October 2026). Scores are measured with each model's maximum reasoning setting, so expect a little less with thinking off.

The router also serves DeepSeek V4 Pro, Qwen3.8 2.4T and Gemma 4. Models marked "your own model" are served by hosts such as DeepInfra and Novita, or by your own GPUs; register the endpoint as below and they route like the rest.

Fine-tuning

Fine-tuning pays off when prompting has stopped helping: a fixed output format, a domain vocabulary, a tone, or a small model that has to behave like a large one on your task. Start from a strong small model (Qwen3.8 27B, Gemma 4, GLM-5.3-Flash) and a LoRA adapter, which trains in hours and serves next to the base model.

Your data is usually already here: sessions keep every turn, so the conversations that went well are your first training set.

Tutorials:

• Unsloth fine-tuning guide: LoRA and QLoRA on one GPU, the fastest way to a first adapter. Its RL guide covers GRPO
• Hugging Face TRL SFT trainer and LoRA Without Regret, on when LoRA matches full fine-tuning
• Axolotl: config-driven training for multi-GPU runs
• Tinker: a training API from Thinking Machines; you write the loop, they run the GPUs
• Managed fine-tuning, which also serves the result: Together (deploying it), Fireworks, Baseten Training and Nebius Token Factory
• Serving it yourself: vLLM's OpenAI-compatible server, with LoRA adapters, or SGLang

Whichever you pick, the result is an OpenAI-compatible endpoint, which is all the router needs.

Bring your own model

Register any OpenAI-compatible chat completions endpoint once, then name it as custom/<name>:

curl -sS -X POST "$STREAM_ACCELERATION_URL/v1/agents/models" \
  -H "api_key: $STREAM_API_KEY" -H "stream-auth-type: jwt" \
  -H "Authorization: Bearer $SERVER_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "support-qwen",
    "base_url": "https://model-abc123.api.baseten.co/environments/production/sync/v1",
    "model": "Qwen/Qwen3.8-27B",
    "api_key": "'"$BASETEN_API_KEY"'",
    "context_window": 262144,
    "per_million_input_tokens": 0.2,
    "per_million_output_tokens": 0.8,
    "trains_on_data": "no",
    "retention": "none"
  }'

key := os.Getenv("BASETEN_API_KEY")
window := int64(262144)
_, err := client.DefineModel(ctx, acceleration.CustomModelRequest{
	Name:          "support-qwen",
	BaseUrl:       "https://model-abc123.api.baseten.co/environments/production/sync/v1",
	Model:         "Qwen/Qwen3.8-27B",
	ApiKey:        &key,
	ContextWindow: &window,
})

llm: custom/support-qwen

A custom model works everywhere a model does: in agent.yaml, a session's llm, a router's providers list (put a built-in model after it as the fallback), and a policy's allowed_models. Only your app can name its models. GET, PUT and DELETE /v1/agents/models/{id} read, replace and remove one.

A shared router only calls public https endpoints. On a self-hosted router you can also reach a model on your own network, such as vLLM on the next machine, by allowing private endpoints:

models:
  private_endpoints: true # or ROUTER_MODELS_PRIVATE_ENDPOINTS=true

Only turn this on for a router no one else's apps share, since it lets any app point the router at your network. Keys are encrypted with ROUTER_AUTH_KEK, so a model with an api_key needs one set. The register calls above, sent to a router you run yourself, take X-Api-Key and Stream-Auth-Type: server in place of api_key and stream-auth-type: jwt.