Inference hardware and hosting

This page is for teams running open-weight models behind a voice or chat agent. The agents backend, the router, routes each turn to a model, and that model has to run somewhere: a hosted API, a dedicated deployment of your own weights, or GPUs you rent and operate. Below is what hardware each class of model needs in 2026, roughly what it costs, which hosts are worth knowing, and how each one plugs into Stream agents.

Who this is for

A live voice conversation is judged on two numbers per stream: time to first token, which decides how long the caller waits after they stop talking, and tokens per second, which has to stay ahead of the speech it is turned into. Concurrency comes third: how many of those streams one deployment holds at once before either number slips. A chat agent is more forgiving, but the same three numbers apply.

Sizing starts from four things:

• Total and active parameters. Most large open models are mixture-of-experts (MoE). Every expert has to sit in memory, so memory is set by the total, but only the active parameters are read per token, so speed is set by the active count. A 750B model with 40B active needs the memory of a 750B model and generates at roughly the speed of a 40B one.
• Weight precision. Bytes per parameter: about 2 at BF16, 1 at FP8, 0.5 at FP4 or INT4. A 27B dense model is about 56 GB at BF16, 28 GB at FP8 and 15 GB at 4-bit.
• KV cache. Every token of context in every live stream holds memory on top of the weights. Long system prompts, tool schemas and long conversations multiply it by the number of concurrent calls.
• Latency, not batch throughput. Benchmarks that quote tokens per second for a whole GPU at large batch sizes describe cost per token, not what one caller hears. A voice agent wants small batches and fast single-stream decode, so it usually runs on more hardware per user than an offline batch job would.

Inference hardware

Prices below are on-demand rental rates per GPU-hour, as listed in October 2026. They move month to month, differ by a factor of two or more between clouds for the same card, and reserved or committed terms are commonly 25 to 45% lower.

Desk: laptop, workstation or one prosumer GPU

An RTX 5090 is the fastest single-stream option for anything that fits in its 32 GB: a 4-bit Qwen3.8-27B decodes at around 60 to 75 tokens per second. The catch in 2026 is price: its list price is $1,999, but memory shortages have pushed street prices to around $4,600. A Mac Studio M5 Ultra (from $5,499 with 96 GB, $9,499 and up with 256 GB) trades some speed for memory, which lets it hold MoE models a 5090 cannot load. A DGX Spark ($4,699, 128 GB) runs the CUDA stack, vLLM and SGLang included, but its 273 GB/s of memory bandwidth makes single-stream decode the slowest of the three.

This tier makes sense for development, demos and a single internal user. It does not serve production calls: one box holds a handful of streams at best, and has no failover.

One datacenter GPU or a small node

An L40S runs a 27B model at FP8 for a few concurrent calls. An H100 or H200 runs the same model with headroom for many more, and holds small and mid MoE models such as Qwen3.8-Flash-Next (125B total, about 6B active) or gpt-oss-120b. An MI300X has the most memory per card in this group at the lowest median price, provided your serving stack runs well on ROCm.

This is the tier for a dedicated deployment of a fine-tuned small model, or for a fast model whose latency you want to control yourself.

One 8-GPU node

The large open models of 2026 are MoE models between about 750B and 2.8T total parameters:

At FP8, GLM-5.3 needs about 750 GB for weights alone, which an 8x B200 node (1.44 TB) holds with room for KV cache. DeepSeek V4 Pro at FP8 and Kimi K3 at its native 4-bit need the larger memory of 8x B300 (2.3 TB) or 8x MI355X. A node costs roughly $45 to $80 an hour on demand, or $33,000 to $58,000 a month running around the clock, before you account for the second node you want for failover.

This tier makes sense once you have steady traffic on one of these models, or a fine-tune of one that no shared API serves.

Rack scale

GB200 and GB300 NVL72 racks join 72 GPUs into one NVLink domain, which is what the largest models at long context and high concurrency are served on. Vera Rubin NVL72, the successor (it was called NVL144 during development), started shipping to a handful of large clouds in the second half of 2026, and Vera Rubin NVL144 CPX, a long-context variant, is expected at the end of 2026. AMD's equivalent, the Helios rack with 72 MI455X GPUs, is due in volume in the second half of 2026 and is waitlisted today. None of these is sold by the hour to a small team: CoreWeave lists GB200 NVL72 at about $10.50 per GPU-hour, and GB300 and Rubin capacity is quoted per contract.

Specialised inference silicon

Chips built only for inference give the lowest latency per stream available, and you reach them through their own APIs rather than renting the hardware:

The trade is choice: each serves a short, changing list of models, and a fine-tune of your own usually means a dedicated endpoint negotiated with the vendor.

Hosting options

Three ways to get open models served, from least to most work.

Serverless per-token APIs

Shared endpoints for popular open models, billed per token, with nothing to size or scale. Rate limits apply, and a model can be retired with a few weeks' notice.

Dedicated deployments of your own weights

Reserved GPUs running one model for you, billed per GPU-hour whether traffic arrives or not. This is where fine-tunes go, and where latency stops depending on other tenants.

Raw GPU clouds

Machines by the hour, and the serving stack is yours: vLLM, SGLang or TensorRT-LLM, plus autoscaling, health checks and upgrades.

How this plugs into Stream agents

The router routes to these open-weight hosts out of the box once their key is set:

atlascloud, baidu (Qianfan), baseten, cerebras, cloudflare, coreweave, deepinfra, digitalocean, fireworks, gmicloud, inceptron, ionet, nextbit, novita, parasail, phala, sailresearch, siliconflow, together, venice and wafer.

A model target names the host and the model as provider/model:

llm: baseten/zai-org/GLM-5.3-Flash

together/zai-org/GLM-5.3-Flash is the same weights on another host. Because several hosts serve the same open model, the router fails over between them. See the LLM router for how routes are defined.

flowchart LR
    Agent[Agent turn] --> Router
    Router -->|baseten/zai-org/GLM-5.3-Flash| Baseten
    Router -->|together/zai-org/GLM-5.3-Flash| Together[Together AI]
    Router -->|your own model| Own[vLLM or SGLang on your GPUs]

On a self-hosted router, each host reads its key from an environment variable, and a host without one is skipped:

export BASETEN_API_KEY=...
export TOGETHER_API_KEY=...
export FIREWORKS_API_KEY=...
export DEEPINFRA_API_KEY=...
export COREWEAVE_API_KEY=...

To send a host's traffic to a private deployment rather than its shared API, set its base-URL variable, such as TOGETHER_BASE_URL or COREWEAVE_BASE_URL. Baseten's shared Model APIs use BASETEN_INFERENCE_BASE_URL.

Your own weights on your own GPUs

For a model you serve yourself with vLLM, SGLang or any OpenAI-compatible server, register it as your own model: see Bring your own model.

Nebius, Lambda, RunPod and Modal are not built-in hosts. Use them the same way: run an OpenAI-compatible server there, or point at their OpenAI-compatible API, and register it as your own model.

Picking one

• Start on a serverless API. It is the cheapest way to find out which model your agent needs, and the router can fail over between two hosts of the same weights.
• For the lowest latency per stream on a model they serve, try the specialised silicon APIs before buying hardware for speed.
• Move to a dedicated deployment when traffic is steady enough to keep a replica busy, when you fine-tune, or when you need latency that does not depend on other tenants.
• Run your own GPUs only at sustained high utilisation, or when data residency or contracts require it. Budget for at least two replicas and the people to run them.