How routing works
You send one request to one URL. We pick who serves it. This page explains the picking.
- The router picks the cheapest working provider for every request
- Providers that are unavailable, incompatible, or too slow are filtered out first
- Spot-served requests can be interrupted — check the
X-SI-Spotheader - You never pay more than the on-demand rate
Provider pool
The router evaluates providers across a growing pool of inference backends. As of June 2026, over a dozen providers are in the pool. New providers are added regularly as they pass health and pricing verification. You never need to manage provider accounts or know which providers are in the pool.
Scoring
For each request the router evaluates every eligible provider and picks the cheapest one that meets reliability and speed requirements.
Filtering
Before scoring, the router removes any provider that:
- Is missing a required capability (function calling, JSON mode, tool choice, etc.)
- Has a context window smaller than your prompt
- Is currently unavailable or unhealthy
- Is too unreliable for this request type
Health probes
The router continuously probes each provider’s availability and latency. When a probe detects an issue, there is a 5-minute grace period before the provider is marked unhealthy. This prevents brief network blips from unnecessarily removing a provider from the candidate list.
Failover
The router tries providers in score order. If the top pick returns an error, the next candidate gets the same request. You see only the final result — or a 502 upstream_error if every candidate fails.
Once a stream starts, the provider is locked. If it fails mid-stream, the error surfaces in the SSE stream and you are billed only for the tokens that already arrived.
Spot capacity
Some requests are served by spot (reclaimable) capacity. These are cheaper but can be interrupted. The X-SI-Spot response header tells you whether a spot provider served your request. See Response headers for the full list.