The LLM API Failover Pattern I Use in Production

Every LLM provider I have ever used has gone down on me. Not once. Repeatedly. OpenAI, Anthropic, Groq, all of them, at some point in the last two years, threw 500s or timed out mid-request while a customer was waiting on the other end. If your product depends on one API key to one endpoint, you are one incident away from an angry email.

I run consulting projects where “the AI feature is down” is not an acceptable sentence to say to a client. So over the last year I built a failover pattern that has kept things running through three separate provider outages. Here is exactly how it works.

Tier your models before you tier your providers

The first mistake I made was treating “failover” as one flat list of providers. It is not. A chatbot answering FAQs and a code-review agent doing multi-step reasoning have different needs, and they fail differently.

I split my usage into three tiers:

• Frontier tier: complex reasoning, long context, agentic tool use. Slow is acceptable. Wrong is not.
• Mid tier: summarization, classification, most chat traffic. Needs to be fast and cheap, quality can flex a bit.
• Fast tier: autocomplete-style calls, real-time suggestions. Latency budget is under 500ms or the feature is useless.

Each tier gets its own priority chain of providers, not a shared one. A provider that is great for fast-tier throughput (say, Groq or Cerebras on their LPU hardware) is a poor fit for frontier-tier reasoning depth. Mixing them into one list means your failover logic optimizes for the wrong thing half the time.

The health probe, not the error

Here is where most failover code I have reviewed for clients goes wrong: it waits for a request to fail before switching providers. That is reactive, and it means every single user request during an outage eats the full timeout before falling back. At 30 seconds per timeout, that is a terrible experience for the first few dozen users while your system “discovers” the outage.

Instead, run a lightweight background probe every 10-15 seconds per provider. Not a full completion request, just a minimal one, maybe 5 tokens of output, hitting the actual inference endpoint (not a status page - status pages lag reality by minutes, sometimes hours).

async def probe(provider):
    try:
        start = time.monotonic()
        resp = await provider.complete(
            messages=[{"role": "user", "content": "ping"}],
            max_tokens=5,
            timeout=3.0
        )
        latency = time.monotonic() - start
        return HealthState(ok=True, latency=latency)
    except Exception:
        return HealthState(ok=False, latency=None)

Store the result in memory (Redis if you run multiple app instances) with a short TTL. Every real user request checks this cached health state first, then routes. No user ever waits out a timeout to discover a provider is down. The probe already knew.

Defining “healthy” is harder than it sounds

A provider that responds in 200ms is healthy. A provider that responds in 200ms with a garbled or truncated completion is not, and a raw uptime check will never catch that second case.

I define three states, not two:

1. Healthy - probe succeeds, latency under the tier’s threshold.
2. Degraded - probe succeeds but latency is 2-3x the rolling average, or the completion is truncated/malformed.
3. Down - probe fails outright, or times out.

Degraded providers stay in rotation but drop to the back of the priority chain. This matters more than it sounds like it should. I have seen providers stay technically “up” (200 status code, valid JSON) while quietly serving completions at 4x normal latency for an hour during a capacity crunch. Pure up/down health checks miss that entirely, and your users just experience a slow app with no idea why.

Routing: first healthy in the chain, always

Once you have per-tier priority chains and real health state, routing is almost boring, which is the point.

def route(tier: str) -> Provider:
    for provider in PRIORITY_CHAINS[tier]:
        state = health_cache.get(provider.id)
        if state and state.ok and state.latency < THRESHOLDS[tier]:
            return provider
    for provider in PRIORITY_CHAINS[tier]:
        state = health_cache.get(provider.id)
        if state and state.ok:
            return provider  # degraded, but alive
    raise NoProviderAvailable(tier)

Two passes: first for fully healthy providers, then a fallback pass that accepts degraded ones before giving up entirely. Giving up entirely should be rare enough that when it happens, it pages someone.

The thundering herd problem on recovery

This one bit me hard the first time. A provider goes down, all your traffic correctly fails over to the second option in the chain. Twenty minutes later the primary provider comes back. Every single one of your app instances, all checking health at once, sees “healthy” and swings 100% of traffic back immediately.

The primary provider, which just recovered from an outage, gets hit with your full load in one instant and falls right back over. I watched this happen twice before I fixed it.

The fix is a gradual recovery ramp. When a provider flips from down to healthy, do not route to it at full weight immediately. Ramp it in: 10% of eligible traffic for the first two minutes, 50% for the next five, full weight after that, and roll back to the fallback immediately if error rate ticks up during the ramp. It is the same idea as a canary deploy, just applied to a provider you do not control.

What I ended up building

I wrote this failover logic three separate times for three separate client projects before I got annoyed enough to build it once, properly, and reuse it. That became Smart Inference - a router that does risk-aware provider scoring and automatic failover across Fireworks, Cerebras, Groq, Novita, and DeepInfra, among others. It is OpenAI-compatible, so switching an existing integration over is a one-line change to the base URL, not a rewrite.

The honest limitation: it optimizes for availability and cost across providers with genuinely different model weights and quantization behind the same “model name,” so a failover mid-conversation can produce a subtly different completion style than the primary provider would have. For most product use cases that is invisible. For anything where exact model behavior is load-bearing (fine-tuned few-shot prompts tuned against one specific provider’s exact weights, for instance), you will want to pin that call to a single provider and skip the chain entirely.

Where I landed

Tier your models before you tier your providers. Probe actively, do not wait for failures. Treat “slow” as a real health state, not just “up” or “down.” Route in two passes, healthy first, degraded second. And when a provider recovers, ramp it back in instead of slamming it.

None of this is exotic engineering. It is mostly just admitting, early, that every provider will go down eventually, and building for that instead of hoping it will not happen this quarter.