Gateway basics · 2026-09-22
LLM gateway failover architecture: how high availability is built
How a gateway can route a request to a fallback upstream, for models that depend on a single upstream, when that upstream goes down — which errors are safe to retry, and the idempotency risk to watch for.
Why failover is different from just switching models
The core distinction covered in the AI gateway vs LLM gateway difference article is that a gateway makes multiple providers reachable from a single interface. Failover is not an automatic consequence of that; for specific models that depend on a single upstream on their own end (for example, a self-hosted box or a single third-party route), it means that when the primary upstream returns an error (a timeout, a 5xx, a capacity error), the request is rerouted to a predefined fallback upstream. The fallback tier is usually another server hosting the same model, but that is not guaranteed. On some LLMTR rows the fallback tier is a different model of the same class: the model identifier in your request stays the same, but the response may come from the fallback model at that moment.
What separates this from manual retry logic is that the decision is made on the gateway side, not the client side; your client code never needs to know which upstream is currently available. This behavior is not guaranteed for every model or every gateway — a model backed by a provider API that already has its own vendor-side redundancy typically doesn't need an extra failover chain at the gateway layer.
Which errors are safe to retry, and which aren't
Timeouts and transient capacity errors (5xx) are usually safe to retry; the same request can succeed on a different provider. An invalid request shape (400) or an authorization error (401/403), on the other hand, is not fixed by switching providers; attempting failover on these only adds latency and masks the real problem.
This distinction requires a gateway to correctly classify errors; a system that retries every error the same way treats a permanent error as if it were transient, producing unnecessary cost and latency.
- Timeouts and 5xx: usually safe to retry and failover on.
- 400/401/403: not fixed by switching providers; attempting failover only adds latency.
- Without correct error classification, failover produces unnecessary cost.
Idempotency: the risk of the same request running twice
If a request was partially processed by the primary provider and then timed out, the same request sent to a second provider via failover carries a risk of double-billing if the first provider actually produced a response but failed to deliver it to you. For an operation with a side effect (for example, a tool triggering a real-world action), failover should only trigger once you're certain the response was not received.
Frequently asked questions
Should failover trigger on every kind of error?
No. Failover only makes sense on errors considered transient (timeouts, 5xx). On permanent errors (invalid request, authorization), switching providers does not fix the problem — it only adds latency.
Does using failover affect model consistency?
It can. The model identifier in your request doesn't change, but the fallback tier is not always the same model; on some rows the fallback is a different model and the response comes from it. In that case capabilities, tokenization, tone, and output format can change. Even when the fallback is the same model, different serving software can produce small format differences. For flows that need strict consistency, validate the output against a schema on your side and consider running them on a model that doesn't depend on a fallback chain.