Gateway basics · 2026-09-22
LLM gateway failover architecture: how high availability is built
How a gateway can route a request to a fallback endpoint when the primary provider goes down, which errors are safe to retry, and the idempotency risk to watch for.
Why failover is different from just switching models
The core distinction covered in the AI gateway vs LLM gateway difference article is that a gateway makes multiple providers reachable from a single interface. Failover is not an automatic consequence of that: it is a separate design decision under which, when the primary provider returns an error (a timeout, a 5xx, a capacity error), the request is rerouted to a predefined fallback endpoint. The fallback is usually another endpoint serving the same model. Some gateway designs instead define a different model of the same class as the fallback; in such a setup the model identifier in the request stays the same, but the response may come from the fallback model at that moment.
What separates this from manual retry logic is that the decision is made on the gateway side, not the client side; your client code never needs to know which endpoint is currently available. This behavior is not present in every model or every gateway; an API that already has redundancy on the provider's own side may not need an extra failover chain at the gateway layer.
Which errors are safe to retry, and which aren't
Timeouts and transient capacity errors (5xx) are usually safe to retry; the same request can succeed on a different provider. An invalid request shape (400) or an authorization error (401/403), on the other hand, is not fixed by switching providers; attempting failover on these only adds latency and masks the real problem.
This distinction requires a gateway to correctly classify errors; a system that retries every error the same way treats a permanent error as if it were transient, producing unnecessary cost and latency.
- Timeouts and 5xx: usually safe to retry and failover on.
- 400/401/403: not fixed by switching providers; attempting failover only adds latency.
- Without correct error classification, failover produces unnecessary cost.
Idempotency: the risk of the same request running twice
If a request was partially processed by the primary provider and then timed out, the same request sent to a second provider via failover carries a risk of double-billing if the first provider actually produced a response but failed to deliver it to you. For an operation with a side effect (for example, a tool triggering a real-world action), failover should only trigger once you're certain the response was not received.
Frequently asked questions
Should failover trigger on every kind of error?
No. Failover only makes sense on errors considered transient (timeouts, 5xx). On permanent errors (invalid request, authorization), switching providers does not fix the problem — it only adds latency.
Does using failover affect model consistency?
It can. Even when the fallback is another endpoint serving the same model, different serving software can produce small format differences; in a design where a different model is defined as the fallback, capabilities, tokenization, tone, and output format can change as well. For flows that need strict consistency, validate the output against a schema on your side and check the model's behavior regularly with your own test set.