How to add automatic model fallback to your AI app

Automatic model fallback routes an AI request to a pre-configured backup model the moment the primary fails — without changing your application code or surfacing the failure to users. It handles hard failures (5xx), rate limits (429), and latency timeouts, keeping your AI feature running through provider outages.

Why every AI app needs a fallback

Every major LLM provider — OpenAI, Anthropic, Google — has experienced outages and rate-limiting incidents. The arithmetic is unforgiving. OpenAI’s pay-as-you-go API carries a 99.5% uptime SLA. That sounds high, but 99.5% still allows up to 43.8 hours of downtime per year. Measured reality can be lower: one analysis of Claude’s status dashboard put its 30-day uptime at 99.32%, translating to nearly 5 hours of downtime per month.

AI model downtime is also qualitatively different from traditional infrastructure downtime. A regional cloud outage has geographic failover. LLM provider downtime can be global, with no geographic failover available, hitting all users simultaneously.

If your product feature is a single await openai.chat(...) call with no backup, that dependency is your new single point of failure. When the model goes down, your feature goes down with it.

The three failure modes you must handle

Not all failures look alike. Your fallback strategy needs to cover all three:

Hard failures — HTTP 503, 502, or a provider going fully dark. The API doesn’t respond or returns an explicit server error. Easy to detect; the danger is having no path after detection.

Rate limits — HTTP 429. Often short-lived (seconds to minutes), but long enough to break a user session during peak load. Transient in theory; operationally persistent enough to need a backup.

Silent quality failures — the API returns 200 OK, but the output is wrong. Standard HTTP error tracking misses this entirely. Your circuit breaker doesn’t trip. Your alerts stay green. Users just get bad answers until someone notices.

Handling 5xx and 429 with retries is table stakes. The hard problem is catching quality degradation before your users do — and that requires eval gates, not just error handling.

What you’d have to build yourself

The DIY path is longer than it looks:

1. Wrap every AI call in try-catch
2. Configure SDKs for each provider you want as a backup
3. Add retry logic with exponential backoff before the fallback triggers
4. Implement cooldown logic so a rate-limited model isn’t retried immediately
5. Add latency-based timeouts (fall back if the primary takes >8s)
6. Log which model served each request for observability
7. Repeat for every new AI function in your codebase

LiteLLM makes this easier by adding a fallbacks parameter to standard completion calls:

# LiteLLM: primary fails after num_retries, then tries fallbacks in order
response = completion(
    model="claude-opus-4-8",
    messages=messages,
    fallbacks=["gpt-4o", "gemini-2-pro"]
)

This works. But you’re still running infrastructure — the proxy, its routing config, provider credentials, health-check tuning, and the observability layer — and maintaining all of it as your model list changes. And it still doesn’t tell you whether your fallback model produces equivalent-quality output. It just routes.

Declare it, don’t code it

fnAI treats fallback as a runtime concern. You declare a primary and a backup in fn.yml. The runtime monitors every call, detects failures, and switches — without any changes to your application code.

# fn.yml
name: classify
model: claude-opus-4-8
fallback: claude-sonnet-4-6
eval:
  dataset: tickets.jsonl
  threshold: 0.95

When the primary fails, the runtime switches within the same request cycle:

$ fn tail classify@v3 --events

12:04:01  ✓  primary  claude-opus-4-8    142ms  ok
12:04:05  ✗  primary  claude-opus-4-8    ——     503 unavailable
12:04:05  ↺  failover claude-sonnet-4-6   91ms  ok  ← auto
12:04:06  ✓  fallback claude-sonnet-4-6   96ms  ok

incident: opus-4-8 unavailable 41s · 7 requests · 0 user errors
your function never stopped answering.

Zero application code changed. Zero user errors.

The part a gateway alone can’t give you

The eval gate is what makes this different from a pure routing solution. When you run fn eval classify@v3, fnAI measures your fallback model against the same dataset and threshold as your primary:

$ fn eval classify@v3 --dataset tickets.jsonl

✓ 47 / 48 cases passed
✗ case 23: expected "billing", got "support"

score: 97.9% (threshold 95%) ✓ pass
consistency: 5 / 5 runs identical
cost: claude-sonnet-4-6 $0.0008  ✓ still passes on fallback

You know the fallback clears your quality bar before production — not after a silent regression lands in your logs. The function call stays identical regardless of which model is serving it:

const ticket = await fn.run("classify@v3", { input })
// same result shape whether claude-opus-4-8 or claude-sonnet-4-6 served the request
// { category: "billing", priority: "high", confidence: 0.94 }

The response schema is the contract. The eval threshold is the quality floor. The runtime picks the model that satisfies both.

Reliability by default, not by effort

The goal of automatic model fallback isn’t to write smarter error handling. It’s to stop writing error handling altogether and have the runtime own it. Every defensive try-catch you add to your AI feature is infrastructure tax — code that doesn’t ship product, that has to be maintained, and that still doesn’t catch silent quality failures.

When fallback is a runtime concern, it applies to every function you deploy, for free, without a line of defensive code.

---

fnAI is pre-launch. If you’re building AI into a production product and want automatic fallback, eval gates, and model versioning without running infrastructure, join the waitlist.

FAQ

What is automatic model fallback? Automatic model fallback routes an AI request to a pre-configured backup model when the primary fails, returning a successful response without any application code change or user-visible error.

What failure modes does model fallback cover? Hard failures (5xx), rate limits (429), and latency-based timeouts. Silent quality failures — where the API returns 200 OK but output is wrong — require eval gates rather than error-handling alone.

How reliable are LLM providers in production? OpenAI’s standard pay-as-you-go API carries a 99.5% uptime SLA, allowing up to 43.8 hours of downtime per year. Measured uptime can be lower during peak periods and major model launches.

What’s the difference between a retry and a fallback? A retry re-sends the request to the same model. It handles transient glitches but is useless during a persistent outage. A fallback switches to a different model entirely. Production systems need both: retry on transient errors, fall back on persistent ones.

Do I need to test my fallback model? Yes. Switching to a backup model without verifying output quality risks silent regressions. Eval gates — running your quality dataset against the fallback before promoting — confirm it clears the same bar as the primary.

Is fnAI available now? fnAI is pre-launch. Join the waitlist at fnai.dev/#waitlist for early access.