AI Provider Outages: How to Build a Fallback

When your AI provider goes down, the right fallback depends on two things: which error you got back, and whether a human is waiting on the result. A 429 means you hit your own rate limit and should slow down, not fail over. A 529 or 503 means the provider is overloaded and failover can actually help. And for any step where nobody is blocked on the answer, the cheapest reliable fallback is to queue the job and retry it later, not to bolt on a second provider. The popular advice, "add a backup model," is right for a small slice of your automations and wrong for most of them.

This matters more than it used to. The OpenAI outage on June 10, 2025 ran over twelve hours and took the API down, not just the chat app. A retry loop with exponential backoff does nothing against an outage measured in hours. If an AI step sits in the middle of a flow that charges customers or sends messages, "the model returned an error" is not a harmless event, it is a half-finished run. So the question is not whether to handle provider failure. It is how, and the answer is different for each step.

First, read the error: not every failure is an outage

Operators lump every red run into "the AI is down." The API tells you more than that, and the distinction decides your response. Here is what the two big providers actually return.

What happened	OpenAI	Anthropic	Whose fault	Right response
You hit a rate or quota limit	429	429 rate_limit_error	Yours	Slow down. Honor the retry-after header. Reduce concurrency. Do not fail over.
Provider is over capacity	503	529 overloaded_error	Theirs	Retry with backoff and jitter, then fail over if the job is urgent.
Internal server error	500	500 api_error	Theirs	Retry with backoff. Short ones self-heal.
Request timed out	client timeout	504 timeout_error	Mixed	Decouple the step. Retrying the same slow call rarely helps.
Bad request, auth, not found	400 / 401 / 404	400 / 401 / 404	Yours (config)	Never retry. Alert a human.

The line that trips people up is 429 versus 529. They look almost identical in a log, and both feel like "too much traffic." But a 429 is your account hitting its own tier limit, and Anthropic's docs are explicit that a 529 is the API being overloaded across all users, unrelated to your usage. Failing over to a second provider on a 429 is the wrong move: you were not blocked by an outage, you were sending too fast, and the fix is to back off and lower your concurrency. Do the failover on a 429 and you often just rate-limit the backup too, or you spend money working around a problem a five-second pause would have solved.

The 4xx config errors are the other trap. A 401 or a 404 will never fix itself no matter how many times you retry. Those belong in an alert to a human, not in a backoff loop that quietly burns an hour before anyone notices. Deciding retry versus fail versus alert per error class is the same instinct we lay out in when should an automation retry: the response has to match the failure, not a single catch-all.

The fallback trap: a second provider is not a free upgrade

The instinct after your first outage is to wire in a backup: if OpenAI errors, call Anthropic, or the reverse. It sounds like pure insurance. It is not, for two reasons most "add a fallback" tutorials skip.

The first is silent degradation. A different model returns different output for the same prompt. If your automation validates that output against a JSON schema, and the backup's answer happens to fit the schema, the run continues on quietly worse results. Nothing errors. Nothing alerts. Your extraction accuracy drops, your classifications get sloppier, and the first sign is a customer complaint two weeks later. Schema-valid is not the same as correct, which is the whole point of why AI returning valid JSON is not enough. A fallback that passes validation but produces worse answers is arguably more dangerous than a hard failure, because a hard failure you can see.

The second is the untested path. A fallback that has never run is not a safety net, it is a second integration that will meet production for the first time during your worst hour. Different auth, a slightly different request shape, a different max-token default, and the "backup" errors right alongside the primary. If you add a fallback, you have to force it to run in testing, point the primary at a bad key, and confirm the backup returns output you would actually accept. Otherwise you have doubled your surface area and bought nothing.

Does this step even need a fallback?

Here is the reframe that saves most of the work. Before you design any failover, ask one question about the step: is a human blocked on this exact result right now?

Sort every AI step into three buckets.

Real-time and human-blocking. A chatbot reply, a live form that shows the user an AI result, a step in a checkout. Someone is staring at a spinner. These are the only steps that justify a true cross-provider fallback, because the alternative is a visible failure in front of a customer. Build the failover, test it, and accept the cost.

Near-real-time. A support ticket that should be triaged within minutes, a lead that should be enriched before a rep calls. No one is watching a spinner, but the work is time-sensitive. These do not need a second provider. They need a retry with a longer window and an alert if they are still failing after, say, fifteen minutes. A twelve-hour outage is rare; a few-minute blip is common, and a patient retry rides right over it.

Batch and overnight. Backfills, nightly summaries, bulk categorization, anything with no human waiting on any single item. These need no failover at all. If the provider is down, the job waits and runs when it recovers. Better still, this is exactly the work that belongs on the provider's asynchronous Batch API in the first place, which runs the same model for half the token price and is designed for "submit now, collect later." We cover that split in batch versus real-time AI calls. For batch work, queue-and-retry is not a downgrade from a fancy fallback, it is the correct architecture.

The mistake we see most often is operators building a cross-provider failover for a step that runs at 2 a.m. against a queue nobody touches until morning. That is engineering effort spent insuring against a delay that costs nothing.

The safest failover is the same model, a different door

When a step genuinely needs failover, there is a hierarchy of options ranked by how much they change your output.

The best failover is the same model on a different cloud. Claude runs on the Anthropic API, on Amazon Bedrock, and on Google Vertex AI. GPT models run on the OpenAI API and on Azure. Failing over from one route to another keeps you on the same model weights, so the behavior is the closest possible match to what you tested. The model ID string changes between routes, so your flow needs to swap the identifier along with the endpoint and credentials, but the answers stay in family. This is the failover with the least hidden risk, because it does not introduce a new model's quirks mid-incident.

The next option is a different model from the same or another vendor, and this is where you accept the degradation risk from the section above. It can be the right call, especially if your primary model has no alternate cloud, but only if you have tested the backup on your real prompts and confirmed the output is one you would ship. Never fail over to a model you have not run against your own inputs.

Whatever you pick, add two guards. A circuit breaker so that after a handful of failures you stop hammering the down provider and go straight to the fallback or the queue, instead of retrying into a wall and paying for every attempt. And a dedupe key on any step that writes, charges, or sends, so that a failover retry does not re-run an action that already went through before the error. An outage plus a naive retry is one of the classic ways a single order becomes three charges.

How to start

Pick your single most important AI step, the one whose failure a customer would actually notice, and do three things. Classify the error handling first: make sure a 429 backs off, a 529 or 503 retries or fails over, and a 401 or 404 pages a human instead of looping. Then sort that step into real-time, near-real-time, or batch, and give it the lightest fallback that fits: a tested same-model failover only if a human is truly blocked, a patient retry otherwise. Finally, force the failure in testing before you trust it, and confirm the fallback does not double-fire any step that touches money or messages.

Most automations need far less failover machinery than the "add a backup provider" advice implies, and the ones that need it need it done carefully. If you want a second set of eyes on which of your AI steps actually warrant a fallback and which just need a queue, that is the kind of thing we sort out in workflow automation systems. Or tell us where your automation broke during the last outage at our contact page and we will point you at the right pattern.