The direct answer
Classify the failure and establish what already happened before retrying or changing models. Use a fallback only when its data rules, task acceptance criteria, and authorization still hold. Otherwise queue work within an explicit deadline or stop with an honest status; a partial response or a timeout is not a completed task.
Start with outcome, not a catch-all fallback
A handler that sends every failed request to another provider can turn a temporary outage into a policy violation. It may also repeat completed work or hide an application defect. The useful first question is not “which model is next?” but “which state did this attempt reach?”
Keep the logical task, its individual attempts, and any external effects separate. Record a minimized attempt identifier, route/configuration revision, error category, and whether output or tool execution began. Do not attach raw customer text or credentials to the routine incident record.
The OpenAI error guide distinguishes billing-related errors from temporary rate or service errors. The same HTTP status can have different causes. Read the documented error body rather than treating every 429 as permission to retry.
A synthetic support-draft triage table
Imagine a fabricated internal tool that drafts a reply and assigns one of three labels: billing, technical, or other. It cannot send the reply or modify an account. The table below is an original playbook design, not an observed provider incident or test result.
| Observation | Decision | Visible state |
|---|---|---|
| Temporary overload before output | Wait within the task deadline; consider an approved fallback | Waiting, not completed |
| Temporary request/token throttling | Honor a valid server delay and local attempt limits | Waiting or deadline exceeded |
| Credit, spend, or usage limit | Stop automatic recovery; route to the authorized operator | Unavailable |
| Malformed input, invalid credentials, denied permission | Correct configuration or input; do not switch identities to evade the denial | Stopped for correction |
| Partial draft followed by stream failure | Mark incomplete; do not append another model’s output as a continuation | Incomplete draft |
| Timeout with unknown final outcome | Reconcile the existing attempt before deciding what to repeat | Outcome being checked |
These are application decisions. Map them to the exact endpoint’s documented behavior. Do not infer that all timeout failures are safe to replay or that a provider’s status page explains every tenant-specific problem.
Require three gates before changing the model
Data gate: the fallback destination is approved for this data category, endpoint, region, retention requirements, and enabled features. Recheck the actual payload; a different endpoint may accept a different format or require another minimization step. An outage does not expand the destination allowlist.
Task gate: the alternative has met the workload’s acceptance criteria. For the draft tool, that means an allowed label, required output fields, a grounded draft, and no unsupported commitment. Preserve the same rubric and required checks rather than quietly accepting a weaker answer because service is degraded.
Authority gate: the fallback remains within the initiating user’s scope, budget, and permitted actions. It follows the required enforcement path and does not acquire broader tools or credentials. A revised message or action requires whatever fresh approval its changed consequences demand.
Keep a reviewed fallback configuration for the specific workload. If any gate is unknown, queue or stop. This is narrower than “try any available model” and easier for an operator to assess during an incident.
Treat partial and unknown outcomes explicitly
The OpenAI rate-limit guide notes that errors after streaming starts can arrive as stream events and cautions against automatically replaying requests after consuming output. A successful connection or a few text fragments do not establish a complete response.
For a draft-only tool, isolate the incomplete draft and offer a clearly labeled replacement attempt when appropriate. For workflows that can send, delete, or charge, check the execution system before repeating anything. OpenAI’s Agents API recovery guidance specifically advises inspecting existing work and completed actions. That retrieval procedure is endpoint-specific; implement the equivalent supported outcome check for your own stack.
Keep unresolved usage reserved under the task’s accounting rules. Use the budget guide for that ledger question; this playbook decides the operational state.
Queue or stop without misleading the user
Say “queued” only after a durable queue accepts the task. Set an expiry, cancellation behavior, and an owner for stalled work. When the queue resumes, recheck authorization, current policy, and data validity rather than assuming an old admission remains valid indefinitely.
A synthetic status message could read: “The draft is incomplete. No reply was sent. You can review the partial text or request a new draft.” Use the no-send statement only when application evidence supports it. If the result is unknown, say so and prevent duplicate action buttons while reconciliation is pending.
Do not expose internal exception text, credential hints, or upstream response bodies to the user. Give a useful recovery choice without promising a restoration time you have not established.
Recover the deployment as deliberately as the request
Retain the previous reviewed model, prompt, policy, SDK, and routing configuration as one release bundle. If a new deployment caused malformed requests or changed task behavior, an authorized rollback must preserve current mandatory restrictions and revocations. Reauthorize the older configuration against today’s policy; if it cannot meet that policy, stop the affected work instead of restoring stale permissions. Rollback does not undo messages, charges, or other effects already completed.
Before restoring normal dispatch, check the affected error category, run the workload’s synthetic regression cases, and inspect queues for expired or duplicate tasks. Reopen traffic gradually under the existing limits and watch for recurrence. Keep remaining unknown outcomes assigned to an operator. A provider recovery notice alone does not prove your application configuration works.
OpenAI’s evaluation guidance recommends task-specific criteria and continued evaluation as systems change. Apply the same principle to a fallback configuration; availability does not prove acceptable task behavior.
Practice the decisions before an outage
With recording stubs and fabricated inputs, inject each table condition. Check that denied fallback destinations receive nothing, billing failures stop automatic recovery, partial drafts stay incomplete, and replay is blocked while an action outcome is unknown. Include normal successful work and an allowed fallback so permanent refusal cannot appear to be a useful implementation. These checks are proposed and have not been run for this article.
Boundary is a prelaunch commercial model router. Request-path policies are planned; this playbook does not claim a live Boundary fallback runtime, incident-response service, uptime result, or control over arbitrary agent actions.
Primary sources
Official references supporting the guidance. Examples in this guide are synthetic and are not reports of a production incident.