Multi-Model Fallback Design for AI Chat — Survive 429 Rate Limits and API Outages

Relying on a single LLM API provider means a sudden rate limit (HTTP 429 Too Many Requests), an outage (HTTP 503), or slow responses during peak hours can take the whole service down. If you operate inside a free tier or a metered quota, a two-stage switching design is essential.

---

Primary / secondary configuration patterns

How the Primary and Secondary models split the work, and when to switch.

Level	Primary model	Secondary model	Switch trigger
Same-family switch	High-accuracy (e.g. Gemini Flash)	Lightweight sibling (e.g. Gemini Flash-Lite)	Rate limit (429), timeout
Cross-provider switch	External API (e.g. OpenAI / Gemini)	Cloud Workers (e.g. Cloudflare Workers AI)	API outage (5xx), daily quota exhausted
Limit adjustment	Full context (8000 tokens)	Minimal context (1000 tokens)	Calls during cooldown

---

Implementing the fallback circuit

The steps for adding a multi-model fallback circuit.

1. Catch HTTP 429 and similar errors in a try-catch around the primary API call.
2. On 429, set a cooldown period and temporarily bypass the primary.
3. Adapt the prompt and history for the secondary model, then try the secondary API.

1. Two-stage (Primary → Secondary) wrapper function

Catch the error when the primary API fails and switch to the secondary API behind the same interface.

interface LlmResponse {
  text: string;
  usedModel: "primary" | "secondary";
}

export async function callLlmWithFallback(
  messages: Array<{ role: string; content: string }>
): Promise<LlmResponse> {
  // 1. Try the Primary API
  try {
    const primaryResult = await callPrimaryLlmApi(messages);
    return { text: primaryResult, usedModel: "primary" };
  } catch (error: any) {
    console.warn("Primary LLM API failed. Falling back to Secondary.", error?.message);
    
    // 2. On 429 (Rate Limit) or 5xx, switch to Secondary
    try {
      // Lighten the messages/prompt for the secondary model
      const lightweightMessages = adaptMessagesForSecondary(messages);
      const secondaryResult = await callSecondaryLlmApi(lightweightMessages);
      return { text: secondaryResult, usedModel: "secondary" };
    } catch (secondaryError) {
      throw new Error("Both Primary and Secondary LLM APIs failed.");
    }
  }
}

2. Cooldown-state caching

When 429s keep happening, avoid wasting latency on primary timeouts by skipping the primary for a fixed period (e.g. 5 minutes) and calling the secondary directly.

let primaryCoolDownUntil = 0;

async function smartLlmCall(messages: any) {
  const now = Date.now();

  // During the cooldown period, call Secondary directly
  if (now < primaryCoolDownUntil) {
    return await callSecondaryLlmApi(adaptMessagesForSecondary(messages));
  }

  try {
    return await callPrimaryLlmApi(messages);
  } catch (err: any) {
    if (err?.status === 429) {
      // Set a 5-minute cooldown
      primaryCoolDownUntil = Date.now() + 5 * 60 * 1000;
    }
    return await callSecondaryLlmApi(adaptMessagesForSecondary(messages));
  }
}

3. Adapting the prompt and history for the secondary model

Backup models often have smaller token limits and context sizes than the primary, so on switch you should shrink the system prompt and trim the conversation history.

function adaptMessagesForSecondary(messages: any[]) {
  // Replace the system prompt with a short version and cap history at the last 2 user turns
  const userMessages = messages.filter((m) => m.role === "user").slice(-2);
  return [
    { role: "system", content: "You are an assistant that answers concisely." },
    ...userMessages,
  ];
}

---

Failure boundary — real incidents and fixes

Real-world trouble patterns when switching between multiple models.

[Sample error log]: Error: 400 Context length exceeded on Secondary Model (Max: 2048 tokens, Received: 4500 tokens)
[Root cause]: The long conversation history used with the Primary was passed unchanged to the Secondary, exceeding its token limit — so both models went down.

Context overflow on the backup (double failure)

After the Primary fails, the switch to the Secondary carries the long Primary system prompt and bulky history unchanged, so the Secondary also hits its token limit — total outage.

• Fix: Always route the fallback through an adapter like adaptMessagesForSecondary that shortens the prompt and re-slices the history array.

Unlimited retries that burn quota

Naively looping a while on 429 retries accumulates API wait time (client-side timeouts) and rapidly consumes quota.

• Fix: Limit immediate retries on the same model to at most 1 with exponential backoff, then route to the backup model promptly.

---

Related links

• How the “Kaga” AI chat runs on a free tier — architecture and character design
• Circuit breaker and retry patterns in practice (JA)
• Sliding-window rate limiting with Redis (JA)