Model fallbacks & high availability

Any model can rate-limit, drift, or have a bad day. With MeshTok you get 175 models behind one endpoint — swapping models is a one-line change with no new SDK, no new auth, no new billing. This page shows two patterns for automatic failover so your app stays up when a single provider doesn't.

Pattern 1: sequential fallback (simple)

Try models in priority order. On any error (5xx, rate limit, timeout), move to the next. This is what most production apps need.

from openai import OpenAI

client = OpenAI(base_url="https://meshtok.com/v1", api_key="sk-...")

# Priority order — first that works wins
FALLBACK_CHAIN = [
    "openai/gpt-5.2",           # primary: best quality
    "anthropic/claude-sonnet-5", # fallback 1: different lab
    "deepseek/deepseek-v4-pro",  # fallback 2: cheap & reliable
    "bigmodel/glm-5.2",          # fallback 3: China-side redundancy
]

def chat_with_fallback(messages, **kwargs):
    last_err = None
    for model in FALLBACK_CHAIN:
        try:
            resp = client.chat.completions.create(
                model=model, messages=messages, **kwargs
            )
            return resp.choices[0].message.content
        except Exception as e:
            print(f"[fallback] {model} failed: {e}, trying next...")
            last_err = e
    raise last_err  # all models failed

Pattern 2: parallel race (lowest latency)

Fire 2-3 models concurrently, take the first successful response, cancel the rest. Best for latency-sensitive paths where you'd rather pay 2x than wait.

import asyncio
from openai import AsyncOpenAI

client = AsyncOpenAI(base_url="https://meshtok.com/v1", api_key="sk-...")

async def try_model(model, messages):
    try:
        resp = await client.chat.completions.create(model=model, messages=messages)
        return resp.choices[0].message.content
    except Exception:
        return None

async def race(models, messages):
    tasks = [asyncio.create_task(try_model(m, messages)) for m in models]
    for coro in asyncio.as_completed(tasks):
        result = await coro
        if result:
            for t in tasks: t.cancel()  # stop the losers
            return result
    raise RuntimeError("all models failed")

# usage: first of 3 to finish wins
answer = await race([
    "google/gemini-2.5-pro",
    "deepseek/deepseek-v4-flash",
    "moonshot/kimi-k2",
], [{"role": "user", "content": "Hello"}])

Choosing fallback models

A good fallback chain spans different upstream labs so a single outage doesn't take out your whole chain. Pick from these tiers:

• Frontier (pick 1-2): GPT-5.x, Claude Sonnet/Opus, Gemini 2.5 Pro
• Strong open-weight (pick 1-2): DeepSeek V4, GLM 5.2, Qwen 3, Kimi K2
• Cheap/fast (pick 1 as last resort): DeepSeek V4 Flash, GLM Flash, Haiku

Cross-lab redundancy is the whole point — if every model in your chain is GPT-based, an OpenAI outage takes you down regardless of fallback logic.

What MeshTok handles for you

• Single auth: same sk-... key works for all 175 models
• Single endpoint: https://meshtok.com/v1 — no per-provider base URLs
• Unified billing: every model bills to the same balance, itemized in usage
• Unified schema: OpenAI request/response shape for every model, including Claude and Gemini
• Quota isolation per key: use the spend limits feature to cap each fallback tier's budget independently