Build a Resilient Multi-Provider LLM Client in TypeScript: Timeouts, Retries, Circuit Breakers, and Failover

Every team that ships an AI feature eventually has the same outage. The model provider returns a wall of 529s for ninety seconds during a traffic spike, or a region goes dark, or a rate limit you did not know you were near suddenly starts rejecting one request in three. The demo never showed this because the demo made one call at a time to a warm endpoint. Production makes thousands of calls against an upstream you do not control, and the difference between a feature that degrades gracefully and one that throws a 500 at every user is a few hundred lines of unglamorous client code that almost nobody writes until after the first incident.

This is the code that closes that gap, and in 2026 it stopped being optional. The reason is structural: the model market fragmented. A year ago a sensible default was to wire your app to one frontier provider and move on. Now there is a credible heavyweight reasoning model from a Chinese lab, a family of small efficient coding models from a hyperscaler explicitly built to reduce dependence on a single API, and the same handful of frontier models showing up behind three different cloud resellers with three different rate-limit pools. The upside is leverage and price competition. The cost is that "which model serves this request" is now a runtime decision your code has to be able to make — and to re-make, instantly, when the first choice is failing.

What almost nobody tells you is that the machinery underneath a resilient client is not complicated. It is a deadline, a retry loop, a state machine that remembers which upstreams are sick, and a list you walk in order until something answers. You can buy this bundled into a proprietary gateway with a per-request markup and a dashboard, or you can own it in about two hundred and fifty lines of TypeScript that runs anywhere, talks to any provider, and is trivially testable without spending a cent on tokens. This tutorial builds the second thing.

By the end you will have a small, dependency-free client with four composable parts: a Provider interface that unifies every upstream behind one method, a withTimeout wrapper that enforces a hard per-attempt deadline, a withRetry function that does exponential backoff with full jitter, a CircuitBreaker that stops a dead provider from eating your latency budget, and a ResilientClient that wires them together and fails over across an ordered chain. Every piece is covered by deterministic tests, and the whole thing is provider-agnostic — you plug in Anthropic, OpenAI, a local model, or a deterministic fake through a single one-line interface.

The companion code lives at CrashBytes/ByteSizedExamples/resilient-llm-client-typescript. Clone it, run npm install && npm test, and follow along.

Why one provider is a single point of failure

Before we write code, it is worth being precise about what we are defending against, because the failure modes shape every design decision that follows. A call to a hosted model can fail in at least five distinct ways, and they do not want the same response.

A rate limit wants you to wait and try again, ideally after the delay the server told you to wait. A transient overload wants the same patience but without a hint. A timeout means the request is probably never coming back and you should cut it loose. A bad request — a malformed payload, a revoked key, an unknown model — will fail identically no matter how many times you retry, so retrying is pure waste. And a full provider outage means no amount of patience with that provider will help; the only recovery is a different provider entirely.

The naive client treats all five the same. It either gives up immediately, which turns a recoverable blip into a user-facing error, or it retries everything forever, which turns a permanent 400 into an infinite loop and hammers a struggling upstream when it is least able to cope. The whole craft here is matching the response to the failure.

The market context is what makes the fifth row — the full outage — newly worth engineering for. Through the first half of 2026 the model landscape stopped being a one-horse race. The teams with the calmest incident channels were the ones who could re-point traffic at a second provider without a deploy, because they had already written the abstraction that made the provider a runtime choice rather than a compile-time constant. For the market backdrop on why provider diversity went mainstream this year, see the multi-provider resilience analysis that runs alongside this tutorial; for the harder edge case of a model you depend on being deprecated out from under you, see the model-continuity piece on planning for a shutdown.

The one abstraction: a Provider interface

Everything in this library hangs off a single interface. If you get this right, every resilience concern below becomes provider-agnostic for free; if you get it wrong, you end up special-casing each upstream in three different places.

// src/types.ts
export interface ChatMessage {
  role: 'system' | 'user' | 'assistant'
  content: string
}

export interface ChatRequest {
  messages: ChatMessage[]
  model?: string
  maxTokens?: number
}

export interface ChatResponse {
  text: string
  model: string
  provider: string
  raw?: unknown
}

export interface Provider {
  readonly name: string
  chat(request: ChatRequest, signal: AbortSignal): Promise<ChatResponse>
}

The important detail is the AbortSignal. A Provider is not just "a function that calls a model" — it is a function that calls a model and promises to stop when told. That promise is what lets the timeout layer actually free the socket instead of leaving a doomed request running in the background, quietly consuming a connection from your pool. Any upstream that can be expressed as "take a request, honor an abort signal, return a response" fits behind this interface: a fetch call, an official SDK client, a local process, or a fake.

Because the whole library speaks only this interface, swapping Anthropic for OpenAI or adding a third fallback is a one-line change to a list. The client never learns the name of a single concrete provider.

Encoding retryability into the error type

The second design decision is where the retry-or-not knowledge lives. The wrong answer is "in a giant if-statement at the call site that string-matches error messages." The right answer is "on the error itself."

// src/errors.ts (abridged)
export class LLMError extends Error {
  readonly retryable: boolean
  constructor(
    message: string,
    retryable: boolean,
    options?: { cause?: unknown }
  ) {
    super(message, options)
    this.name = new.target.name
    this.retryable = retryable
  }
}

export class RateLimitError extends LLMError {
  readonly retryAfterMs?: number
  constructor(message = 'rate limited', retryAfterMs?: number) {
    super(message, true)
    this.retryAfterMs = retryAfterMs
  }
}

export class ServerError extends LLMError {
  constructor(
    message = 'upstream server error',
    readonly status?: number
  ) {
    super(message, true)
  }
}

export class TimeoutError extends LLMError {
  constructor(message = 'request timed out') {
    super(message, true)
  }
}

export class ClientRequestError extends LLMError {
  constructor(
    message = 'client request error',
    readonly status?: number
  ) {
    super(message, false)
  }
}

Now "should I retry this" is a property lookup, not a parse. A RateLimitError and a ServerError and a TimeoutError are all retryable; a ClientRequestError (your 400s, 401s, 404s) is not. A single predicate ties it together and handles the foreign errors — a raw network failure from fetch — that do not subclass our hierarchy:

export function isRetryable(error: unknown): boolean {
  if (error instanceof LLMError) return error.retryable
  if (error instanceof Error) {
    return /ECONNRESET|ETIMEDOUT|ENOTFOUND|EAI_AGAIN|ECONNREFUSED|network|fetch failed/i.test(
      error.message
    )
  }
  return false
}

The payoff for this small piece of discipline is large: every layer below makes its decisions by asking one question, and the answer always lives in exactly one place.

Layer 1: a hard per-attempt timeout

The first and most under-appreciated layer is the deadline. A hung request is worse than a failed one, because a failure returns control and a hang does not — it sits on a connection, holds a slot in your concurrency limit, and makes your p99 latency a function of the slowest upstream you ever talk to. The fix is to race every attempt against a timer and abort the loser.

// src/timeout.ts
import { TimeoutError } from './errors.js'

export function withTimeout<T>(
  fn: (signal: AbortSignal) => Promise<T>,
  ms: number
): Promise<T> {
  const controller = new AbortController()
  let timer: ReturnType<typeof setTimeout>

  const timeout = new Promise<never>((_resolve, reject) => {
    timer = setTimeout(() => {
      controller.abort()
      reject(new TimeoutError(`request timed out after ${ms}ms`))
    }, ms)
  })

  return Promise.race([fn(controller.signal), timeout]).finally(() => {
    clearTimeout(timer)
  })
}

Two things make this robust. First, when the timer wins the race it both rejects with a TimeoutError and aborts the signal it handed to fn, so a well-behaved provider tears down its request instead of leaking it. Second, even a provider that ignores the signal entirely still loses the race — the caller is unblocked at the deadline no matter what. The finally clears the timer so a fast success does not leave a dangling handle that keeps the process alive.

Choosing the timeout value is a real decision, not a default to leave at thirty seconds. Set it just above your p99 for normal responses, because a request that has already run longer than ninety-nine percent of successful ones is far more likely to be stuck than to be about to succeed. Cutting it loose and failing over is usually faster than waiting.

Layer 2: retry with exponential backoff and full jitter

A transient failure deserves another try — but a retry done badly is how a small upstream wobble becomes a self-inflicted outage. The two classic mistakes are retrying things that will never succeed, and retrying everything at the same instant so that the moment a struggling provider recovers, your entire fleet stampedes it in lockstep.

The fixes are exponential backoff (wait longer after each failure) and full jitter (randomize each wait so retries spread out instead of synchronizing). Here is the whole retry function:

// src/retry.ts (abridged)
import { isRetryable as defaultIsRetryable, RateLimitError } from './errors.js'

export async function withRetry<T>(
  fn: (attempt: number) => Promise<T>,
  options: RetryOptions = {}
): Promise<T> {
  const maxAttempts = options.maxAttempts ?? 3
  const baseDelayMs = options.baseDelayMs ?? 200
  const maxDelayMs = options.maxDelayMs ?? 10_000
  const retryable = options.isRetryable ?? defaultIsRetryable
  const sleep = options.sleep ?? defaultSleep
  const random = options.random ?? Math.random

  let lastError: unknown
  for (let attempt = 1; attempt <= maxAttempts; attempt++) {
    try {
      return await fn(attempt)
    } catch (error) {
      lastError = error
      if (attempt >= maxAttempts || !retryable(error)) throw error

      const exponential = Math.min(maxDelayMs, baseDelayMs * 2 ** (attempt - 1))
      let delayMs = Math.floor(random() * exponential)

      if (
        error instanceof RateLimitError &&
        typeof error.retryAfterMs === 'number'
      ) {
        delayMs = Math.min(maxDelayMs, error.retryAfterMs)
      }

      options.onRetry?.({ attempt, delayMs, error })
      await sleep(delayMs)
    }
  }
  throw lastError
}

Walk the important lines. The loop bails immediately on a non-retryable error, so a 400 fails on the first attempt instead of three times. The backoff is full jitter: the cap doubles each attempt, but the actual delay is a uniform random draw between zero and that cap, which is the variant AWS found spreads retries most evenly. And a Retry-After from a rate-limit response always wins over the computed backoff, because the server just told you exactly how long to wait and guessing is strictly worse than listening.

The other quietly important choice is that sleep and random are injectable. That is not over-engineering — it is what makes the backoff testable without real timers and without flakiness, which we will use in a moment.

Layer 3: a circuit breaker per provider

Retries handle a blip. They are exactly the wrong tool for a sustained outage. If a provider has been down for two minutes, then every single request still pays the full timeout-times-retries tax before failing over — and you are hammering a dead endpoint thousands of times while it tries to recover. A circuit breaker fixes this by remembering recent failures and short-circuiting calls to a provider that is clearly sick.

It is a three-state machine. Closed is normal: requests flow and consecutive failures are counted. After enough failures the breaker trips open: requests are rejected instantly, without even attempting the call, for a cooldown period. When the cooldown elapses the breaker goes half-open and allows a probe or two through; if they succeed it closes, if any fails it snaps back open.

// src/circuit-breaker.ts (abridged)
export class CircuitBreaker {
  private state: CircuitState = 'closed'
  private failures = 0
  private successes = 0
  private openedAt = 0

  constructor(private readonly options: CircuitBreakerOptions = {}) {
    this.failureThreshold = options.failureThreshold ?? 5
    this.cooldownMs = options.cooldownMs ?? 30_000
    this.successThreshold = options.successThreshold ?? 1
    this.now = options.now ?? Date.now
  }

  canRequest(): boolean {
    if (this.state === 'open') {
      if (this.now() - this.openedAt >= this.cooldownMs) {
        this.state = 'half-open'
        this.successes = 0
        return true
      }
      return false
    }
    return true
  }

  onSuccess(): void {
    if (this.state === 'half-open') {
      if (++this.successes >= this.successThreshold) this.close()
    } else {
      this.failures = 0
    }
  }

  onFailure(): void {
    if (this.state === 'half-open') return this.open()
    if (++this.failures >= this.failureThreshold) this.open()
  }
}

Notice that now is injectable, exactly like sleep and random were. The breaker is a time-dependent state machine, and the only way to test "it reopens after the cooldown" without making the test suite sleep for thirty real seconds is to control the clock. We will.

One subtlety worth calling out: in our client the breaker counts one failure per exhausted provider, not one per retry. So failureThreshold: 5 means "five chat() calls in a row could not get an answer out of this provider," which is the signal you actually care about, rather than "five retries within one unlucky request," which is noise.

Composing it: the ResilientClient and failover

Now the four layers come together. The client holds an ordered list of providers and a circuit breaker per provider. For each request it walks the chain: skip any provider whose breaker is open, otherwise run that provider wrapped in a timeout wrapped in retry, and on success return. If a provider is exhausted, record one failure on its breaker and fall over to the next. If every provider fails, throw an aggregate error that carries all the causes.

// src/client.ts (the core loop)
async chat(request: ChatRequest): Promise<ChatResponse> {
  const failures: Array<{ provider: string; error: unknown }> = [];

  for (let i = 0; i < this.providers.length; i++) {
    const provider = this.providers[i];
    const breaker = this.breakers.get(provider.name)!;

    if (!breaker.canRequest()) {
      failures.push({ provider: provider.name, error: new CircuitOpenError(provider.name) });
      this.failover(i, new CircuitOpenError(provider.name));
      continue;
    }

    try {
      const response = await withRetry(
        () => withTimeout((signal) => provider.chat(request, signal), this.timeoutMs),
        this.retry,
      );
      breaker.onSuccess();
      return response;
    } catch (error) {
      breaker.onFailure();
      failures.push({ provider: provider.name, error });
      this.onProviderError?.({ provider: provider.name, error });
      this.failover(i, error);
    }
  }

  throw new AllProvidersFailedError(failures);
}

The nesting order matters and is easy to get backwards. Timeout is the innermost wrapper, so it applies to each individual attempt; retry wraps timeout, so each retry gets its own fresh deadline; the breaker sits outside both, recording a single verdict per provider after retries are spent. Read the composition out loud and it is exactly the policy you want: "try this provider, giving each attempt its own deadline and retrying transient failures a few times; if the whole provider is exhausted, remember that and move to the next one."

The aggregate AllProvidersFailedError is the small touch that makes 3 a.m. debugging bearable. When everything is down you do not get a generic 500 — you get one error object that names every provider that failed and why, so the on-call engineer can tell "all three providers rate-limited us simultaneously" from "our API key is revoked" in one glance.

Plugging in a real provider

The whole point of the Provider interface is that a real upstream is a thin adapter. Here is the Anthropic one, written with fetch so the library stays dependency-free; in a larger app you would more often wrap the official SDK client inside this same shape. The only real work is translating HTTP status codes into the typed, retryable errors the rest of the system already understands.

// src/providers/anthropic.ts (abridged)
export class AnthropicProvider implements Provider {
  readonly name = 'anthropic'

  async chat(request: ChatRequest, signal: AbortSignal): Promise<ChatResponse> {
    const model = request.model ?? this.model // defaults to claude-opus-4-8
    const system = request.messages
      .filter(m => m.role === 'system')
      .map(m => m.content)
      .join('\n\n')
    const messages = request.messages
      .filter(m => m.role !== 'system')
      .map(m => ({ role: m.role, content: m.content }))

    const response = await this.fetchFn(`${this.baseUrl}/v1/messages`, {
      method: 'POST',
      signal,
      headers: {
        'content-type': 'application/json',
        'x-api-key': this.apiKey,
        'anthropic-version': '2023-06-01',
      },
      body: JSON.stringify({
        model,
        max_tokens: request.maxTokens ?? 1024,
        system,
        messages,
      }),
    })

    if (!response.ok) throw await this.toError(response)

    const data = await response.json()
    const text = data.content
      .filter(b => b.type === 'text')
      .map(b => b.text)
      .join('')
    return { text, model: data.model, provider: this.name, raw: data }
  }
}

The status-code mapping is where the resilience policy connects to reality. A 429 becomes a RateLimitError that reads the Retry-After header so the retry layer can honor it. A 500 or 529 becomes a retryable ServerError. Anything else in the 4xx range becomes a non-retryable ClientRequestError, because a malformed request or a bad key will fail identically forever and retrying it only wastes time and quota.

private async toError(response: Response): Promise<Error> {
  const message = `anthropic ${response.status}`;
  if (response.status === 429) {
    const header = response.headers.get('retry-after');
    const retryAfterMs = header ? Number(header) * 1000 : undefined;
    return new RateLimitError(message, retryAfterMs);
  }
  if (response.status >= 500) return new ServerError(message, response.status);
  return new ClientRequestError(message, response.status);
}

Note that fetchFn is injected with a default of the global fetch. That is the same testability trick again: a test can pass a fake fetch that returns a 429 with a Retry-After header and assert the adapter produces the right typed error, all without a network or an API key. A second provider — OpenAI, a local model, a hosted reseller — is just another file implementing the same interface, and adding it to the chain is one line.

Testing it deterministically

This is the part proprietary gateways cannot give you: a test suite that exercises every resilience path in milliseconds, with no network, no API key, and no flakiness. The injectable sleep, random, and now we threaded through every layer are what make it possible. The companion repo ships nineteen tests across four files; here are the load-bearing ones.

A scriptable FakeProvider is the workhorse. You hand it a list of behaviors and each chat() call consumes the next one, so a test can say "fail twice with a 503, then succeed" with total precision:

const flaky = new FakeProvider('primary', [
  { type: 'error', error: new ServerError('down', 503) },
  { type: 'error', error: new ServerError('down', 503) },
  { type: 'ok', text: 'recovered' },
])

Retry math becomes a pure assertion once random is fixed and sleep is a no-op. With random pinned to 0.5 and a 200 ms base, the first backoff cap is 200 ms so the delay is 100 ms, and the second cap is 400 ms so the delay is 200 ms — exactly, every run:

it('uses full-jitter exponential backoff with the injected random', async () => {
  const delays: number[] = []
  await withRetry(failTwiceThenSucceed, {
    maxAttempts: 3,
    baseDelayMs: 200,
    random: () => 0.5,
    sleep: async () => {},
    onRetry: ({ delayMs }) => delays.push(delayMs),
  })
  expect(delays).toEqual([100, 200]) // deterministic
})

The circuit breaker is tested by controlling its clock. "It reopens after the cooldown" would be a thirty-second test against a real clock; with an injected now it is instant:

let now = 0
const cb = new CircuitBreaker({
  failureThreshold: 1,
  cooldownMs: 1000,
  now: () => now,
})
cb.onFailure()
expect(cb.canRequest()).toBe(false) // open, still in cooldown
now = 1000
expect(cb.canRequest()).toBe(true) // cooldown elapsed -> half-open

And failover is tested end to end with two fakes — primary always fails, secondary succeeds — asserting that the response comes from the secondary and that, after enough failures, the primary's breaker is open and it is skipped entirely:

await client.chat(request)
await client.chat(request)
expect(client.breakerState('flaky')).toBe('open')
const res = await client.chat(request)
expect(res.provider).toBe('backup') // primary skipped — breaker open

The whole suite runs in a few seconds and never touches a token.

Production considerations

A few things separate a tutorial client from one you trust with real traffic.

Order the chain by cost and capability, not just availability. Failover is not free — your fallback may be slower, pricier, or slightly less capable. Put your preferred provider first and treat the rest as a graceful-degradation ladder, not equals. If correctness matters more than cost on a given route, you might even run the request against two providers and take the first to answer; if cost dominates, keep the cheap model primary and only climb on failure. The same approach that makes a swap safe — a regression gate you can run against any provider — is worth building alongside this; see the LLM evaluation harness tutorial for that half of the story.

Tune the breaker to your traffic. A failureThreshold of 5 and a 30-second cooldown are sane defaults for steady traffic, but a low-volume endpoint may trip too slowly to matter and a high-volume one may want a shorter cooldown so it recovers faster. The values are configuration, not constants, precisely so you can turn these knobs per deployment without touching the library.

Make idempotency a precondition of retry. Retrying a chat completion is safe because it has no side effects. The moment a request triggers a tool call that charges a card or sends an email, blind retry becomes dangerous. Keep the retry layer on the model call itself and put side effects behind their own idempotency keys.

Emit the hooks. The client exposes onProviderError and onFailover callbacks for exactly one reason: a failover that happens silently is a failover you discover from a billing surprise. Wire them to your metrics so "we have been serving every request from the backup provider for six hours" shows up on a dashboard, not in a postmortem.

Where this is all heading is a world where multi-provider routing is the assumed default for any serious AI application rather than a resilience nicety — a claim I make precise, dated, and falsifiable in this prediction on multi-provider failover becoming standard.

Conclusion

A resilient LLM client is not a framework and not a product. It is four small, boring layers — a deadline, a retry loop, a breaker, and an ordered chain — each of which does one thing and composes cleanly with the others. Written behind a single Provider interface, the whole thing is about two hundred and fifty lines, depends on nothing, talks to any upstream, and is tested without spending a cent. The payoff is that the question that used to cause incidents — "what do we do when the provider is down" — already has an answer your code executes on its own.

Clone the companion repo, run the test suite, then point a real adapter at your provider and add a second one behind it. The day the first provider has a bad hour, your users will not notice, and that quiet non-event is the entire return on a couple hundred lines of unglamorous code.

Further reading

• Plan for the provider you depend on going away — the model-continuity case for treating provider risk as an engineering problem, which this client operationalizes.
• Build an LLM-as-Judge Evaluation Harness in TypeScript — the regression gate that makes swapping providers safe, and a sibling small, tested, provider-agnostic primitive.
• Multi-provider resilience and the 2026 model fragmentation — the market context for why provider diversity went mainstream this year.
• Prediction: multi-provider failover becomes the default for enterprise LLM apps by the end of 2027 — my dated, falsifiable claim on where this is heading.
• Companion code on GitHub — the full, tested project from this tutorial.