Reliable LLM Apps: Timeouts, Retries, Fallbacks, and Circuit Breakers

Keep LLM features working when providers are slow or down: SDK retries and timeouts, deadlines, fallback models, circuit breakers, and graceful degradation in TypeScript.

Advanced18 min readUpdated October 7, 2026

On this page· How LLM calls fail

Why this matters

Model APIs are remote services under heavy global demand. Requests get rate-limited, overloaded, slow, or occasionally fail, and an LLM call takes seconds where a database query takes milliseconds, so every failure is felt. Users do not care whose fault it is: a support widget that shows a spinner forever is broken. Reliability engineering for LLM apps is ordinary distributed-systems practice (timeouts, retries, fallbacks, degradation) applied to an unusually slow and expensive dependency.

How LLM calls fail#

Failure	Looks like	Retry?	Response
Rate limited	`429`	Yes, after `retry-after`	Back off; smooth your traffic; raise limits
Overloaded	`529`	Yes, with backoff	Retry, then fall back
Server error	`500`, `502`, `503`	Yes	Retry, then fall back
Timeout or network error	Connection error, no status	Yes	Retry within the deadline
Invalid request	`400`, `413`, `422`	No	A bug or oversized input: fix it, do not retry
Auth or permission	`401`, `403`	No	Configuration problem: alert
Truncated answer	`stop_reason: "max_tokens"`	Sometimes	Raise `max_tokens` or ask for shorter output
Bad output	Fails schema or business rules	Once, with feedback	See structured outputs

The rule: retry what is transient, fix what is not. Retrying a 400 five times just fails five times, slower.

What the SDK already does#

The Anthropic SDK retries 408, 409, 429, 5xx (including 529), and connection errors twice by default, with exponential backoff and jitter, and honors the server's retry-after hints. Its default timeout is 10 minutes, generous because some responses are long. Tune both to each use case:

lib/clients.ts

import Anthropic from "@anthropic-ai/sdk";
 
// Interactive: fail fast so a fallback still has time to answer.
export const interactive = new Anthropic({ maxRetries: 1, timeout: 20_000 });
 
// Background jobs: patient, because nobody is waiting.
export const background = new Anthropic({ maxRetries: 5, timeout: 5 * 60_000 });
 
// Per-request overrides are also possible:
// await interactive.messages.create(params, { timeout: 8_000, maxRetries: 0, signal });

Retry, fall back, degrade#

Hover or tap a component to see what it does.

Retry, fallback, and degrade: transient errors are retried with backoff, persistent outages switch to a fallback model, and if everything fails the app degrades gracefully instead of breaking.

Show text description

Each step to the right handles a rarer and more severe failure. Most requests never leave the first box.

Request
Primary model: Every call has a timeout sized to the expected response, and a signal tied to the user's request.
Retry: The SDK retries 408, 409, 429, and 5xx errors (including 529 overloaded) with exponential backoff, honoring retry-after.
Fallback model: A different model or deployment, tested with your evals so the quality drop is known and acceptable.
Degraded mode: A cached answer, search results without a summary, or a clear 'try again shortly' with the user's input preserved.

Retries handle blips. For longer incidents, a fallback, such as another model or another deployment, keeps the feature working, ideally one you have evaluated so the quality difference is known. If everything fails, degrade gracefully: a cached answer, search results without an AI summary, or a clear message that keeps the user's input.

All of this must fit inside a deadline: the time the user (or the calling service) is willing to wait. Per-attempt timeouts and a total deadline stop retries from consuming the whole budget, so the fallback still has time to run:

Attempt
Backoff wait
Fallback

Retries within a deadline: two overloaded responses and one timeout use 11.8 seconds, so the fallback model still has enough of the 30-second budget to answer.

Show text description

A single request with a 30-second deadline. Per-attempt timeouts and a total deadline keep retries from eating the whole budget.

Attempts and waits
Step	Starts at	Duration
user request (deadline 30 s)	0.0 s	21.4 s
attempt 1 → 529 overloaded	0.0 s	1.2 s
backoff ~0.5 s	1.2 s	0.6 s
attempt 2 → timeout (8 s)	1.8 s	8.0 s
backoff ~1 s	9.8 s	1.1 s
attempt 3 → 529 overloaded	10.9 s	0.9 s
fallback model → success	11.8 s	9.6 s

A circuit breaker#

When the primary is clearly down, retrying every request against it wastes the deadline of every user. A circuit breaker notices repeated failures and skips the primary for a cooldown period, then lets a single trial request through to check whether it has recovered:

lib/breaker.ts

type State = "closed" | "open" | "half-open";
 
export class CircuitBreaker {
  private state: State = "closed";
  private failures: number[] = [];
  private openedAt = 0;
  private trialInFlight = false;
 
  private readonly opts: { failureThreshold: number; windowMs: number; cooldownMs: number };
  private readonly now: () => number;
 
  constructor(
    opts = { failureThreshold: 5, windowMs: 30_000, cooldownMs: 30_000 },
    now: () => number = Date.now, // injectable for tests
  ) {
    this.opts = opts;
    this.now = now;
  }
 
  /** Whether a request may go to the protected dependency right now. */
  allow(): boolean {
    if (this.state === "open" && this.now() - this.openedAt >= this.opts.cooldownMs) {
      this.state = "half-open"; // let one trial request through
      return true;
    }
    return this.state !== "open" && !(this.state === "half-open" && this.trialInFlight);
  }
 
  onSuccess() {
    this.state = "closed";
    this.failures = [];
    this.trialInFlight = false;
  }
 
  onFailure() {
    const t = this.now();
    this.trialInFlight = false;
    if (this.state === "half-open") return this.open(t);
    this.failures = [...this.failures.filter((f) => t - f < this.opts.windowMs), t];
    if (this.failures.length >= this.opts.failureThreshold) this.open(t);
  }
 
  markTrial() {
    if (this.state === "half-open") this.trialInFlight = true;
  }
 
  get current(): State {
    return this.state;
  }
 
  private open(t: number) {
    this.state = "open";
    this.openedAt = t;
    this.failures = [];
  }
}

Putting it together#

lib/resilient.ts

import Anthropic from "@anthropic-ai/sdk";
import { CircuitBreaker } from "./breaker.ts";
 
export interface Target {
  name: string;
  client: Anthropic;
  model: Anthropic.Model;
  breaker: CircuitBreaker;
}
 
/** Errors worth trying elsewhere: overload, server errors, rate limits, timeouts, network. */
export function isTransient(error: unknown): boolean {
  if (error instanceof Anthropic.APIConnectionError) return true; // includes timeouts
  if (error instanceof Anthropic.APIError) return error.status === 408 || error.status === 409 || error.status === 429 || (error.status ?? 0) >= 500;
  return false;
}
 
export class AllTargetsFailed extends Error {}
 
/**
 * Try each target in order within one deadline. Non-transient errors (bad requests, auth)
 * are thrown immediately: a fallback would fail the same way.
 */
export async function createResilient(
  targets: Target[],
  params: Omit<Anthropic.MessageCreateParamsNonStreaming, "model">,
  opts: { deadlineMs: number; signal?: AbortSignal },
): Promise<{ message: Anthropic.Message; target: string }> {
  const deadline = AbortSignal.any([AbortSignal.timeout(opts.deadlineMs), ...(opts.signal ? [opts.signal] : [])]);
  const errors: string[] = [];
 
  for (const target of targets) {
    if (deadline.aborted) break;
    if (!target.breaker.allow()) {
      errors.push(`${target.name}: circuit open`);
      continue;
    }
    target.breaker.markTrial();
    try {
      const message = await target.client.messages.create({ ...params, model: target.model }, { signal: deadline });
      target.breaker.onSuccess();
      return { message, target: target.name };
    } catch (error) {
      if (!isTransient(error) && !deadline.aborted) throw error;
      target.breaker.onFailure();
      errors.push(`${target.name}: ${error instanceof Error ? error.message : String(error)}`);
    }
  }
  throw new AllTargetsFailed(`All targets failed: ${errors.join("; ")}`);
}

lib/targets.ts

import { CircuitBreaker } from "./breaker.ts";
import { interactive } from "./clients.ts";
import type { Target } from "./resilient.ts";
 
// Breakers live at module scope so they are shared by all requests in this process.
export const targets: Target[] = [
  { name: "primary", client: interactive, model: "claude-opus-5-5", breaker: new CircuitBreaker() },
  { name: "fallback", client: interactive, model: "claude-sonnet-5-5", breaker: new CircuitBreaker() },
];

A route handler uses it and degrades when everything fails, keeping the user's question so they do not lose their work:

app/api/answer/route.ts

import { z } from "zod";
import { AllTargetsFailed, createResilient } from "@/lib/resilient";
import { targets } from "@/lib/targets";
 
const Body = z.object({ question: z.string().min(1).max(4000) });
 
export async function POST(request: Request) {
  const parsed = Body.safeParse(await request.json());
  if (!parsed.success) return Response.json({ error: "Invalid request" }, { status: 400 });
 
  try {
    const { message, target } = await createResilient(
      targets,
      { max_tokens: 2048, messages: [{ role: "user", content: parsed.data.question }] },
      { deadlineMs: 25_000, signal: request.signal },
    );
    const text = message.content.flatMap((b) => (b.type === "text" ? [b.text] : [])).join("");
    return Response.json({ text, degraded: target !== "primary" });
  } catch (error) {
    if (error instanceof AllTargetsFailed) {
      console.error(error.message); // also record it on the trace
      return Response.json(
        { text: null, degraded: true, message: "The assistant is temporarily unavailable. Your question was kept; please try again in a minute.", question: parsed.data.question },
        { status: 503, headers: { "retry-after": "60" } },
      );
    }
    throw error; // a bug (for example a 400): let it surface
  }
}

Testing failure paths#

Failure handling is code that rarely runs, which is exactly why it must be tested. The SDK's error classes can be constructed directly, so a fake client can fail on demand:

resilient.test.ts

import { test } from "node:test";
import assert from "node:assert/strict";
import Anthropic from "@anthropic-ai/sdk";
import { CircuitBreaker } from "./lib/breaker.ts";
import { AllTargetsFailed, createResilient, type Target } from "./lib/resilient.ts";
 
const ok = { id: "msg_1", content: [{ type: "text", text: "hi", citations: null }], stop_reason: "end_turn" } as unknown as Anthropic.Message;
const overloaded = () => new Anthropic.InternalServerError(529, { type: "overloaded_error" }, "Overloaded", new Headers());
const badRequest = () => new Anthropic.BadRequestError(400, { type: "invalid_request_error" }, "Bad request", new Headers());
 
function fake(behavior: () => Promise<Anthropic.Message>) {
  let calls = 0;
  const client = { messages: { create: async () => (calls++, behavior()) } } as unknown as Anthropic;
  return { client, calls: () => calls };
}
 
const target = (name: string, client: Anthropic, breaker = new CircuitBreaker()): Target => ({ name, client, model: "claude-opus-5-5", breaker });
const params = { max_tokens: 100, messages: [{ role: "user" as const, content: "hi" }] };
 
test("falls back when the primary is overloaded", async () => {
  const primary = fake(() => Promise.reject(overloaded()));
  const fallback = fake(() => Promise.resolve(ok));
  const result = await createResilient([target("primary", primary.client), target("fallback", fallback.client)], params, { deadlineMs: 1000 });
  assert.equal(result.target, "fallback");
});
 
test("does not fall back on a bad request", async () => {
  const primary = fake(() => Promise.reject(badRequest()));
  const fallback = fake(() => Promise.resolve(ok));
  await assert.rejects(createResilient([target("primary", primary.client), target("fallback", fallback.client)], params, { deadlineMs: 1000 }), Anthropic.BadRequestError);
  assert.equal(fallback.calls(), 0);
});
 
test("throws AllTargetsFailed when everything is down", async () => {
  const down = fake(() => Promise.reject(overloaded()));
  await assert.rejects(createResilient([target("a", down.client), target("b", down.client)], params, { deadlineMs: 1000 }), AllTargetsFailed);
});
 
test("circuit opens after repeated failures, then lets one trial through after the cooldown", () => {
  let now = 0;
  const breaker = new CircuitBreaker({ failureThreshold: 3, windowMs: 10_000, cooldownMs: 5_000 }, () => now);
  for (let i = 0; i < 3; i++) breaker.onFailure();
  assert.equal(breaker.current, "open");
  assert.equal(breaker.allow(), false);
 
  now = 5_000;
  assert.equal(breaker.allow(), true); // trial request
  breaker.markTrial();
  assert.equal(breaker.allow(), false); // only one trial at a time
  breaker.onSuccess();
  assert.equal(breaker.current, "closed");
});
 
test("skips a target whose circuit is open", async () => {
  const breaker = new CircuitBreaker({ failureThreshold: 1, windowMs: 10_000, cooldownMs: 60_000 });
  breaker.onFailure();
  const primary = fake(() => Promise.resolve(ok));
  const fallback = fake(() => Promise.resolve(ok));
  const result = await createResilient([target("primary", primary.client, breaker), target("fallback", fallback.client)], params, { deadlineMs: 1000 });
  assert.equal(result.target, "fallback");
  assert.equal(primary.calls(), 0);
});

More practices that matter#

Smooth your own traffic. Rate limits are per organization and model. Background jobs should run with a concurrency limit, or through the Batches API, so they never starve interactive traffic.
Protect yourself from your users. Per-user rate limits and budgets (see the billing article) stop one user or a runaway loop from consuming everyone's capacity.
Make user actions idempotent. A user who double-clicks "Send" or retries after a timeout should not create two tickets or two charges.
Handle mid-stream failures in the UI. Keep the partial text, mark it as interrupted, and offer a retry, as in the streaming article.
Persist long work. Anything that takes longer than a request belongs in a durable job (durable execution), not in a request handler.
Test the degraded path. Run a game day: force the primary to fail in staging and watch what users would see.

Common mistakes#

Retrying everything. Retrying 400s and auth errors wastes time and hides bugs.
Retries on top of retries. Your retry loop around the SDK's retry loop around a proxy's retries multiplies load during an outage. Retry in one layer.
No deadline. Per-attempt timeouts without an overall deadline can keep a user waiting for minutes.
Untested fallbacks. A fallback model that has never been evaluated on your prompts may answer badly; one that has never been called may not even be configured correctly.
Error pages that lose the user's input. Degrade with their text preserved.

Try it

Break it on purpose

Run the tests with node --test resilient.test.ts.
Add a test where the primary hangs: a fake create that only rejects when its signal aborts. Check that a 200 ms deadline ends with AllTargetsFailed in about 200 ms. (Timers from AbortSignal.timeout do not keep Node.js running, so start a setTimeout in the test to keep the process alive.) Then ask: what would you change so a hung primary leaves time for the fallback? (Hint: a per-attempt timeout shorter than the deadline.)
Point the primary at an invalid model name in staging and check that your route returns a clear error rather than falling back silently.

Check your understanding

Key takeaways

Retry transient failures (429, 5xx, 529, timeouts) and fix the rest; the SDK already retries twice with backoff.
Tune timeouts and retries per use case, and stream long outputs.
Wrap calls in a deadline with a fallback target and a circuit breaker; degrade gracefully while keeping user input.
Test failure paths with fake clients and real SDK error classes.

Next steps#

Security

Threat-model your AI features and defend against prompt injection.

Cost engineering and model routing

Reliability's twin: keeping the bill predictable.

Finished this article? Track it to see your progress on the roadmap.