Tutorial: Chaos-Test an LLM Client with a Fault-Injecting Proxy

Build a proxy that injects 429s, 529s, slow responses, and dropped connections, then load-test a naive client against a resilient one with fallbacks and a breaker.

Advanced22 min readUpdated October 20, 2026

On this page· What you will build

Why this matters

Reliability builds retries, a fallback model, a deadline, and a circuit breaker. That code only runs when something goes wrong, which is exactly when you have no time to find out it does not work. This tutorial makes things go wrong on purpose. You put a small proxy between the SDK and the API that injects the failures real providers produce: rate limits with retry-after, overload errors, slow responses, and dropped connections. Then you send the same load through a default client and a resilient one, scenario by scenario, and compare how many requests were answered, how long users waited, and how much extra traffic the retries caused.

What you will build#

scenario        client      answered  fallback  p50 ms  p95 ms  calls/req  faults injected
primary-outage  naive           0/24         0    1323    1459        3.0  {"overloaded":72}
primary-outage  resilient      24/24        24     111     654        1.5  {"overloaded":12}
slow-primary    naive          24/24         0    5146    5158        1.0  {"slow":24}
slow-primary    resilient      24/24        24     101    6601        1.5  {"slow":12}

A chaos proxy: an HTTP server the SDK talks to instead of the API. It forwards requests upstream unless a rule says to fail them first.
Five scenarios: healthy, flaky, rate-limited, primary outage, and slow primary.
Two clients: the SDK's defaults, and the article's resilient design with short attempts, a deadline, a fallback model, and a circuit breaker.
A load test that runs every scenario against both clients and prints the comparison.

Hover or tap a component to see what it does.

Chaos proxy load test: two clients send the same load through a proxy that injects real failure shapes before forwarding to the API.

Show text description

The proxy makes failures happen on demand, so the code that handles them actually runs.

Load test: 24 requests per client per scenario, 6 at a time; prints the comparison table.
Naive client: One model, 2 retries, a 10-minute timeout.
Resilient client: 3-second attempts, an 8-second deadline, a breaker per model, Sonnet as fallback.
Chaos proxy: Fails requests by rule and model, with the API's real status codes and error envelopes.
Claude API: Requests that pass the rules are forwarded with the caller's headers.

You need Node.js 22.18 or later and an Anthropic API key. A full run sends about 350 short requests; use --requests 8 while you experiment.

1. Set up the project#

mkdir chaos-test && cd chaos-test
npm init -y
npm pkg set type=module
npm install @anthropic-ai/sdk
npm install -D typescript @types/node
mkdir src test
npm pkg set scripts.load="node --env-file=.env src/load.ts" scripts.test="node --test" scripts.typecheck="tsc"

Copy the tsconfig.json from Foundations Tutorial 1 and add your key to .env.

2. The chaos proxy#

The SDK accepts a baseURL. Point it at this proxy and every request passes through code you control.

src/chaos-proxy.ts

import { createServer, type IncomingMessage, type Server, type ServerResponse } from "node:http";
import type { AddressInfo } from "node:net";
import { setTimeout as sleep } from "node:timers/promises";
 
export type Fault =
  | { kind: "overloaded" } // 529, the API is overloaded
  | { kind: "rate_limited"; retryAfterSeconds: number } // 429 with a retry-after header
  | { kind: "server_error" } // 500
  | { kind: "slow"; delayMs: number } // answers, but late
  | { kind: "reset" }; // the connection drops without a response
 
export interface Rule {
  fault: Fault;
  rate: number; // share of matching requests that get the fault, 0 to 1
  model?: string; // only requests for this model; all models when omitted
}
 
export const SCENARIOS: Record<string, Rule[]> = {
  healthy: [],
  flaky: [{ fault: { kind: "overloaded" }, rate: 0.2 }, { fault: { kind: "reset" }, rate: 0.05 }],
  "rate-limited": [{ fault: { kind: "rate_limited", retryAfterSeconds: 1 }, rate: 0.3 }],
  "primary-outage": [{ fault: { kind: "overloaded" }, rate: 1, model: "claude-opus-5-5" }],
  "slow-primary": [{ fault: { kind: "slow", delayMs: 5_000 }, rate: 1, model: "claude-opus-5-5" }],
};
 
export interface ProxyStats {
  requests: number;
  faults: Record<string, number>;
}
 
const errorBody = (type: string, message: string) => JSON.stringify({ type: "error", error: { type, message } });
 
/**
 * A fault-injecting HTTP proxy. Point the SDK's baseURL at it; it forwards to `upstream`
 * (the real API, or a mock) unless a rule says to fail the request first.
 */
export async function startChaosProxy(upstream: string, rules: Rule[], random: () => number = Math.random) {
  const stats: ProxyStats = { requests: 0, faults: {} };
 
  async function handle(req: IncomingMessage, res: ServerResponse) {
    const chunks: Buffer[] = [];
    for await (const chunk of req) chunks.push(chunk as Buffer);
    const body = Buffer.concat(chunks);
    stats.requests++;
 
    let model = "";
    try {
      model = (JSON.parse(body.toString("utf8")) as { model?: string }).model ?? "";
    } catch {}
    const rule = rules.find((r) => (!r.model || r.model === model) && random() < r.rate);
    if (rule) stats.faults[rule.fault.kind] = (stats.faults[rule.fault.kind] ?? 0) + 1;
 
    switch (rule?.fault.kind) {
      case "overloaded":
        res.writeHead(529, { "content-type": "application/json" }).end(errorBody("overloaded_error", "Overloaded (injected)"));
        return;
      case "rate_limited":
        res
          .writeHead(429, { "content-type": "application/json", "retry-after": String(rule.fault.retryAfterSeconds) })
          .end(errorBody("rate_limit_error", "Rate limited (injected)"));
        return;
      case "server_error":
        res.writeHead(500, { "content-type": "application/json" }).end(errorBody("api_error", "Internal error (injected)"));
        return;
      case "reset":
        req.socket.destroy();
        return;
      case "slow":
        await sleep(rule.fault.delayMs);
        if (res.destroyed) return; // the client gave up
    }
 
    // Forward to the upstream API, with the caller's headers (including its API key).
    const headers = Object.fromEntries(Object.entries(req.headers).filter(([k]) => !["host", "connection", "content-length"].includes(k))) as Record<string, string>;
    const upstreamRes = await fetch(new URL(req.url ?? "/", upstream), { method: req.method, headers, body: body.length ? body : undefined });
    const keep = ["content-type", "request-id", "retry-after", "anthropic-ratelimit-requests-remaining"];
    res.writeHead(upstreamRes.status, Object.fromEntries(keep.flatMap((k) => (upstreamRes.headers.get(k) ? [[k, upstreamRes.headers.get(k)!]] : []))));
    res.end(Buffer.from(await upstreamRes.arrayBuffer()));
  }
 
  const server: Server = createServer((req, res) => {
    handle(req, res).catch(() => res.destroy());
  });
  await new Promise<void>((resolve) => server.listen(0, "127.0.0.1", resolve));
  const { port } = server.address() as AddressInfo;
  return {
    url: `http://127.0.0.1:${port}`,
    stats,
    reset: () => Object.assign(stats, { requests: 0, faults: {} }),
    close: () => new Promise<void>((resolve) => server.close(() => resolve())),
  };
}

Each fault matches what the API really sends: the same status codes, the same error envelope with a type, and a retry-after header on rate limits, so the SDK's retry logic reacts exactly as it would in production. The proxy listens only on 127.0.0.1 and forwards your API key upstream: keep it a local test tool.

3. The circuit breaker#

src/breaker.ts

type State = "closed" | "open" | "half-open";
 
/** Stop sending traffic to a dependency that keeps failing; let one trial through after a cooldown. */
export class CircuitBreaker {
  state: State = "closed";
  private failures: number[] = [];
  private openedAt = 0;
  private trialInFlight = false;
  private readonly threshold: number;
  private readonly windowMs: number;
  private readonly cooldownMs: number;
  private readonly now: () => number;
 
  constructor(opts: { threshold: number; windowMs: number; cooldownMs: number }, now: () => number = Date.now) {
    this.threshold = opts.threshold;
    this.windowMs = opts.windowMs;
    this.cooldownMs = opts.cooldownMs;
    this.now = now;
  }
 
  allow(): boolean {
    if (this.state === "open" && this.now() - this.openedAt >= this.cooldownMs) this.state = "half-open";
    if (this.state === "open") return false;
    if (this.state === "half-open") {
      if (this.trialInFlight) return false; // one trial at a time
      this.trialInFlight = true;
    }
    return true;
  }
 
  onSuccess() {
    this.state = "closed";
    this.failures = [];
    this.trialInFlight = false;
  }
 
  onFailure() {
    const t = this.now();
    this.trialInFlight = false;
    if (this.state === "half-open") return this.open(t);
    this.failures = [...this.failures.filter((f) => t - f < this.windowMs), t];
    if (this.failures.length >= this.threshold) this.open(t);
  }
 
  private open(t: number) {
    this.state = "open";
    this.openedAt = t;
    this.failures = [];
  }
}

A compact version of the article's breaker. The clock is injectable, so a test can move time instead of waiting for it.

4. Two clients#

src/clients.ts

import Anthropic from "@anthropic-ai/sdk";
import { CircuitBreaker } from "./breaker.ts";
 
export interface Outcome {
  ok: boolean;
  servedBy?: string; // which model answered
  error?: string;
  ms: number;
}
 
export type Ask = (question: string) => Promise<Outcome>;
const MESSAGES = (q: string) => [{ role: "user" as const, content: q }];
 
/** The default: one model, the SDK's default retries (2) and timeout (10 minutes). */
export function naiveClient(baseURL: string): Ask {
  const client = new Anthropic({ baseURL });
  return async (question) => {
    const started = performance.now();
    try {
      await client.messages.create({ model: "claude-opus-5-5", max_tokens: 1024, messages: MESSAGES(question) });
      return { ok: true, servedBy: "claude-opus-5-5", ms: performance.now() - started };
    } catch (error) {
      return { ok: false, error: error instanceof Error ? error.constructor.name : "error", ms: performance.now() - started };
    }
  };
}
 
const isTransient = (error: unknown) =>
  error instanceof Anthropic.APIConnectionError || // includes timeouts
  (error instanceof Anthropic.APIError && (error.status === 408 || error.status === 409 || error.status === 429 || (error.status ?? 0) >= 500));
 
/**
 * The article's design: short per-attempt timeouts, one deadline for the whole request,
 * a fallback model, and a circuit breaker per model shared by all requests.
 */
export function resilientClient(baseURL: string, opts = { deadlineMs: 8_000, attemptTimeoutMs: 3_000 }): Ask {
  const client = new Anthropic({ baseURL, maxRetries: 1, timeout: opts.attemptTimeoutMs });
  const targets = ["claude-opus-5-5", "claude-sonnet-5-5"].map((model) => ({
    model,
    breaker: new CircuitBreaker({ threshold: 3, windowMs: 30_000, cooldownMs: 10_000 }),
  }));
 
  return async (question) => {
    const started = performance.now();
    const deadline = AbortSignal.timeout(opts.deadlineMs);
    const errors: string[] = [];
    for (const target of targets) {
      if (deadline.aborted) break;
      if (!target.breaker.allow()) {
        errors.push(`${target.model}: circuit open`);
        continue;
      }
      try {
        await client.messages.create({ model: target.model, max_tokens: 1024, messages: MESSAGES(question) }, { signal: deadline });
        target.breaker.onSuccess();
        return { ok: true, servedBy: target.model, ms: performance.now() - started };
      } catch (error) {
        if (!isTransient(error) && !deadline.aborted) throw error; // a bug, such as a 400: do not hide it
        target.breaker.onFailure();
        errors.push(`${target.model}: ${error instanceof Error ? error.constructor.name : "error"}`);
      }
    }
    // Degrade: the caller shows a "try again" message and keeps the user's input.
    return { ok: false, error: errors.join("; ") || "deadline exceeded", ms: performance.now() - started };
  };
}

The resilient client's numbers are a budget: an attempt may take 3 seconds, a retry doubles that, and the whole request must finish in 8. That leaves room for the fallback even when the primary times out twice.

5. The load test#

src/load.ts

import { parseArgs } from "node:util";
import { SCENARIOS, startChaosProxy } from "./chaos-proxy.ts";
import { type Ask, naiveClient, type Outcome, resilientClient } from "./clients.ts";
 
const { values } = parseArgs({
  options: {
    scenario: { type: "string", default: "all" },
    requests: { type: "string", default: "24" },
    concurrency: { type: "string", default: "6" },
  },
});
const upstream = process.env.UPSTREAM_URL ?? "https://api.anthropic.com";
const names = values.scenario === "all" ? Object.keys(SCENARIOS) : [values.scenario];
const total = Number(values.requests);
 
const pct = (xs: number[], p: number) => [...xs].sort((a, b) => a - b)[Math.min(xs.length - 1, Math.ceil((p / 100) * xs.length) - 1)] ?? 0;
 
async function run(ask: Ask): Promise<Outcome[]> {
  const outcomes: Outcome[] = [];
  let next = 0;
  await Promise.all(
    Array.from({ length: Number(values.concurrency) }, async () => {
      while (next < total) {
        next++;
        outcomes.push(await ask("In one sentence, what is a circuit breaker?"));
      }
    }),
  );
  return outcomes;
}
 
console.log(`${"scenario".padEnd(16)}${"client".padEnd(11)}${"answered".padStart(9)}${"fallback".padStart(10)}${"p50 ms".padStart(8)}${"p95 ms".padStart(8)}${"calls/req".padStart(11)}  faults injected`);
for (const name of names) {
  const rules = SCENARIOS[name];
  if (!rules) throw new Error(`Unknown scenario ${name}. Known: ${Object.keys(SCENARIOS).join(", ")}`);
  for (const [label, make] of [["naive", naiveClient], ["resilient", resilientClient]] as const) {
    const proxy = await startChaosProxy(upstream, rules); // fresh proxy and client: no state carried over
    const outcomes = await run(make(proxy.url));
    await proxy.close();
    const ok = outcomes.filter((o) => o.ok);
    const fallback = ok.filter((o) => o.servedBy !== "claude-opus-5-5").length;
    const ms = outcomes.map((o) => o.ms);
    console.log(
      `${name.padEnd(16)}${label.padEnd(11)}${`${ok.length}/${outcomes.length}`.padStart(9)}${String(fallback).padStart(10)}` +
        `${String(Math.round(pct(ms, 50))).padStart(8)}${String(Math.round(pct(ms, 95))).padStart(8)}` +
        `${(proxy.stats.requests / outcomes.length).toFixed(1).padStart(11)}  ${JSON.stringify(proxy.stats.faults)}`,
    );
  }
}

calls/req is the number of HTTP requests that reached the proxy per user request. It is the cost of resilience: every retry is traffic, and during an incident that traffic lands on a provider that is already struggling.

6. Run the chaos#

npm run load

scenario        client      answered  fallback  p50 ms  p95 ms  calls/req  faults injected
healthy         naive          24/24         0     129     180        1.0  {}
healthy         resilient      24/24         0     108     151        1.0  {}
flaky           naive          24/24         0     109    1517        1.3  {"overloaded":7,"reset":1}
flaky           resilient      24/24         1     116     588        1.2  {"overloaded":3,"reset":1}
rate-limited    naive          21/24         0     136    2054        1.5  {"rate_limited":16}
rate-limited    resilient      24/24         1     116    1143        1.4  {"rate_limited":10}
primary-outage  naive           0/24         0    1323    1459        3.0  {"overloaded":72}
primary-outage  resilient      24/24        24     111     654        1.5  {"overloaded":12}
slow-primary    naive          24/24         0    5146    5158        1.0  {"slow":24}
slow-primary    resilient      24/24        24     101    6601        1.5  {"slow":12}

What each scenario teaches:

Healthy. Both clients are the same. Resilience costs nothing when nothing fails.
Flaky. The SDK's own retries absorb occasional 529s and dropped connections for both clients. The naive client's p95 is higher because its backoff runs longer.
Rate-limited. The SDK honors retry-after, so most requests succeed after a wait; with 30% of attempts limited, a few naive requests still fail three times in a row. The resilient client falls back for those.
Primary outage. The naive client fails every request, and does it slowly: three attempts each, 72 calls into an overloaded API. The resilient client's first wave of requests tries the primary twice each, the breaker opens after three failures, and every later request goes straight to the fallback. Twelve faulted calls instead of seventy-two, and every user gets an answer.
Slow primary. The naive client answers every request, after five seconds each: its 10-minute default timeout never fires. The resilient client's first requests wait out two 3-second attempts before falling back, which is the 6.6-second p95, and then the breaker routes everything else to the fallback immediately. A slow dependency is often worse than a dead one, because nothing ever fails.

7. Test the failure paths#

The tests use a tiny local upstream and a random that always fires, so every assertion is exact.

test/chaos.test.ts

import assert from "node:assert/strict";
import { createServer } from "node:http";
import type { AddressInfo } from "node:net";
import { after, test } from "node:test";
import { CircuitBreaker } from "../src/breaker.ts";
import { startChaosProxy } from "../src/chaos-proxy.ts";
import { resilientClient } from "../src/clients.ts";
 
// A tiny upstream that answers like the Messages API, and returns 400 for a "bad" question.
const upstream = createServer(async (req, res) => {
  let body = "";
  for await (const chunk of req) body += chunk;
  const { model, messages } = JSON.parse(body) as { model: string; messages: { content: string }[] };
  if (messages[0]!.content === "bad") {
    res.writeHead(400, { "content-type": "application/json" }).end(JSON.stringify({ type: "error", error: { type: "invalid_request_error", message: "bad" } }));
    return;
  }
  res.writeHead(200, { "content-type": "application/json" }).end(
    JSON.stringify({ id: "msg_1", type: "message", role: "assistant", model, stop_reason: "end_turn", stop_sequence: null, usage: { input_tokens: 5, output_tokens: 2 }, content: [{ type: "text", text: "ok" }] }),
  );
});
await new Promise<void>((r) => upstream.listen(0, "127.0.0.1", r));
const UPSTREAM = `http://127.0.0.1:${(upstream.address() as AddressInfo).port}`;
process.env.ANTHROPIC_API_KEY = "test-key";
after(() => upstream.close());
 
const always = () => 0; // every rule fires
 
test("the proxy injects the configured fault, only for the matching model", async () => {
  const proxy = await startChaosProxy(UPSTREAM, [{ fault: { kind: "rate_limited", retryAfterSeconds: 7 }, rate: 1, model: "claude-opus-5-5" }], always);
  const call = (model: string) =>
    fetch(`${proxy.url}/v1/messages`, { method: "POST", headers: { "content-type": "application/json" }, body: JSON.stringify({ model, messages: [{ role: "user", content: "hi" }] }) });
  const limited = await call("claude-opus-5-5");
  assert.equal(limited.status, 429);
  assert.equal(limited.headers.get("retry-after"), "7");
  assert.equal((await call("claude-sonnet-5-5")).status, 200); // forwarded
  assert.deepEqual(proxy.stats, { requests: 2, faults: { rate_limited: 1 } });
  await proxy.close();
});
 
test("during a primary outage, the resilient client falls back and the breaker stops the retries", async () => {
  const proxy = await startChaosProxy(UPSTREAM, [{ fault: { kind: "overloaded" }, rate: 1, model: "claude-opus-5-5" }], always);
  const ask = resilientClient(proxy.url);
  const outcomes = [];
  for (let i = 0; i < 6; i++) outcomes.push(await ask("hello"));
  assert.ok(outcomes.every((o) => o.ok && o.servedBy === "claude-sonnet-5-5"));
  // Opus was tried by the first 3 requests (2 attempts each), then the breaker opened: 6 failures, 6 fallbacks.
  assert.deepEqual(proxy.stats, { requests: 12, faults: { overloaded: 6 } });
  await proxy.close();
});
 
test("a bad request is thrown, not retried or hidden behind a fallback", async () => {
  const proxy = await startChaosProxy(UPSTREAM, [], always);
  await assert.rejects(resilientClient(proxy.url)("bad"), /400/);
  assert.equal(proxy.stats.requests, 1);
  await proxy.close();
});
 
test("the breaker opens after the threshold, then lets one trial through after the cooldown", () => {
  let now = 0;
  const b = new CircuitBreaker({ threshold: 3, windowMs: 1_000, cooldownMs: 500 }, () => now);
  for (let i = 0; i < 3; i++) b.onFailure();
  assert.equal(b.allow(), false);
  now = 600;
  assert.equal(b.allow(), true); // the trial
  assert.equal(b.allow(), false); // only one at a time
  b.onSuccess();
  assert.equal(b.state, "closed");
});

npm test   # 4 passing, no API key needed

Common mistakes#

Testing only the happy path. Failure handling that has never run is a guess. Inject each failure kind at least once, in a test and in a load run.
Default timeouts on interactive paths. Ten minutes is right for a batch job and wrong for a user waiting on a page. A slow dependency never fails; your timeout has to make it fail.
Retries without a deadline. Each layer retrying on its own multiplies traffic and wait time. Budget the whole request.
Retrying non-transient errors. A 400 fails the same way every time. Throw it, so the bug is found.
A fallback you have not evaluated. The resilient client answered every request from the fallback during the outage; know from your evals how good those answers are.
Chaos against production. Run the proxy on your machine or in a test environment, with its own API key and spending limit.

Try it

Break it some more

Add a truncate fault that cuts the response body in half, and check that the SDK raises an error rather than returning a partial message.
Add a scenario where the primary recovers after 20 seconds, and log the breaker's state changes to watch it go open, half-open, and closed.
Make the resilient client stream, and inject the slow fault after the first bytes instead of before. Which timeout fires now?
Add a third target on a different provider deployment, and a scenario where both Anthropic models are overloaded.

Check your understanding

Key takeaways

Test failure handling by injecting real failure shapes: 429 with retry-after, 529, slow responses, and dropped connections.
Set per-attempt timeouts and an overall deadline for interactive requests, and leave room in it for a fallback.
Use a circuit breaker so a dependency that is down costs a few calls, not a full retry cycle per request.
Compare clients on answered requests, latency percentiles, and extra calls, and re-run the comparison when you change retry or timeout settings.

Next steps#

Security

Next article: prompt injection, data leaks, and agent risks.

Reliability

Revisit the retry, fallback, and breaker design this test exercises.

Finished this article? Track it to see your progress on the roadmap.