Tutorial: Build a Model Benchmark Harness to Choose Model and Effort

Build a harness that runs labelled tasks across Claude models and effort levels, measures accuracy, latency, and cost, and picks the cheapest good-enough setup.

Intermediate20 min readUpdated October 18, 2026

On this page· What you will build

Why this matters

Choosing models and providers ends with a rule: measure on your own tasks, and pick the cheapest configuration that is good enough. This tutorial builds the tool that makes that rule practical. Given a set of labelled examples, it runs every model and effort combination you care about, scores them, and tells you which one to ship, with numbers you can put in a pull request. It is also the seed of an eval suite: the same harness tells you whether a prompt change or a new model release made things better or worse.

What you will build#

A command-line benchmark that:

Checks that every configured model is available to your key before spending anything.
Loads a labelled dataset of support tickets and validates it with Zod.
Runs six model × effort configurations over the same tickets, a few requests at a time.
Reports accuracy, p50 and p95 latency, total cost, and cost per correct answer, then recommends the cheapest configuration above an accuracy bar you set in advance.
Lists the hardest tickets and saves every run as JSON for comparison.

Hover or tap a component to see what it does.

Model benchmark harness: labelled tickets run through Haiku, Sonnet, and Opus at several effort levels, and the report recommends the cheapest configuration that clears an accuracy bar set in advance.

Show text description

Every configuration sees exactly the same inputs, so differences in the report come from the model and effort, not from the data.

Tickets: Real-looking support tickets with the severity a person assigned, validated with Zod before use.
Runner: Checks the models exist, then runs every configuration over the same tickets with a small concurrency limit.
Haiku 5.5: Cheapest and fastest. Tried at low and medium effort.
Sonnet 5.5: Mid-priced. Tried at low and medium effort.
Opus 5.5: Most capable. Tried at low and medium effort.
Report: Accuracy, p50 and p95 latency, cost, and cost per correct answer; picks the cheapest configuration above the bar set in advance.

You need Node.js 22.18 or later and an Anthropic API key in ANTHROPIC_API_KEY. One full run makes 120 small requests and costs well under a dollar at the prices in config.ts, most of it on Opus. Doing Foundations Tutorial 1 first helps: this is the same triage task, now asked of real models.

1. Set up the project#

Create the project

mkdir model-bench && cd model-bench
npm init -y
npm pkg set type=module
npm install @anthropic-ai/sdk zod
npm install -D typescript @types/node
mkdir src data test

Add your key and the scripts
Put ANTHROPIC_API_KEY=sk-ant-... in a .env file (and .env and results/ in .gitignore), then:
```
npm pkg set scripts.bench="node --env-file=.env src/bench.ts" scripts.test="node --test" scripts.typecheck="tsc"
```
Copy the tsconfig.json from Foundations Tutorial 1.

2. The dataset#

A benchmark is only as good as its labels. Twenty tickets, each with the severity a person assigned, spread across all three classes:

data/tickets.json

[
  { "id": "t01", "severity": "high", "text": "Checkout is down for every customer since 09:10. Card payments return a 502 and we are losing orders." },
  { "id": "t02", "severity": "high", "text": "Several customers report they were charged twice for the same order this morning." },
  { "id": "t03", "severity": "high", "text": "Nobody on the EU cluster can log in. The sign-in page spins forever." },
  { "id": "t04", "severity": "high", "text": "A user says they can see another company's invoices in their dashboard. Possible data leak." },
  { "id": "t05", "severity": "high", "text": "The nightly backup job has failed for three days and the last good backup is from Monday." },
  { "id": "t06", "severity": "high", "text": "Our public API returns 500 for all write requests; integrations of our biggest customers are failing." },
  { "id": "t07", "severity": "medium", "text": "Search results take 8 to 10 seconds to load for some users on large workspaces." },
  { "id": "t08", "severity": "medium", "text": "CSV export fails with an error when the report has more than 50,000 rows." },
  { "id": "t09", "severity": "medium", "text": "Password reset emails arrive about 20 minutes late for Outlook users." },
  { "id": "t10", "severity": "medium", "text": "The mobile app crashes when uploading a photo larger than 10 MB. Smaller photos work." },
  { "id": "t11", "severity": "medium", "text": "Webhook deliveries are retried twice even when our endpoint returns 200." },
  { "id": "t12", "severity": "medium", "text": "Dashboard charts show yesterday's numbers until the user refreshes the page manually." },
  { "id": "t13", "severity": "medium", "text": "Some customers in Brazil see prices in USD instead of BRL on the pricing page." },
  { "id": "t14", "severity": "low", "text": "Typo on the pricing page: 'Anual plan' should be 'Annual plan'." },
  { "id": "t15", "severity": "low", "text": "Could you add a dark mode to the settings screen? Not urgent, just a nice-to-have." },
  { "id": "t16", "severity": "low", "text": "The tooltip on the export button is slightly cut off on very small screens." },
  { "id": "t17", "severity": "low", "text": "How do I change the email address on my account?" },
  { "id": "t18", "severity": "low", "text": "The footer still says 2025 in the copyright line." },
  { "id": "t19", "severity": "low", "text": "Feature request: let us reorder columns in the orders table by drag and drop." },
  { "id": "t20", "severity": "low", "text": "The onboarding video is a little quiet; maybe raise the volume in the next version." }
]

Some of these are deliberately borderline. Is a backup that has failed for three days "high" when nothing is down yet? A person decided yes. Disagreements like that are exactly what a benchmark should surface, and they are often a labelling or task-definition problem rather than a model problem.

3. Everything you tune, in one file#

src/config.ts

// Everything you would tune lives here, so a benchmark run is reviewable in one diff.
 
// USD per million tokens at the time of writing; check the pricing page before relying on these.
// Claude Haiku 5.5's rate applies to prompts of 100K tokens or fewer, which covers every prompt here.
export const PRICES = {
  "claude-opus-5-5": { input: 4, output: 20 },
  "claude-sonnet-5-5": { input: 2, output: 10 },
  "claude-haiku-5-5": { input: 0.1, output: 0.5 },
} as const;
 
export type Model = keyof typeof PRICES;
export type Effort = "low" | "medium" | "high";
export interface Config {
  model: Model;
  effort: Effort;
}
 
export const CONFIGS: Config[] = [
  { model: "claude-haiku-5-5", effort: "low" },
  { model: "claude-haiku-5-5", effort: "medium" },
  { model: "claude-sonnet-5-5", effort: "low" },
  { model: "claude-sonnet-5-5", effort: "medium" },
  { model: "claude-opus-5-5", effort: "low" },
  { model: "claude-opus-5-5", effort: "medium" },
];
 
// Decide what "good enough" means BEFORE looking at results, or you will move the bar to fit them.
export const ACCURACY_BAR = 0.9;
 
// How many requests run at once. Higher is faster but closer to your rate limit.
export const CONCURRENCY = 4;
 
export const SYSTEM_PROMPT =
  "You triage customer support tickets. " +
  "high: an outage, data loss or exposure, billing errors, or most users blocked. " +
  "medium: a real bug with a workaround, or one that affects some users. " +
  "low: cosmetic issues, questions, and feature requests. " +
  "Reply with exactly one word: low, medium, or high.";

This harness uses Claude Haiku 5.5, the current Haiku. Unlike Haiku 4.5 in the article's example, it supports the effort setting, so all three models are compared at the same two effort levels.

The system prompt defines each class instead of just naming it. "High" means different things in different companies; the definitions are part of the task, and they belong in version control next to the benchmark.

4. Two small helpers#

Running 20 requests one after another is slow; running all 120 at once hits rate limits. A tiny pool keeps a fixed number in flight, the same idea as Promise.all from the TypeScript article, with a ceiling.

src/stats.ts

/** Run `fn` over `items` with at most `limit` calls in flight, keeping results in input order. */
export async function mapPool<T, R>(items: T[], limit: number, fn: (item: T) => Promise<R>): Promise<R[]> {
  const results = new Array<R>(items.length);
  let next = 0;
  const worker = async () => {
    while (next < items.length) {
      const i = next++;
      results[i] = await fn(items[i]!);
    }
  };
  await Promise.all(Array.from({ length: Math.min(limit, items.length) }, worker));
  return results;
}
 
/** Nearest-rank percentile of a list of numbers (p between 0 and 100). */
export function percentile(values: number[], p: number): number {
  if (values.length === 0) return 0;
  const sorted = [...values].sort((a, b) => a - b);
  return sorted[Math.min(sorted.length - 1, Math.ceil((p / 100) * sorted.length) - 1)]!;
}

Each worker takes the next index until none are left. Because JavaScript runs one callback at a time, next++ never hands the same index to two workers.

Latency is reported as p50 and p95, not an average: users feel the slow tail, and one 10-second outlier hides inside an average.

test/stats.test.ts

import assert from "node:assert/strict";
import { test } from "node:test";
import { setTimeout as sleep } from "node:timers/promises";
import { mapPool, percentile } from "../src/stats.ts";
 
test("percentile uses the nearest rank", () => {
  const values = [100, 200, 300, 400, 500, 600, 700, 800, 900, 1000];
  assert.equal(percentile(values, 50), 500);
  assert.equal(percentile(values, 95), 1000);
  assert.equal(percentile([], 50), 0);
});
 
test("mapPool keeps input order even when calls finish out of order", async () => {
  const results = await mapPool([30, 10, 20], 3, async (ms) => {
    await sleep(ms);
    return ms;
  });
  assert.deepEqual(results, [30, 10, 20]);
});
 
test("mapPool never runs more than `limit` calls at once", async () => {
  let running = 0;
  let peak = 0;
  await mapPool(Array.from({ length: 10 }, (_, i) => i), 3, async () => {
    peak = Math.max(peak, ++running);
    await sleep(5);
    running--;
  });
  assert.equal(peak, 3);
});

npm test   # 3 passing, no API calls

5. The benchmark#

src/bench.ts

import { mkdir, readFile, writeFile } from "node:fs/promises";
import Anthropic from "@anthropic-ai/sdk";
import { z } from "zod";
import { ACCURACY_BAR, CONCURRENCY, type Config, CONFIGS, PRICES, SYSTEM_PROMPT } from "./config.ts";
import { mapPool, percentile } from "./stats.ts";
 
const client = new Anthropic();
 
const Severity = z.enum(["low", "medium", "high"]);
const Tickets = z.array(z.object({ id: z.string(), severity: Severity, text: z.string().min(1) })).min(1);
type Ticket = z.infer<typeof Tickets>[number];
 
interface Outcome {
  id: string;
  expected: string;
  predicted: string | null;
  correct: boolean;
  ms: number;
  usd: number;
  error?: string;
}
 
// 1. Fail fast on a typo or a model your key cannot use, before spending anything.
const available = new Set<string>();
for await (const model of client.models.list()) available.add(model.id);
const missing = [...new Set(CONFIGS.map((c) => c.model))].filter((id) => !available.has(id));
if (missing.length > 0) throw new Error(`Not available to this API key: ${missing.join(", ")}`);
 
// 2. Load and validate the dataset, exactly like any other untrusted input.
const tickets = Tickets.parse(JSON.parse(await readFile("data/tickets.json", "utf8")));
 
async function classify(config: Config, ticket: Ticket): Promise<Outcome> {
  const started = performance.now();
  try {
    const response = await client.messages.create({
      model: config.model,
      max_tokens: 2048, // covers thinking plus the one-word answer
      output_config: { effort: config.effort },
      system: SYSTEM_PROMPT,
      messages: [{ role: "user", content: `Ticket: ${ticket.text}` }],
    });
    const text = response.content.flatMap((b) => (b.type === "text" ? [b.text] : [])).join("");
    const predicted = /\b(low|medium|high)\b/i.exec(text)?.[1]?.toLowerCase() ?? null;
    const price = PRICES[config.model];
    const usd = (response.usage.input_tokens * price.input + response.usage.output_tokens * price.output) / 1_000_000;
    return { id: ticket.id, expected: ticket.severity, predicted, correct: predicted === ticket.severity, ms: performance.now() - started, usd };
  } catch (error) {
    // The SDK has already retried 429s and 5xx; what reaches here counts as a wrong answer.
    const message = error instanceof Anthropic.APIError ? `${error.status ?? "network"} ${error.message}` : String(error);
    return { id: ticket.id, expected: ticket.severity, predicted: null, correct: false, ms: performance.now() - started, usd: 0, error: message };
  }
}
 
// 3. Run every configuration over the same tickets.
const rows = [];
for (const config of CONFIGS) {
  const outcomes = await mapPool(tickets, CONCURRENCY, (ticket) => classify(config, ticket));
  const correct = outcomes.filter((o) => o.correct).length;
  const usd = outcomes.reduce((sum, o) => sum + o.usd, 0);
  const latencies = outcomes.map((o) => o.ms);
  rows.push({
    ...config,
    accuracy: correct / outcomes.length,
    p50: percentile(latencies, 50),
    p95: percentile(latencies, 95),
    usd,
    usdPerCorrect: correct > 0 ? usd / correct : Infinity,
    errors: outcomes.filter((o) => o.error).length,
    outcomes,
  });
  process.stdout.write(".");
}
console.log("\n");
 
// 4. Report, and pick the cheapest configuration that clears the bar.
const passing = rows.filter((r) => r.accuracy >= ACCURACY_BAR).sort((a, b) => a.usd - b.usd);
const winner = passing[0];
 
console.log(`${"model".padEnd(19)}${"effort".padEnd(8)}${"accuracy".padStart(9)}${"p50".padStart(8)}${"p95".padStart(8)}${"cost".padStart(10)}${"$/correct".padStart(11)}`);
for (const r of rows) {
  const mark = r === winner ? "  <- cheapest above the bar" : r.errors ? `  (${r.errors} errors)` : "";
  console.log(
    `${r.model.padEnd(19)}${r.effort.padEnd(8)}${`${Math.round(r.accuracy * 100)}%`.padStart(9)}` +
      `${`${(r.p50 / 1000).toFixed(1)}s`.padStart(8)}${`${(r.p95 / 1000).toFixed(1)}s`.padStart(8)}` +
      `${`$${r.usd.toFixed(4)}`.padStart(10)}${`$${r.usdPerCorrect.toFixed(5)}`.padStart(11)}${mark}`,
  );
}
const bar = `${Math.round(ACCURACY_BAR * 100)}%`;
console.log(winner ? `\nBar: ${bar} accuracy. Choose ${winner.model} at ${winner.effort} effort.` : `\nNo configuration reached ${bar}. Improve the prompt or the data before choosing.`);
 
// The tickets most configurations got wrong are where the prompt, the labels, or the task definition need work.
const misses = new Map<string, number>();
for (const r of rows) for (const o of r.outcomes) if (!o.correct) misses.set(o.id, (misses.get(o.id) ?? 0) + 1);
const hardest = [...misses].sort((a, b) => b[1] - a[1]).slice(0, 3);
if (hardest.length > 0) console.log(`Hardest tickets: ${hardest.map(([id, n]) => `${id} (wrong in ${n}/${rows.length})`).join(", ")}`);
 
// 5. Keep every run, so you can compare after changing a prompt or when a new model ships.
await mkdir("results", { recursive: true });
const file = `results/${new Date().toISOString().replace(/[:.]/g, "-")}.json`;
await writeFile(file, JSON.stringify({ bar: ACCURACY_BAR, systemPrompt: SYSTEM_PROMPT, rows }, null, 2));
console.log(`Saved ${file}`);

A few choices worth understanding:

The Models API check comes first. A typo in a model ID would otherwise surface as 20 failed requests per configuration, after you had paid for the configurations before it.
A failure counts as a wrong answer. The SDK already retries rate limits and overloaded errors (maxRetries defaults to 2). If a request still fails, a user would have received nothing, so it is not excluded from the score.
max_tokens: 2048 for a one-word answer. The limit covers thinking as well, and the token explorer showed what happens when thinking runs out of room.
Cost per correct answer is the honest unit. A configuration that is half the price but right half as often costs the same per useful result, before counting the cost of the wrong ones.

6. Run it and read the report#

npm run bench

example report (illustrative numbers)

......
 
model              effort   accuracy     p50     p95      cost  $/correct
claude-haiku-5-5   low           85%    0.6s    1.1s   $0.0009   $0.00005
claude-haiku-5-5   medium        90%    0.9s    1.6s   $0.0021   $0.00012  <- cheapest above the bar
claude-sonnet-5-5  low           90%    1.0s    1.8s   $0.0158   $0.00088
claude-sonnet-5-5  medium        95%    1.6s    2.9s   $0.0402   $0.00212
claude-opus-5-5    low           95%    1.4s    2.4s   $0.0301   $0.00158
claude-opus-5-5    medium        95%    2.3s    4.0s   $0.0880   $0.00463
 
Bar: 90% accuracy. Choose claude-haiku-5-5 at medium effort.
Hardest tickets: t05 (wrong in 4/6), t09 (wrong in 2/6), t13 (wrong in 1/6)
Saved results/2026-10-18T19-41-05-937Z.json

How to read a report like this:

Start with the bar, not the winner. Which configurations clear it at all? If none do, the next step is the prompt or the labels, not a bigger model.
Look at the gap in cost. In this illustration the cheapest passing configuration costs a few percent of the most expensive one, for the same task. At a million tickets a month that difference is real money.
Check the latency column against your product. A triage step that runs in a background job can afford a slow p95; an autocomplete cannot.
Read the hardest tickets. Open t05 in the dataset. If most configurations disagree with your label, ask whether the label or the definition in the prompt is wrong before blaming the models.

Common mistakes#

Choosing the bar after seeing the results. You will unconsciously move it to fit the configuration you already liked. Commit it in config.ts first.
Benchmarking with made-up examples only. Synthetic tickets are fine for building the harness; decisions need real traffic.
Comparing average latency. Averages hide the slow tail that users actually notice. Report p50 and p95.
Excluding failed requests from accuracy. A request that failed after retries is a user who got nothing. Count it.
Benchmarking once and never again. Models, prompts, and prices change. Keep the saved results and rerun on every change that matters.

Try it

Extend the harness

Add a --repeat 3 flag that runs every configuration three times and reports the range of accuracy. How much does the same configuration vary between runs?
Write a compare.ts script that loads the two most recent files in results/ and prints, per configuration, the change in accuracy and cost.
Change one sentence of the system prompt, for example the definition of "high", and rerun. Did accuracy move more than the noise you measured in step 1?
Add 10 tickets from a domain you know, including at least three you find hard to label yourself. Which configurations fall below the bar now?

Check your understanding

Key takeaways

A model choice is an engineering decision backed by your own labelled data, not by public leaderboards.
Run every configuration on identical inputs and compare accuracy, p50 and p95 latency, and cost per correct answer.
Set the accuracy bar before running, then choose the cheapest configuration above it.
Check models exist before spending, count failures as wrong, and keep every run so you can compare later.
Small datasets are noisy; repeat runs and grow the dataset before a decision you will ship.

Next steps#

API keys, billing, and security

Next article: keep keys server-side, cost every call, and enforce budgets.

Evals

Grow this harness into a full evaluation suite.

Finished this article? Track it to see your progress on the roadmap.