Tutorial: Compare Conversation Memory Strategies with a Replay Lab

Replay one scripted conversation with full history, a sliding window, and compaction, then compare tokens, cache hits, cost, and what the model still remembers.

Intermediate22 min readUpdated October 19, 2026

On this page· What you will build

Why this matters

Multi-turn chat and memory describes the ways to keep a conversation affordable: send everything, keep a window, or summarize. Which one is right depends on numbers you only get by measuring: how many tokens each sends, how much of that the prompt cache absorbs, and whether the model still knows the user's name on turn 30. This tutorial builds a lab that replays the same conversation through each strategy and reports exactly that. Along the way it shows a rule of current Claude models that catches many hand-written memory schemes: you cannot edit earlier turns and keep replaying their thinking blocks.

What you will build#

A command-line lab that replays a 30-turn trip-planning conversation, with four facts planted in the first turns, through four strategies:

Strategy	What it sends each turn
Full history	Everything so far. Append-only.
Window of 4	The last four exchanges, with thinking blocks removed once older turns are dropped.
Window of 4 (naive)	The same window, but replaying the thinking blocks. Kept on purpose, to show what goes wrong.
Compaction at 2,000	Everything, until the prompt passes 2,000 tokens; then one summary replaces it all.

After the conversation it asks four recall questions ("What is my name?") and reports prompt tokens, the share read from cache, cost, and recall per strategy.

Hover or tap a component to see what it does.

Conversation memory lab: one scripted conversation replayed with full history, a sliding window, and compaction, comparing tokens, cache hits, cost, and recall.

Show text description

Every strategy sees the same conversation; only the choice of what to resend changes.

Script: A trip-planning chat with four facts planted early, then recall questions at the end.
Memory strategy: Full history, a sliding window (with thinking stripped), or simple compaction.
Claude: With --check, the API rejects any request whose history was edited without stripping thinking.
Summarizer: Only for compaction: turns the whole history into one summary when the prompt passes a limit.
Report: Per strategy: prompt tokens, share read from cache, cost, and recall of the planted facts.

You need Node.js 22.18 or later and an Anthropic API key. A full run makes about 140 small requests on claude-opus-5-5 and typically costs between one and two dollars; --turns 15 halves it.

1. Thinking blocks are tied to the conversation that produced them#

Before the code, the rule that shapes it. On current Claude models, including claude-opus-5-5, every thinking block carries a signature over the conversation that produced it: the system prompt, the tools, and every message before it. When you send history back, the API checks that this prefix is unchanged. For accounts created on or after 31 August 2026 the check is enforced, and a request that replays a thinking block under an edited history is rejected with a 400; older accounts can opt into the same check, which is what this lab's --check flag does.

Appending is always fine. These edits are not, while thinking blocks from after the edit are still replayed:

Removing turns from the start of the history (a sliding window).
Summarizing older turns while keeping recent turns verbatim, with their thinking.
Changing the system prompt or the tool list mid-conversation.

The fixes are simple once you know them: after an edit, strip the thinking blocks from the turns you keep, or start fresh from a single summary and replay nothing else. Both strategies below do exactly that, and the naive window shows the failure.

2. Set up the project#

mkdir memory-lab && cd memory-lab
npm init -y
npm pkg set type=module
npm install @anthropic-ai/sdk
npm install -D typescript @types/node
mkdir src test
npm pkg set scripts.lab="node --env-file=.env src/lab.ts" scripts.test="node --test" scripts.typecheck="tsc"

Copy the tsconfig.json from Foundations Tutorial 1, put ANTHROPIC_API_KEY=... in .env, and add the client module:

src/anthropic.ts

import Anthropic from "@anthropic-ai/sdk";
 
export const anthropic = new Anthropic({
  timeout: 60_000, // milliseconds
  maxRetries: 2, // retries 408, 409, 429, 5xx, and connection errors with backoff
});
 
export const MODEL = "claude-opus-5-5";

3. The script#

src/script.ts

// A scripted trip-planning conversation. Four facts are planted early; the rest is ordinary chat
// that pushes them further and further back.
export const SCRIPT: string[] = [
  "Hi! My name is Ada and I'm planning a week in Lisbon.",
  "My budget is 4000 euros for everything except flights.",
  "One important thing: I'm allergic to peanuts.",
  "My flight lands on 14 March at 9:00.",
  "Which neighbourhood should I stay in for a first visit?",
  "Is it easy to get there from the airport by metro?",
  "What should I see on my first afternoon?",
  "Can you suggest a good viewpoint for sunset?",
  "How many days should I keep for Sintra?",
  "Is it better to take the train or a tour to Sintra?",
  "What about Cascais, is it worth a half day?",
  "Recommend three museums, please.",
  "Which of those is best on a rainy day?",
  "Are there any food markets I shouldn't miss?",
  "What is a typical Portuguese breakfast?",
  "Where can I hear fado without it being a tourist trap?",
  "Do I need to book restaurants in advance?",
  "How much should I tip?",
  "Is Lisbon walkable or should I buy a transport card?",
  "What shoes should I pack for the hills?",
  "Any day trips besides Sintra and Cascais?",
  "What is the weather like in March?",
  "Is it safe to walk around at night?",
  "Which areas are best for nightlife?",
  "Can you suggest a relaxed last day?",
  "What souvenirs are worth buying?",
  "How early should I get to the airport for my return?",
  "Can you summarize the plan so far in five bullet points?",
  "What did we decide about Sintra?",
  "Thanks, this is really helpful!",
];
 
// Questions about the planted facts, asked once the conversation is over.
export const RECALL: { question: string; expect: RegExp }[] = [
  { question: "Quick check: what is my name?", expect: /\bAda\b/ },
  { question: "And what is my budget again?", expect: /4[ ,.]?000/ },
  { question: "What am I allergic to?", expect: /peanut/i },
  { question: "When does my flight land?", expect: /14 March|March 14|9:00/i },
];

The recall checks are regular expressions, not a model judging another model: cheap, deterministic, and good enough for facts with one right answer.

4. The strategies#

Each strategy decides what messages array to send for a new user turn, and how to store the exchange afterwards. The lab uses the beta message types because the --check option lives in the beta namespace.

src/memory.ts

import type Anthropic from "@anthropic-ai/sdk";
 
export type Message = Anthropic.Beta.BetaMessageParam;
type Block = Anthropic.Beta.BetaContentBlock;
 
/** A way of deciding what the model sees on each turn. */
export interface Memory {
  name: string;
  /** The full `messages` array to send for this user turn. */
  messagesFor(userText: string): Message[];
  /** Store a completed exchange, with the assistant's content blocks exactly as received. */
  remember(userText: string, assistant: Block[]): void;
  /** Called after each response with the size of the prompt it was given. */
  afterTurn?(promptTokens: number): Promise<void>;
}
 
const isThinking = (b: { type: string }) => b.type === "thinking" || b.type === "redacted_thinking";
 
/** Remove thinking blocks from assistant turns, keeping text and tool calls. */
export function stripThinking(messages: Message[]): Message[] {
  return messages.map((m) =>
    m.role === "assistant" && typeof m.content !== "string" ? { ...m, content: m.content.filter((b) => !isThinking(b)) } : m,
  );
}
 
/** Send everything, every time. Append-only, so thinking blocks and the prompt cache stay valid. */
export function fullHistory(): Memory {
  const messages: Message[] = [];
  return {
    name: "full history",
    messagesFor: (userText) => [...messages, { role: "user", content: userText }],
    remember(userText, assistant) {
      messages.push({ role: "user", content: userText }, { role: "assistant", content: assistant });
    },
  };
}
 
/**
 * Send only the last `keep` exchanges. Dropping early turns changes the conversation that every
 * kept thinking block was produced in, so they must be stripped; `naive` forgets to, on purpose.
 */
export function slidingWindow(keep: number, { naive = false } = {}): Memory {
  const messages: Message[] = [];
  return {
    name: naive ? `window of ${keep} (naive)` : `window of ${keep}`,
    messagesFor(userText) {
      const kept = messages.slice(-keep * 2); // whole exchanges, so it still starts with a user turn
      const trimmed = kept.length < messages.length;
      return [...(trimmed && !naive ? stripThinking(kept) : kept), { role: "user", content: userText }];
    },
    remember(userText, assistant) {
      messages.push({ role: "user", content: userText }, { role: "assistant", content: assistant });
    },
  };
}
 
/**
 * Simple compaction: when the prompt grows past `maxPromptTokens`, summarize everything into one
 * message and start over from it. Nothing from before the summary is replayed, so nothing is edited.
 */
export function simpleCompaction(maxPromptTokens: number, summarize: (messages: Message[]) => Promise<string>): Memory {
  let summary: string | null = null;
  let messages: Message[] = [];
  const context = (): Message[] =>
    summary ? [{ role: "user", content: `<summary_of_earlier_conversation>\n${summary}\n</summary_of_earlier_conversation>` }, ...messages] : messages;
 
  return {
    name: `compaction at ${maxPromptTokens}`,
    messagesFor: (userText) => [...context(), { role: "user", content: userText }],
    remember(userText, assistant) {
      messages.push({ role: "user", content: userText }, { role: "assistant", content: assistant });
    },
    async afterTurn(promptTokens) {
      if (promptTokens <= maxPromptTokens) return;
      summary = await summarize(context()); // the old summary is part of what gets summarized
      messages = [];
    },
  };
}

Two shapes worth noticing:

The window slices whole exchanges (keep * 2 messages), so the array always starts with a user turn.
Compaction is "summary plus what came after", never "summary plus old turns". After a compaction, the summary message is the first message and stays fixed until the next compaction, so the history is append-only again. The summary is a user message followed directly by the next user message; the API combines consecutive messages from the same role into one turn.

5. The lab#

src/lab.ts

import { parseArgs } from "node:util";
import type Anthropic from "@anthropic-ai/sdk";
import { anthropic, MODEL } from "./anthropic.ts";
import { fullHistory, type Memory, type Message, simpleCompaction, slidingWindow } from "./memory.ts";
import { RECALL, SCRIPT } from "./script.ts";
 
const { values } = parseArgs({
  options: {
    turns: { type: "string", default: String(SCRIPT.length) },
    check: { type: "boolean", default: false }, // make the API reject any edited history
  },
});
const turns = SCRIPT.slice(0, Number(values.turns));
 
const SYSTEM = "You are a friendly travel-planning assistant. Keep every answer under 80 words.";
const PRICE = { input: 4, cacheWrite: 5, cacheRead: 0.2, output: 20 }; // USD per million tokens, claude-opus-5-5
 
interface Totals {
  input: number;
  cacheRead: number;
  usd: number;
  errors: string[];
}
let current: Totals;
 
const textOf = (content: Anthropic.Beta.BetaContentBlock[]) => content.flatMap((b) => (b.type === "text" ? [b.text] : [])).join("");
const promptTokens = (u: Anthropic.Beta.BetaUsage) => u.input_tokens + (u.cache_creation_input_tokens ?? 0) + (u.cache_read_input_tokens ?? 0);
 
function account(u: Anthropic.Beta.BetaUsage) {
  current.input += promptTokens(u);
  current.cacheRead += u.cache_read_input_tokens ?? 0;
  current.usd +=
    (u.input_tokens * PRICE.input +
      (u.cache_creation_input_tokens ?? 0) * PRICE.cacheWrite +
      (u.cache_read_input_tokens ?? 0) * PRICE.cacheRead +
      u.output_tokens * PRICE.output) /
    1_000_000;
}
 
async function send(messages: Message[], system = SYSTEM) {
  const response = await anthropic.beta.messages.create({
    model: MODEL,
    max_tokens: 1024,
    output_config: { effort: "low" },
    system,
    cache_control: { type: "ephemeral" }, // cache the whole prefix; any edit to history shows up as a miss
    messages,
    // --check opts this request into the history check: an edited history becomes a 400.
    ...(values.check
      ? { betas: ["thinking-binding-controls-2026-08-01"], thinking: { type: "adaptive", block_binding: { prefix_mismatch_behavior: "error" } } }
      : {}),
  });
  account(response.usage);
  return response;
}
 
// The summarizer sees a plain transcript, so no thinking blocks travel into the new context.
async function summarize(messages: Message[]): Promise<string> {
  const transcript = messages
    .map((m) => `${m.role}: ${typeof m.content === "string" ? m.content : m.content.flatMap((b) => (b.type === "text" ? [b.text] : [])).join(" ")}`)
    .join("\n");
  const response = await send(
    [{ role: "user", content: transcript }],
    "Summarize this conversation for your own future reference. Keep names, numbers, dates, preferences, decisions, and open questions. Write under 150 words.",
  );
  return textOf(response.content);
}
 
const strategies: Memory[] = [fullHistory(), slidingWindow(4), slidingWindow(4, { naive: true }), simpleCompaction(2000, summarize)];
 
console.log(`Replaying ${turns.length} turns per strategy${values.check ? ", with the history check on" : ""}...\n`);
const rows: { name: string; totals: Totals; recalled: number }[] = [];
 
for (const memory of strategies) {
  current = { input: 0, cacheRead: 0, usd: 0, errors: [] };
  let recalled = 0;
  try {
    for (const userText of turns) {
      const response = await send(memory.messagesFor(userText));
      memory.remember(userText, response.content);
      await memory.afterTurn?.(promptTokens(response.usage));
    }
    // Ask about the planted facts without storing the questions, so every strategy gets the same test.
    for (const { question, expect } of RECALL) {
      const response = await send(memory.messagesFor(question));
      if (expect.test(textOf(response.content))) recalled++;
    }
  } catch (error) {
    current.errors.push(error instanceof Error ? error.message.slice(0, 160) : String(error));
  }
  rows.push({ name: memory.name, totals: current, recalled });
  process.stdout.write(".");
}
console.log("\n");
 
console.log(`${"strategy".padEnd(26)}${"prompt tokens".padStart(14)}${"cached".padStart(9)}${"cost".padStart(10)}${"recall".padStart(9)}`);
for (const { name, totals, recalled } of rows) {
  const cached = totals.input ? `${Math.round((totals.cacheRead / totals.input) * 100)}%` : "-";
  console.log(
    `${name.padEnd(26)}${totals.input.toLocaleString("en-US").padStart(14)}${cached.padStart(9)}` +
      `${`$${totals.usd.toFixed(4)}`.padStart(10)}${`${recalled}/${RECALL.length}`.padStart(9)}`,
  );
  for (const e of totals.errors) console.log(`  ✗ ${e}`);
}

cache_control at the top level asks the API to cache the whole prefix of each request, so the "cached" column shows how much of each strategy's input was served from the cache instead of being processed again. The summarizer's own calls are charged to the compaction strategy, because they are part of its cost.

6. Run it#

npm run lab

report against the local mock (illustrative numbers)

strategy                   prompt tokens   cached      cost   recall
full history                      42,872      90%   $0.0590      4/4
window of 4                        6,782       0%   $0.0586      0/4
window of 4 (naive)               10,748       0%   $0.0745      0/4
compaction at 2000                31,674      87%   $0.0555      4/4

Read it row by row:

Full history sends by far the most tokens, but nearly all of them are cache reads at a fraction of the input price, and it remembers everything.
The window sends six times fewer tokens yet costs about the same: each turn drops the oldest exchange, so the prefix changes every time and nothing is ever read from the cache. It also forgets every planted fact.
The naive window sends more than the correct one, because it replays thinking blocks the model cannot use, and it is a 400 waiting to happen on an enforced account.
Compaction remembers all four facts through its summary and keeps most of its input cached, because between compactions its history is append-only again.

Your real numbers will differ in size, but not in direction. The lesson generalizes: fewer tokens is not the same as cheaper once caching is involved, and anything that edits history pays twice, in cache misses and in lost context.

Now prove the thinking rule:

npm run lab -- --check --turns 10

excerpt

window of 4 (naive)                  975       0%   $0.0086      0/4
  ✗ 400 {"type":"error","error":{"type":"invalid_request_error","message":"messages.1.content.0: Invalid `signature` in `thinking` block. The block is bound to a di

With the check on, the naive window is rejected the first time it replays a thinking block from a trimmed history, while the window that strips thinking and the compaction strategy run normally. Keep --check on in CI for any code that builds messages arrays itself: it turns a silent quality loss into a failing test.

7. Test the strategies#

test/memory.test.ts

import assert from "node:assert/strict";
import { test } from "node:test";
import type Anthropic from "@anthropic-ai/sdk";
import { fullHistory, simpleCompaction, slidingWindow, stripThinking } from "../src/memory.ts";
 
const reply = (text: string) =>
  [
    { type: "thinking", thinking: "", signature: `sig-${text}` },
    { type: "text", text, citations: null },
  ] as Anthropic.Beta.BetaContentBlock[];
 
const types = (m: Anthropic.Beta.BetaMessageParam) => (typeof m.content === "string" ? ["string"] : m.content.map((b) => b.type));
 
test("full history appends and never edits", () => {
  const memory = fullHistory();
  memory.remember("one", reply("a"));
  const first = memory.messagesFor("two");
  memory.remember("two", reply("b"));
  const second = memory.messagesFor("three");
  assert.deepEqual(second.slice(0, first.length - 1), first.slice(0, -1)); // the earlier prefix is unchanged
  assert.deepEqual(types(second[1]!), ["thinking", "text"]);
});
 
test("a sliding window keeps whole exchanges and starts with a user turn", () => {
  const memory = slidingWindow(2);
  for (const n of ["1", "2", "3", "4"]) memory.remember(n, reply(n));
  const messages = memory.messagesFor("5");
  assert.equal(messages.length, 5); // 2 exchanges + the new turn
  assert.equal(messages[0]!.role, "user");
  assert.equal(messages[0]!.content, "3");
});
 
test("once it trims, the window strips thinking; the naive one does not", () => {
  const safe = slidingWindow(1);
  const naive = slidingWindow(1, { naive: true });
  for (const memory of [safe, naive]) for (const n of ["1", "2"]) memory.remember(n, reply(n));
  assert.deepEqual(types(safe.messagesFor("3")[1]!), ["text"]);
  assert.deepEqual(types(naive.messagesFor("3")[1]!), ["thinking", "text"]);
});
 
test("stripThinking keeps text and leaves user turns alone", () => {
  const [user, assistant] = stripThinking([{ role: "user", content: "hi" }, { role: "assistant", content: reply("x") }]);
  assert.equal(user!.content, "hi");
  assert.deepEqual(types(assistant!), ["text"]);
});
 
test("compaction replaces history with one summary once the prompt is too big", async () => {
  const memory = simpleCompaction(100, async (messages) => `summary of ${messages.length} messages`);
  memory.remember("1", reply("a"));
  await memory.afterTurn!(50); // under the limit: nothing happens
  assert.equal(memory.messagesFor("2").length, 3);
  await memory.afterTurn!(150);
  const messages = memory.messagesFor("2");
  assert.equal(messages.length, 2);
  assert.match(String(messages[0]!.content), /summary of 2 messages/);
});

npm test   # 5 passing, no API calls

Common mistakes#

Trimming history and replaying its thinking blocks. On enforced accounts it is a 400; elsewhere the model receives reasoning that no longer matches the conversation. Strip thinking from retained turns after any edit.
Keep-tail summaries. "Summary plus the last few turns verbatim" replays thinking produced under the full history. Summarize everything, or strip the kept turns' thinking.
Judging a strategy by tokens sent. With prompt caching, an append-only history can cost less than a much shorter one that changes every turn.
Summaries that drop the facts that matter. Tell the summarizer what to keep (names, numbers, decisions), and test recall the way this lab does.
Storing only the text of replies. Store the content blocks; thinking blocks must be replayed exactly as received when they are replayed at all.

Try it

Extend the lab

Add a fifth strategy: a window of 8. Where does it land between the window of 4 and full history on cost and recall?
Change the compaction limit to 1,000 and to 4,000. How do cost and recall move?
Add a fact in turn 20 ("Change of plan: my budget is now 5000 euros") and a recall question for it. Which strategies get the update right?
Replace the hand-written compaction with server-side compaction (compact-2026-01-12) using a low trigger, and compare its row with yours.

Check your understanding

Key takeaways

Measure memory strategies on the same conversation: tokens sent, cache share, cost, and recall.
Append-only histories keep both the prompt cache and thinking blocks valid; every edit costs a cache miss.
After trimming history, strip thinking from the retained turns; when compacting, start from one summary and replay nothing older.
Use the history check (--check here) in tests to catch code that edits history by accident.

Next steps#

Streaming to a React UI

Next article: stream model output into an interface.

Prompt caching and cost

The cache mechanics behind the "cached" column.

Finished this article? Track it to see your progress on the roadmap.