How LLMs Work: Tokens, Context Windows, Statelessness, and Reasoning

Tokens: the unit of everything#

A large language model does not see characters or words. It sees tokens: chunks of text from a fixed vocabulary, often a common word, part of a longer word, a space plus a word, or a punctuation mark. In English a token averages roughly three to four characters; code, non-English text, and unusual strings use more tokens per character.

Tokens matter because everything is measured in them: price (per million input and output tokens), the context window, output limits, rate limits, and latency (each output token takes time to generate). Never estimate with another provider's tokenizer; count with the API you will call.

import Anthropic from "@anthropic-ai/sdk";
 
const client = new Anthropic();
 
const samples = [
  "Hello, world!",
  "Tokenization splits text into subword units.",
  "const total = items.reduce((sum, item) => sum + item.price, 0);",
  "Bonjour, comment ça va aujourd'hui ?",
];
 
for (const text of samples) {
  const { input_tokens } = await client.messages.countTokens({
    model: "claude-opus-5-5",
    messages: [{ role: "user", content: text }],
  });
  console.log(`${String(input_tokens).padStart(3)} tokens · ${String(text.length).padStart(3)} chars · ${text}`);
}

node --env-file=.env.local count-tokens.ts

The count includes a few tokens of message framing, so tiny strings look expensive. Run it on a real prompt from your app and you have the most important number for estimating cost.

Generation is a loop of next-token predictions#

Given all the tokens so far, the model computes a probability for every token in its vocabulary. One token is selected and appended, and the whole thing repeats. Step through it:

Four practical consequences follow:

• Output is streamed naturally. Each appended token can be sent to the user immediately, which is why streaming is the default for chat interfaces.
• Output length drives latency. A 1,000-token answer takes much longer than a 100-token one. Asking for concise answers is a performance optimization.
• Generation stops for a reason. The response's stop_reason tells you which: end_turn (finished), max_tokens (your limit cut it off), tool_use (it wants a tool), or refusal. Always check it.
• Answers vary between runs. Selection is not always the single most likely token, so the same prompt can produce different wording. Design for this: validate structured output and evaluate on many samples, not one.

The context window#

The context window is the maximum number of tokens one request can involve: everything you send plus everything the model writes back. Current Claude models accept up to 1 million input tokens (Claude Haiku 4.5: 200,000) and can write up to 128,000 output tokens per response. Those limits are generous; the real constraint is that every input token costs money and time on every request.

The model knows nothing about your product, your user, or the conversation except what is in this window. Deciding what goes in, and in what order, is context engineering, and it is most of the work in the LLM App Engineering section.

The API is stateless#

The model does not remember previous requests. A chat "remembers" because your app stores the conversation and sends the whole history again on every turn.

You can prove it in a few lines:

import Anthropic from "@anthropic-ai/sdk";
 
const client = new Anthropic();
 
async function ask(messages: Anthropic.MessageParam[]) {
  return client.messages.create({
    model: "claude-opus-5-5",
    max_tokens: 1024,
    output_config: { effort: "low" },
    messages,
  });
}
 
const text = (message: Anthropic.Message) =>
  message.content.flatMap((block) => (block.type === "text" ? [block.text] : [])).join("");
 
const intro: Anthropic.MessageParam = { role: "user", content: "Hi, my name is Ada." };
const first = await ask([intro]);
console.log("1:", text(first));
 
// A new request without history: the model has never heard of Ada.
const forgetful = await ask([{ role: "user", content: "What is my name?" }]);
console.log("2 (no history):", text(forgetful));
 
// The same question with the history resent. Append the assistant turn exactly as received.
const remembered = await ask([intro, { role: "assistant", content: first.content }, { role: "user", content: "What is my name?" }]);
console.log("3 (with history):", text(remembered));
console.log("input tokens grew from", first.usage.input_tokens, "to", remembered.usage.input_tokens);

Statelessness has three consequences you will design around:

1. Cost grows with conversation length, because each turn re-sends all earlier turns. Long chats need trimming or summarizing, and prompt caching makes the repeated prefix cheap.
2. You own memory. What to keep, where to store it, and whose data it is are application decisions (covered in multi-turn chat and agent memory).
3. Replay content blocks unchanged. Responses can contain reasoning and tool blocks that must be sent back exactly as received. Store and resend response.content, not just the text.

Reasoning: thinking before answering#

Current models can think before answering: they generate internal reasoning tokens, then the final answer. With adaptive thinking the model decides how much reasoning a request needs; you steer the overall depth with effort (low, medium, high, xhigh, max). Higher effort usually improves hard tasks (multi-step problems, code, agentic tool use) and costs more output tokens and time; easy tasks such as classification or short chat replies do well at low.

import Anthropic from "@anthropic-ai/sdk";
 
const client = new Anthropic();
 
const response = await client.messages.create({
  model: "claude-opus-5-5",
  max_tokens: 16000,
  thinking: { type: "adaptive", display: "summarized" }, // show a readable summary of the reasoning
  output_config: { effort: "high" },
  messages: [
    {
      role: "user",
      content:
        "A bat and a ball cost $1.10 in total. The bat costs $1.00 more than the ball. " +
        "How much does the ball cost? Answer in one sentence.",
    },
  ],
});
 
for (const block of response.content) {
  if (block.type === "thinking") console.log("[reasoning summary]", block.thinking);
  if (block.type === "text") console.log("[answer]", block.text);
}
console.log(response.usage); // reasoning is billed as output tokens

Reasoning tokens count as output tokens. By default the raw reasoning is not shown (the thinking field comes back empty); display: "summarized" returns a readable summary, which is useful while developing and for showing progress in a UI.

What models know, and what they make up#

A model's built-in knowledge stops at its training cutoff and never includes your private data. When it lacks information it can still produce fluent, confident text, a hallucination. The engineering answer is not a better prompt alone: put the facts in the context (retrieval, tool results), ask for citations, allow "I don't know", and measure with evals.

Common mistakes#

• Estimating tokens by word count or another vendor's tokenizer. Use the token counting endpoint for the model you call.
• Assuming the model remembers. Every request must contain everything the model needs, including prior turns.
• Saving only the text of responses. Reasoning and tool blocks must be replayed unchanged; store the content blocks.
• Ignoring stop_reason. A response cut off at max_tokens looks like a complete answer to code that only reads the text.
• Maximum effort everywhere. It multiplies cost and latency on tasks that do not need it. Start low for simple tasks and raise effort only where evals show a gain.

Next steps#