LLM Observability: Tracing, Metrics, and Logs with OpenTelemetry

Instrument an LLM app with OpenTelemetry in Next.js: GenAI spans with tokens, cost, cache hits, and time to first token, plus dashboards, alerts, and safe logging.

Advanced18 min readUpdated October 7, 2026

On this page· What a trace shows

Why this matters

A user says "the assistant was slow and gave a weird answer yesterday". Without observability, you cannot answer basic questions: which request was it, which model ran, what context did it see, which tool failed, how many tokens did it cost? LLM apps fail in new ways (slow first tokens, runaway agent loops, cache misses that triple the bill, refusals), and those failures are only visible if every request leaves a trace.

What a trace shows#

A trace is the record of one request, made of spans: timed operations nested inside each other, each with attributes. For an LLM feature, a single trace answers where the time went, what each model call cost, and what happened in which order:

Your server
Database & search
Model API
Tool call

Trace waterfall for one chat request: retrieval, two model calls, and a tool call. The model calls dominate latency, which is where caching, streaming, and smaller outputs pay off.

Show text description

One request to a RAG chat with a tool, as a tracing tool shows it. Each bar is a span; nesting shows which work happened inside which.

Spans in the trace
Span	Starts at	Duration
POST /api/chat	0 ms	2,860 ms
auth + rate limit	0 ms	14 ms
load conversation	14 ms	21 ms
retrieve: embed query	35 ms	96 ms
retrieve: hybrid search	131 ms	24 ms
retrieve: rerank	155 ms	138 ms
llm call 1 → tool_use	293 ms	940 ms
tool get_order	1,233 ms	88 ms
llm call 2 (streamed)	1,321 ms	1,515 ms
save messages	2,836 ms	24 ms

This trace already suggests improvements: the two model calls take about 86% of the time, so streaming the second one (it is), caching the shared prefix, and shortening outputs matter far more than optimizing the 24 ms database query.

The pipeline#

Hover or tap a component to see what it does.

LLM observability pipeline: the app emits OpenTelemetry spans with model, token, and cost attributes; an exporter batches and redacts them; a tracing backend powers dashboards, alerts, and new eval cases.

Show text description

Instrument once with an open standard, and you can change backends without touching application code.

Your app: Each request, model call, retrieval, and tool call is a span with attributes: model, tokens, cost, latency, stop reason.
Exporter: Batches spans off the request path, samples high-volume traffic, and redacts sensitive fields before they leave your system.
Tracing backend: Any OpenTelemetry-compatible backend, or an LLM-focused one. Stores traces for search and aggregates metrics.
Dashboards & alerts: p50 and p95 latency, cost per user and feature, error and refusal rates, cache hit ratio. Alert on changes.
Eval cases: Interesting and failed traces become new eval cases, closing the loop between production and development.

OpenTelemetry (OTel) is the open standard for traces, metrics, and logs. Instrument with it once, and you can send data to any compatible backend (an open-source stack, a cloud APM, or an LLM-focused observability tool) without changing application code. OTel also defines GenAI semantic conventions: standard attribute names such as gen_ai.request.model and gen_ai.usage.input_tokens, so tools can understand your LLM spans. These conventions are still marked as in development, so expect small changes.

Set up OpenTelemetry in Next.js#

Next.js calls a register function from instrumentation.ts at startup. Load the Node.js SDK only in the Node.js runtime:

instrumentation.ts

export async function register() {
  if (process.env.NEXT_RUNTIME === "nodejs") {
    await import("./instrumentation.node.ts");
  }
}

instrumentation.node.ts

import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-http";
import { resourceFromAttributes } from "@opentelemetry/resources";
import { NodeSDK } from "@opentelemetry/sdk-node";
import { BatchSpanProcessor, ParentBasedSampler, TraceIdRatioBasedSampler } from "@opentelemetry/sdk-trace-node";
import { ATTR_SERVICE_NAME } from "@opentelemetry/semantic-conventions";
 
const sdk = new NodeSDK({
  resource: resourceFromAttributes({ [ATTR_SERVICE_NAME]: "support-app" }),
  // Batch spans off the request path. The endpoint comes from OTEL_EXPORTER_OTLP_ENDPOINT.
  spanProcessors: [new BatchSpanProcessor(new OTLPTraceExporter())],
  // Keep every trace in development; sample 20% of traces in production.
  sampler: new ParentBasedSampler({
    root: new TraceIdRatioBasedSampler(process.env.NODE_ENV === "production" ? 0.2 : 1),
  }),
});
sdk.start();

Next.js already creates spans for incoming requests, rendering, route handlers, and fetch. Your job is to add spans for the work it cannot see: model calls, retrieval, and tools.

Trace every model call#

Wrap the SDK in one function that every feature uses. It creates a span with GenAI attributes, records usage, cost, cache behavior, and errors, and returns the message unchanged:

lib/traced-llm.ts

import Anthropic from "@anthropic-ai/sdk";
import { SpanKind, SpanStatusCode, trace } from "@opentelemetry/api";
 
const client = new Anthropic();
const tracer = trace.getTracer("support-app");
 
// USD per million tokens at the time of writing; keep in one place.
const PRICES: Record<string, { input: number; output: number; cacheRead: number; cacheWrite: number }> = {
  "claude-opus-5-5": { input: 4, output: 20, cacheRead: 0.2, cacheWrite: 5 },
  "claude-sonnet-5-5": { input: 2, output: 10, cacheRead: 0.2, cacheWrite: 2.5 },
  "claude-haiku-4-5": { input: 1, output: 5, cacheRead: 0.1, cacheWrite: 1.25 },
};
 
export function costUSD(model: string, u: Anthropic.Usage): number {
  const p = PRICES[model];
  if (!p) return 0;
  const read = u.cache_read_input_tokens ?? 0;
  const write = u.cache_creation_input_tokens ?? 0;
  return (u.input_tokens * p.input + read * p.cacheRead + write * p.cacheWrite + u.output_tokens * p.output) / 1_000_000;
}
 
export interface CallContext {
  feature: string; // "support-chat", "ticket-triage": the unit you budget and alert on
  userId?: string; // an internal ID, never an email
}
 
export async function createMessage(params: Anthropic.MessageCreateParamsNonStreaming, ctx: CallContext): Promise<Anthropic.Message> {
  return tracer.startActiveSpan(`chat ${params.model}`, { kind: SpanKind.CLIENT }, async (span) => {
    span.setAttributes({
      "gen_ai.operation.name": "chat",
      "gen_ai.provider.name": "anthropic",
      "gen_ai.request.model": params.model,
      "gen_ai.request.max_tokens": params.max_tokens,
      "app.feature": ctx.feature,
      ...(ctx.userId ? { "app.user_id": ctx.userId } : {}),
      ...(params.tools ? { "app.tool_count": params.tools.length } : {}),
    });
    try {
      const message = await client.messages.create(params);
      const u = message.usage;
      span.setAttributes({
        "gen_ai.response.id": message.id,
        "gen_ai.response.model": message.model,
        "gen_ai.response.finish_reasons": [message.stop_reason ?? "unknown"],
        // Convention: input_tokens includes cached tokens; cache reads and writes are also reported separately.
        "gen_ai.usage.input_tokens": u.input_tokens + (u.cache_read_input_tokens ?? 0) + (u.cache_creation_input_tokens ?? 0),
        "gen_ai.usage.cache_read.input_tokens": u.cache_read_input_tokens ?? 0,
        "gen_ai.usage.cache_creation.input_tokens": u.cache_creation_input_tokens ?? 0,
        "gen_ai.usage.output_tokens": u.output_tokens,
        "app.cost_usd": costUSD(params.model, u),
        "app.request_id": message._request_id ?? "",
      });
      return message;
    } catch (error) {
      span.recordException(error as Error);
      span.setStatus({ code: SpanStatusCode.ERROR, message: error instanceof Error ? error.message : String(error) });
      if (error instanceof Anthropic.APIError) span.setAttributes({ "error.type": String(error.status ?? "unknown"), "app.request_id": error.requestID ?? "" });
      throw error;
    } finally {
      span.end();
    }
  });
}

The request ID (_request_id on responses, requestID on errors) is what you give to support when you need help with a specific call. Recording it on the span makes it one search away.

Streaming: time to first token#

For streamed responses, the number users feel is time to first token (TTFT), not total duration. Record it as an attribute when the first text arrives:

lib/traced-stream.ts

import Anthropic from "@anthropic-ai/sdk";
import { SpanKind, SpanStatusCode, trace } from "@opentelemetry/api";
import { costUSD, type CallContext } from "./traced-llm.ts";
 
const client = new Anthropic();
const tracer = trace.getTracer("support-app");
 
/** Streams text deltas to `onText` and returns the final message, with a span covering the whole stream. */
export async function streamMessage(
  params: Anthropic.MessageCreateParamsNonStreaming,
  ctx: CallContext,
  onText: (text: string) => void,
): Promise<Anthropic.Message> {
  return tracer.startActiveSpan(`chat ${params.model}`, { kind: SpanKind.CLIENT }, async (span) => {
    span.setAttributes({
      "gen_ai.operation.name": "chat",
      "gen_ai.provider.name": "anthropic",
      "gen_ai.request.model": params.model,
      "gen_ai.request.stream": true,
      "app.feature": ctx.feature,
    });
    const started = performance.now();
    let firstToken = true;
    try {
      const stream = client.messages.stream(params);
      stream.on("text", (text) => {
        if (firstToken) {
          firstToken = false;
          span.setAttribute("gen_ai.response.time_to_first_chunk", (performance.now() - started) / 1000); // seconds
        }
        onText(text);
      });
      const message = await stream.finalMessage();
      span.setAttributes({
        "gen_ai.response.finish_reasons": [message.stop_reason ?? "unknown"],
        "gen_ai.usage.output_tokens": message.usage.output_tokens,
        "app.cost_usd": costUSD(params.model, message.usage),
      });
      return message;
    } catch (error) {
      span.recordException(error as Error);
      span.setStatus({ code: SpanStatusCode.ERROR });
      throw error;
    } finally {
      span.end();
    }
  });
}

Spans for retrieval and tools#

Any function can become a span with a small helper. Wrap retrieval, reranking, and every tool execution, so the trace shows the whole request:

lib/span.ts

import { SpanStatusCode, trace, type Attributes } from "@opentelemetry/api";
 
const tracer = trace.getTracer("support-app");
 
export function withSpan<T>(name: string, attributes: Attributes, fn: () => Promise<T>): Promise<T> {
  return tracer.startActiveSpan(name, { attributes }, async (span) => {
    try {
      return await fn();
    } catch (error) {
      span.recordException(error as Error);
      span.setStatus({ code: SpanStatusCode.ERROR });
      throw error;
    } finally {
      span.end();
    }
  });
}
 
// Usage:
// const hits = await withSpan("retrieve hybrid_search", { "app.top_k": 20 }, () => search(question));
// const result = await withSpan("execute_tool get_order", { "gen_ai.tool.name": "get_order" }, () => runTool(...));

What to watch#

Traces explain individual requests; metrics across all requests tell you when to look. Most tracing backends compute these from span attributes:

Metric	Why	Alert when
p50 / p95 latency, TTFT	What users feel	p95 TTFT doubles
Error rate by type (429, 529, 5xx, timeout)	Capacity and provider issues	Above 2% for 5 minutes
Cost per feature and per user per day	Budget and abuse	A feature's daily cost jumps 50%
Cache read ratio	Caching regressions	Drops after a deploy
`stop_reason` mix	`max_tokens` means truncated answers; `refusal` means unexpected refusals	Any rise in either
Agent turns and tool errors per run	Stuck or looping agents	p95 turns near the limit

Logs, privacy, and prompts#

Prompts and responses are the most useful debugging data and the most sensitive. A safe default:

Spans carry metadata, not content: model, tokens, cost, IDs, stop reasons, feature names.
Content goes to a separate, access-controlled store if you need it, with retention limits (say, 30 days), and with personal data redacted before storage.
Never log secrets: API keys, auth headers, or tool results containing credentials.
Use internal user IDs, not emails or names, in attributes.
Tell users in your privacy policy what you keep and why.

Even metadata-only traces answer most questions; when you need content to debug, a request ID lets you fetch exactly one conversation with proper access controls.

From traces to better quality#

Observability and evals form one loop. Tag traces with user feedback (thumbs down, "regenerate" clicks, escalations to a human), review a sample each week, and turn representative failures into eval cases. Your eval set then grows with exactly the problems real users hit.

Common mistakes#

Logging full prompts everywhere. It leaks personal data into every log system you have. Separate content from metadata.
No cost attribute. Without cost per span, you discover expensive features from the invoice.
Only measuring total latency. For streamed UIs, TTFT is the number that matters.
Synchronous exporting. Export in batches off the request path, or tracing itself slows requests down.
Sampling away the errors. If you sample, keep all error traces (many backends support tail-based sampling for this).

Try it

Trace a request end to end

Run a local OTel-compatible backend (for example, Jaeger with OTLP enabled) and set OTEL_EXPORTER_OTLP_ENDPOINT.
Add instrumentation.ts, use createMessage in one route, and wrap retrieval with withSpan.
Send three requests and find them in the backend. Check the token, cost, and cache attributes, and compare the first request (cache write) with later ones (cache reads).

Check your understanding

Key takeaways

A trace shows where time and money went in one request; metrics across traces tell you when to look.
Set up OpenTelemetry in instrumentation.ts, and wrap model calls, retrieval, and tools in spans with GenAI attributes.
Record tokens, cache reads, cost, stop reason, request ID, and TTFT on every model span.
Keep content out of spans by default, sample carefully, and turn bad traces into eval cases.

Next steps#

Reliability

Use what you observe to handle timeouts, rate limits, and outages.

Cost engineering and model routing

Turn cost attributes into budgets and routing decisions.

Finished this article? Track it to see your progress on the roadmap.