Distributed Tracing in LLM Pipelines with OpenTelemetry

When debugging standard REST microservices, traditional APM tools isolate errors quickly by inspecting HTTP status codes and database query spans. In multi-turn AI agent architectures, request failures rarely look like clean HTTP 500 errors. Instead, systems fail through silent retrieval misses, infinite tool loops, prompt token bloat, and cascading downstream latency spikes.

A user request that normally takes 800ms can suddenly jump to 6.4s because an intermediate planning step triggered three redundant vector queries and two retries against an external database. Without end-to-end distributed tracing, identifying which node caused the latency spike requires manually parsing unstructured application logs.

To achieve clear observability into agent execution, we instrument pipelines with OpenTelemetry (OTel) using standardized GenAI semantic conventions and distributed context propagation.

OpenTelemetry GenAI semantic conventions

OpenTelemetry defines standard attribute keys for AI and LLM operations under the gen_ai namespace. Adhering to these conventions ensures that telemetry backends (like Jaeger, SigNoz, and Arize Phoenix) can calculate token burn rates, latency percentiles, and cost attribution automatically.

A well-instrumented agent request builds a structured span hierarchy.

1. Root Server Span (POST /api/agent/chat): Captures the inbound HTTP request lifecycle, client metadata, and total turnaround time.
2. Orchestrator Span (agent.orchestrator): Tracks loop iterations, graph state updates, and routing decisions.
3. Retrieval Span (retrieval.hybrid_search): Records vector database queries, top-k candidate counts, and cross-encoder scoring latency.
4. LLM Inference Span (gen_ai.chat): Measures model response time, temperature settings, prompt token counts, and completion token counts.
5. Tool Execution Span (tool.database_query): Records external system mutations, SQL queries, and tool schema validation status.

Key GenAI span attributes

When creating inference and agent spans, record these standardized attributes:

• gen_ai.system: The provider identifier (e.g., "openai", "anthropic", "ollama").
• gen_ai.request.model: The target model name (e.g., "gpt-4o", "claude-3-5-sonnet-20241022").
• gen_ai.request.temperature: Sampling temperature passed to the API.
• gen_ai.usage.prompt_tokens: Number of input tokens consumed by the prompt and history.
• gen_ai.usage.completion_tokens: Number of generated output tokens.
• gen_ai.response.finish_reasons: Array indicating why generation completed (e.g., ["stop"], ["tool_calls"]).

Propagating context across asynchronous queues

Production agents frequently decouple user-facing HTTP request handlers from long-running execution loops using task queues like Redis or Kafka. When an agent job moves onto a message broker, the worker drops the in-memory trace context unless you explicitly propagate it.

We solve this using the W3C Trace Context specification. The ingress service serializes the active trace into a traceparent header (format: 00-{trace_id}-{parent_span_id}-{trace_flags}) and attaches it to the queue payload. The background worker extracts this header before starting execution, preserving the end-to-end waterfall graph.

Implementing typed LLM instrumentation in TypeScript

Here is a modular TypeScript wrapper that instruments LLM calls and tool executions using the official @opentelemetry/api package.

import {
  trace,
  context,
  SpanStatusCode,
  type Tracer,
  type Span,
} from "@opentelemetry/api";
 
const tracer: Tracer = trace.getTracer("ai-agent-service", "1.0.0");
 
export interface LLMRequestOptions {
  model: string;
  temperature?: number;
  maxTokens?: number;
  messages: Array<{ role: string; content: string }>;
}
 
export interface LLMResponsePayload {
  content: string;
  finishReason: string;
  usage: {
    promptTokens: number;
    completionTokens: number;
    totalTokens: number;
  };
}
 
/**
 * Wraps an LLM API call inside an OpenTelemetry GenAI semantic span.
 */
export async function tracedLLMCall(
  options: LLMRequestOptions,
  executeCall: () => Promise<LLMResponsePayload>,
): Promise<LLMResponsePayload> {
  return tracer.startActiveSpan("gen_ai.chat", async (span: Span) => {
    // 1. Record input semantic attributes
    span.setAttributes({
      "gen_ai.system": "openai",
      "gen_ai.request.model": options.model,
      "gen_ai.request.temperature": options.temperature ?? 0.7,
      "gen_ai.request.max_tokens": options.maxTokens ?? 1024,
    });
 
    try {
      const startTime = performance.now();
      const response = await executeCall();
      const durationMs = Math.round(performance.now() - startTime);
 
      // 2. Record output attributes and token accounting
      span.setAttributes({
        "gen_ai.usage.prompt_tokens": response.usage.promptTokens,
        "gen_ai.usage.completion_tokens": response.usage.completionTokens,
        "gen_ai.response.finish_reasons": [response.finishReason],
        "llm.latency_ms": durationMs,
      });
 
      span.setStatus({ code: SpanStatusCode.OK });
      return response;
    } catch (error) {
      // 3. Record exception details without leaking unhandled state
      span.recordException(error instanceof Error ? error : new Error(String(error)));
      span.setStatus({
        code: SpanStatusCode.ERROR,
        message: error instanceof Error ? error.message : "Unknown LLM error",
      });
      throw error;
    } finally {
      span.end();
    }
  });
}
 
/**
 * Instruments an agent tool invocation as a nested child span.
 */
export async function tracedToolExecution<T>(
  toolName: string,
  parameters: Record<string, unknown>,
  executeTool: () => Promise<T>,
): Promise<T> {
  return tracer.startActiveSpan(`tool.${toolName}`, async (span: Span) => {
    span.setAttributes({
      "agent.tool.name": toolName,
      "agent.tool.parameters": JSON.stringify(parameters),
    });
 
    try {
      const result = await executeTool();
      span.setStatus({ code: SpanStatusCode.OK });
      return result;
    } catch (err) {
      span.recordException(err instanceof Error ? err : new Error(String(err)));
      span.setStatus({
        code: SpanStatusCode.ERROR,
        message: err instanceof Error ? err.message : "Tool failure",
      });
      throw err;
    } finally {
      span.end();
    }
  });
}

Propagating headers across message queues

When pushing a background job to a message queue, inject the active trace context into headers with propagation.inject():

import { propagation, context } from "@opentelemetry/api";
 
export function createQueueTaskPayload(taskData: Record<string, unknown>) {
  const carrier: Record<string, string> = {};
 
  // Serializes active traceparent and tracestate into the carrier object
  propagation.inject(context.active(), carrier);
 
  return {
    data: taskData,
    traceHeaders: carrier,
  };
}
 
export function executeWorkerTask(payload: { data: unknown; traceHeaders: Record<string, string> }) {
  // Extracts the incoming trace context and executes the worker function within it
  const parentContext = propagation.extract(context.active(), payload.traceHeaders);
 
  return context.with(parentContext, async () => {
    return tracer.startActiveSpan("agent.worker_process", async (span) => {
      try {
        // Run agent state machine with parent trace preserved
        span.setStatus({ code: SpanStatusCode.OK });
      } finally {
        span.end();
      }
    });
  });
}

Architectural tradeoffs and production constraints

When configuring distributed tracing for high-throughput AI services, consider these practical constraints:

• Tail-based sampling over head-based sampling: High-volume applications generate millions of spans. Head-based sampling (randomly sampling 5% at ingestion) risks missing the 0.1% of traces containing critical model hallucination errors or tool exceptions. Route raw spans to an OpenTelemetry Collector and configure tail-based sampling rules that keep 100% of error spans, spans exceeding 2,000ms latency, and high-token outliers.
• PII and prompt data redaction: Storing raw prompt texts and responses inside span attributes simplifies debugging, but creates security and compliance liabilities (GDPR, HIPAA). Add a span processor plugin that strips API keys, credit card numbers, and patient IDs before exporting telemetry out of VPC boundaries.
• Asynchronous batch export: Never use synchronous span exporters in request-response paths. Use the BatchSpanProcessor with bounded buffer queues (e.g., maxQueueSize: 2048, scheduledDelayMillis: 1000) so network timeouts to your OTel collector never degrade user-facing chat streaming latency.

Instrumenting OpenTelemetry semantic spans turns opaque agent loops into structured, observable workflows with pinpoint latency and token attribution across every execution step.