LLM Observability System Design with OpenTelemetry

An LLM call can return HTTP 200 while the application still fails. It may cite the wrong document, repeat a tool call, spend ten times the budget, stream its first token too late, or produce an answer that a validator rejects. Conventional API monitoring sees only part of that failure.

A production design must connect infrastructure health, model behavior, agent decisions, quality, and economics without turning telemetry into a privacy leak. That is the system-design problem.

Quick Answer

A production LLM observability system uses OpenTelemetry to connect each user request to workflow, retrieval, model, and tool spans; records billable tokens, streaming latency, failures, and cost; emits low-cardinality metrics for SLOs; keeps prompt content off by default; and retains complete error, slow, expensive, and sampled baseline traces.

TL;DR

• Trace the full user task. A model span alone cannot explain retrieval, queues, retries, tools, policy checks, or post-processing.
• Create one span per physical attempt. Never overwrite retry data on one span.
• Use provider-reported billable tokens when available. A local tokenizer is an estimate, not billing truth.
• Track end-to-end latency, model duration, time to first chunk, and streaming rate separately.
• Compute cost with an immutable, effective-dated price registry. Keep estimated cost distinct from invoiced cost.
• Export metrics for every request; sample traces for diagnosis. Tail-sample errors, slow and expensive work, denied tools, failed evaluations, plus a random baseline.
• Do not put user IDs, session IDs, trace IDs, or raw prompts in metric labels.
• Treat prompts, outputs, tool arguments, and tool results as sensitive. Content capture is opt-in, redacted, short-lived, and access-controlled.
• Version the telemetry contract. OpenTelemetry GenAI semantic conventions are still developing and now live in a separate repository.

Define the Outcome Before the Telemetry

Use a concrete workload in an interview: a multi-tenant support assistant performs retrieval, calls a hosted model, may check an order through a tool, and streams a response. It serves 500 requests per second at peak.

Functional requirements

1. Follow a request across the API, workflow, retrieval, model, and tools.
2. Measure tokens, latency, errors, cost, retries, cache behavior, and quality.
3. Debug one bad response without collecting all response content.
4. Compare models, prompts, routes, service tiers, and releases.
5. Support hosted APIs and self-hosted GPU inference.
6. Alert on user impact and budget risk within minutes.

Non-functional requirements

• Telemetry must not block the response path when a backend is slow.
• Metric cardinality and trace retention must remain bounded.
• Tenant data must be isolated; credentials and personal data must not enter the normal telemetry path.
• Schema and price changes must preserve historical meaning.
• Missing telemetry must be measurable through collector health and export-loss signals.

Service-level objectives

Avoid one vague “LLM health” target. Define separate indicators.

These numbers are examples, not universal targets. Set them from product needs and measured baselines.

End-to-End LLM Observability Architecture

OpenTelemetry supplies a vendor-neutral API, OTLP transport, Collector, and semantic conventions. It does not decide your SLO, cost model, evaluator, data-retention policy, or authorization model. The Collector documentation describes the receiver–processor–exporter pipeline; the architecture above adds the AI-specific control plane around it.

Model the Request as One Trace

The trace should answer, “Why was this task slow, costly, wrong, or unsafe?” Use the same trace ID from trusted ingress to final response.

Create one inference span for every network request to a model provider. If a logical generation retries twice, put three physical attempt spans under one logical operation span. Parallel tool calls are siblings. For queued, delayed, or offline evaluations, use a span link to the source trace rather than pretending the evaluation was synchronous.

Minimum attribute contract

Use the current OpenTelemetry GenAI span conventions where they apply:

• gen_ai.operation.name: chat, retrieval, invoke_agent, invoke_workflow, or execute_tool.
• gen_ai.provider.name: the provider understood by the instrumentation.
• gen_ai.request.model: the exact model requested.
• gen_ai.response.model: the actual model returned, when available.
• gen_ai.usage.input_tokens and gen_ai.usage.output_tokens: usage returned by the system.
• error.type: a bounded failure class, not a full exception message.

Set low-cardinality sampling attributes such as operation, provider, and requested model when the span starts. End-only facts such as response model, usage, cost, and finish reason cannot help a head sampler.

Use an application namespace such as app.ai.* for fields that are not standard: app.ai.route, app.ai.prompt.version, app.ai.cost.usd, app.ai.price.version, app.ai.policy.outcome, and app.ai.retry.attempt. Do not invent new keys under gen_ai.*; they may collide with future conventions.

Choose the Right Signal for Each Question

Standard and derived metrics

The current OpenTelemetry GenAI metric conventions define these useful instruments and remain in development.

Add product metrics in your own namespace: task success, answer abandonment, retrieval-empty rate, fallback rate, cost per successful task, evaluation pass rate, policy denial, and telemetry drop rate. Metrics must survive trace sampling.

Cardinality rules

Good metric labels are bounded: environment, region, provider, operation, model, route, status class, tool name from a controlled registry, tenant service tier, and release channel.

Bad labels are unbounded: user ID, tenant ID, conversation ID, request ID, trace ID, raw URL, prompt, error message, document ID, and arbitrary tool argument. Prometheus metric guidance warns that every unique label combination creates another time series. Put request identifiers on access-controlled traces or logs, not time-series labels.

Tokens: Decide Which Count Means What

Token accounting breaks when a client, gateway, provider, and inference engine all emit similar numbers.

Hosted models

Use provider-returned billable usage on the client inference span. Preserve separate categories when the provider exposes cached input, reasoning, audio, or other units. A tokenizer run before the call may estimate context size and prevent overflow, but it can differ because the provider may inject text, normalize input, route model versions, or apply billing rules.

Self-hosted models

Keep three views separate:

Do not sum them. Compare them to find gateway retries, server-side prefix caching, speculative decoding, batching, or abandoned streams.

Cancellation edge case

When a client disconnects after three chunks, end the client span with a bounded cancellation outcome. Record provider usage if the provider returns it. If the provider continues generating after disconnection, server compute may exceed client-observed output. That difference is operational waste, not a token-counting bug.

Latency: Four Clocks, Not One

1. End-to-end latency starts at trusted ingress and ends when the response or terminal error reaches the client.
2. Model operation duration covers the physical provider request.
3. Time to first chunk covers the streaming wait before useful output begins.
4. Streaming cadence measures time between output chunks; a chunk may contain zero, one, or many tokens.

For self-hosted inference, also measure admission queue time, scheduling, prefill, decode, and GPU work. Provider APIs rarely expose all of these phases, so do not fabricate them by subtracting unrelated clocks.

Use a monotonic clock inside one process. Across services, rely on span durations and periodically monitor time synchronization. A negative child offset on a trace visualization often indicates clock skew, not time travel.

Cost: Build a Ledger, Not a Dashboard Formula

OpenTelemetry does not standardize provider pricing. A robust design stores usage first, then joins it to an immutable price record.

estimated_cost =
  input_tokens × input_unit_price +
  cached_input_tokens × cached_input_unit_price +
  output_tokens × output_unit_price +
  provider_specific_units × corresponding_unit_price

The formula is complete only for the price dimensions that the selected provider contract defines. Store these fields with each ledger entry:

• provider, requested model, response model, region, service tier, and batch mode;
• each usage category and unit;
• price-card ID, currency, effective start and end time;
• trace ID, logical task ID, attempt, and completion outcome; and
• estimated, reconciled, or invoiced status.

Never overwrite an old price row. Select the price effective at event time. Run a daily reconciliation against the provider billing export and alert on material variance. Operational dashboards should show both cost per call and cost per successful task; retries can make the second number deteriorate while the first looks stable.

Tool Calls and Agent Loops

A safe trace distinguishes proposal, authorization, execution, and effect.

Give every physical tool attempt a span. Count agent turns, inference calls, tool calls, repeated call signatures, elapsed time, and remaining budget. A loop alert should trigger on both absolute limits and a sudden change from the route's normal distribution.

Common Causes

Symptoms

• Provider invoice exceeds the dashboard by a stable or growing percentage.
• p95 looks healthy while users report slow first output.
• One trace shows a root span but missing model or tool children.
• Agent cost rises without traffic growth.
• Trace search becomes slow after adding tenant or request labels.
• Successful HTTP responses have poor evaluation or user-feedback scores.
• Tool counts exceed configured agent budgets.
• Collector queues climb and exported-span counts fall during traffic spikes.
• Sensitive prompt text appears in a general-purpose trace backend.
• GPU utilization is low while model queue time is high, or utilization is high while tokens per second falls.

Root Cause

The root cause is usually a boundary mismatch: the system measures a transport call while the product delivers a multi-step task.

Step-by-step Solution

1. Write a telemetry contract

Define span names, parentage, units, attributes, labels, owners, and retention in version control. Use golden traces so different SDKs produce the same logical tree.

2. Establish resource identity

Every process exports service.name, version, environment, region, and deployment identity. Put model and prompt versions on relevant spans, not service names.

3. Instrument the user task and physical attempts

Auto-instrument standard clients. Add manual spans for workflow, agent, retrieval, inference, tool policy, tool execution, and evaluation. Propagate context only across trusted boundaries.

4. Record usage where it becomes authoritative

Set requested model at span creation. Add response model, finish reason, and usage when returned. Mark local counts as estimated; do not emit usage for a fetch that performs no inference.

5. Measure streaming explicitly

Start before issuing the request, observe first-chunk time once, and end on complete, error, timeout, or cancellation. Chunks are not tokens.

6. Derive cost asynchronously

Send normalized usage to a cost worker, resolve the effective price, and write an append-only ledger entry. Reconciliation updates ledger status, not token history.

7. Separate metrics, traces, and content

Emit bounded metrics for all operations and sampled traces for diagnosis. Approved content goes to a separate encrypted store with short TTL and narrower access.

8. Redact before export

Disable content capture and filter known sensitive attributes before export. OpenTelemetry documents Collector processors for sensitive data. Backend-only redaction is too late after a trust boundary.

9. Tail-sample complete traces

Keep errors, denials, slow or high-cost tasks, failed evaluations, and a random baseline. Wait past expected trace duration and route every span with one trace ID to one sampler. See the official Collector scaling guidance.

10. Alert on outcomes and pipeline health

Alert on availability, latency, TTFC, budget burn, evaluation regression, retries, and tool failures. Also alert on Collector queues, refused records, and export errors.

Minimal Reproducible TypeScript Example

This example creates an agent root span, a tool span, and a streaming model span. It records provider usage, first-chunk latency, and an explicitly illustrative cost estimate without capturing content.

package.json:

{
  "name": "otel-llm-observability-demo",
  "private": true,
  "type": "module",
  "scripts": {
    "start": "tsx src/observe.ts"
  },
  "dependencies": {
    "@opentelemetry/api": "^1.9.0",
    "@opentelemetry/resources": "^2.10.0",
    "@opentelemetry/sdk-trace-base": "^2.10.0",
    "@opentelemetry/sdk-trace-node": "^2.10.0"
  },
  "devDependencies": {
    "tsx": "^4.23.0",
    "typescript": "^5.9.0"
  }
}

src/observe.ts:

import { SpanKind, SpanStatusCode, trace } from "@opentelemetry/api";
import { resourceFromAttributes } from "@opentelemetry/resources";
import {
  InMemorySpanExporter,
  SimpleSpanProcessor,
} from "@opentelemetry/sdk-trace-base";
import { NodeTracerProvider } from "@opentelemetry/sdk-trace-node";
 
const exporter = new InMemorySpanExporter();
const provider = new NodeTracerProvider({
  resource: resourceFromAttributes({
    "service.name": "support-assistant",
    "service.version": "2026.08.0",
    "deployment.environment.name": "development",
  }),
  spanProcessors: [new SimpleSpanProcessor(exporter)],
});
provider.register();
 
const tracer = trace.getTracer("example.llm-observability", "1.0.0");
 
const sleep = (milliseconds: number): Promise<void> =>
  new Promise((resolve) => setTimeout(resolve, milliseconds));
 
async function executeOrderTool(): Promise<void> {
  await tracer.startActiveSpan(
    "execute_tool get_order_status",
    {
      kind: SpanKind.INTERNAL,
      attributes: {
        "gen_ai.operation.name": "execute_tool",
        "gen_ai.tool.name": "get_order_status",
        "app.ai.retry.attempt": 1,
        "app.ai.policy.outcome": "allow",
      },
    },
    async (span) => {
      try {
        await sleep(2);
        span.setAttribute("app.ai.tool.effect", "read_succeeded");
      } catch (error) {
        span.recordException(error as Error);
        span.setAttribute(
          "error.type",
          error instanceof Error ? error.name : "_OTHER",
        );
        span.setStatus({ code: SpanStatusCode.ERROR, message: "tool_failed" });
        throw error;
      } finally {
        span.end();
      }
    },
  );
}
 
async function streamModel(): Promise<number> {
  return tracer.startActiveSpan(
    "chat demo-model",
    {
      kind: SpanKind.CLIENT,
      attributes: {
        "gen_ai.operation.name": "chat",
        "gen_ai.provider.name": "example_provider",
        "gen_ai.request.model": "demo-model",
        "server.address": "api.example.invalid",
        "app.ai.prompt.version": "support-v7",
        "app.ai.retry.attempt": 1,
      },
    },
    async (span) => {
      const startedAt = performance.now();
      let firstChunkAt: number | undefined;
 
      try {
        for (const ignoredChunk of ["Order", " found."]) {
          await sleep(2);
          firstChunkAt ??= performance.now();
          void ignoredChunk;
        }
 
        if (firstChunkAt === undefined) {
          throw new Error("stream_ended_without_chunks");
        }
 
        const inputTokens = 120;
        const outputTokens = 18;
        const estimatedCostUsd = Number(
          (inputTokens * 0.0000025 + outputTokens * 0.00001).toFixed(8),
        );
 
        span.setAttributes({
          "gen_ai.response.model": "demo-model-2026-08",
          "gen_ai.usage.input_tokens": inputTokens,
          "gen_ai.usage.output_tokens": outputTokens,
          "app.ai.time_to_first_chunk_ms": firstChunkAt - startedAt,
          "app.ai.cost.usd": estimatedCostUsd,
          "app.ai.cost.status": "estimated",
          "app.ai.price.version": "price-card-demo-2026-08",
        });
        return estimatedCostUsd;
      } catch (error) {
        span.recordException(error as Error);
        span.setAttribute(
          "error.type",
          error instanceof Error ? error.name : "_OTHER",
        );
        span.setStatus({ code: SpanStatusCode.ERROR, message: "model_failed" });
        throw error;
      } finally {
        span.end();
      }
    },
  );
}
 
async function main(): Promise<void> {
  await tracer.startActiveSpan(
    "invoke_agent support-router",
    {
      kind: SpanKind.INTERNAL,
      attributes: {
        "gen_ai.operation.name": "invoke_agent",
        "gen_ai.agent.name": "support-router",
        "app.ai.route": "order-status",
      },
    },
    async (span) => {
      try {
        await executeOrderTool();
        const costUsd = await streamModel();
        span.setAttribute("app.ai.cost.usd", costUsd);
        span.setAttribute("app.ai.task.outcome", "success");
      } catch (error) {
        span.setAttribute("error.type", "task_failed");
        span.setStatus({ code: SpanStatusCode.ERROR, message: "task_failed" });
        throw error;
      } finally {
        span.end();
      }
    },
  );
 
  await provider.forceFlush();
  for (const span of exporter.getFinishedSpans()) {
    const operation = span.attributes["gen_ai.operation.name"];
    const cost = span.attributes["app.ai.cost.usd"] ?? "-";
    const status = span.status.code === SpanStatusCode.ERROR ? "ERROR" : "OK";
    console.log(`${span.name} | ${status} | ${operation} | ${cost}`);
  }
  await provider.shutdown();
}
 
await main();

Run npm install and npm start. Expected output is:

execute_tool get_order_status | OK | execute_tool | -
chat demo-model | OK | chat | 0.00048
invoke_agent support-router | OK | invoke_agent | 0.00048

Important lines:

• NodeTracerProvider registers a real SDK; the API alone otherwise returns a no-op tracer.
• startActiveSpan preserves parent context across asynchronous work.
• The model span starts with sampling-relevant fields and adds response facts later.
• performance.now() is monotonic within the process.
• Successful spans leave OpenTelemetry status unset; the demo prints non-errors as OK for readability.
• The demo price is fictional. Production code looks up an effective-dated price record and never embeds rates in request code.
• No prompt, response, tool argument, user ID, or API key enters the span.

Common mistakes are importing and initializing telemetry after the application, forgetting span.end() in an error path, calling shutdown() before queued exports flush, and using SimpleSpanProcessor in production. Use batch processing and OTLP to a Collector on the production path.

Production Collector Configuration

This complete Collector configuration accepts OTLP, removes common content attributes, retains anomalous plus baseline traces, batches all signals, and exports over TLS. The three environment variables must be supplied by the deployment.

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318
 
processors:
  memory_limiter:
    check_interval: 1s
    limit_mib: 1024
    spike_limit_mib: 256
 
  transform/trace_privacy:
    error_mode: ignore
    trace_statements:
      - context: span
        statements:
          - delete_key(attributes, "gen_ai.input.messages")
          - delete_key(attributes, "gen_ai.output.messages")
          - delete_key(attributes, "gen_ai.system_instructions")
          - delete_key(attributes, "gen_ai.tool.call.arguments")
          - delete_key(attributes, "gen_ai.tool.call.result")
 
  tail_sampling:
    decision_wait: 60s
    num_traces: 100000
    expected_new_traces_per_sec: 1000
    policies:
      - name: errors
        type: status_code
        status_code:
          status_codes: [ERROR]
      - name: slow
        type: latency
        latency:
          threshold_ms: 3000
      - name: high_cost
        type: numeric_attribute
        numeric_attribute:
          key: app.ai.cost.usd
          min_value: 0.10
      - name: denied_tools
        type: string_attribute
        string_attribute:
          key: app.ai.policy.outcome
          values: [deny]
      - name: baseline
        type: probabilistic
        probabilistic:
          sampling_percentage: 5
 
  batch:
    send_batch_size: 8192
    timeout: 5s
 
exporters:
  otlp/backend:
    endpoint: ${env:OTEL_BACKEND_ENDPOINT}
    headers:
      authorization: "Bearer ${env:OTEL_BACKEND_TOKEN}"
    tls:
      ca_file: ${env:OTEL_BACKEND_CA_FILE}
 
service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, transform/trace_privacy, tail_sampling, batch]
      exporters: [otlp/backend]
    metrics:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [otlp/backend]
    logs:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [otlp/backend]

This is defense in depth, not permission to enable content capture. Structured GenAI events may contain content in event bodies that these span-attribute deletions do not cover. Keep capture disabled in the SDK and add schema-specific event filtering before using such events. Also choose decision_wait and num_traces from your real p99 duration and arrival rate; the example values are starting assumptions.

The Collector configuration reference notes that a configured processor is inactive until it is placed in a pipeline and that processor order matters. Validate the configuration with the exact Collector distribution and version used in production.

Sampling and Capacity Planning

At 500 requests per second, eight spans per request, and an average serialized span size of 1.5 KB, the application produces about 4,000 spans or 6 MB per second—roughly 518 GB per day before backend indexing and replication. Measure your own payload; prompt events can make it much larger.

Do not sample metrics using the trace decision. A 5% trace sample may support debugging, but a p99 latency SLO derived from that conditional sample can be biased. Attach exemplars where supported so a metric spike can lead to a representative trace.

Monitor Collector queue size, queue capacity, enqueue failures, refused spans, export failures, process memory, and tail-sampler late spans. Scale stateless receivers independently. Before stateful tail-sampling replicas, use the Collector load-balancing exporter keyed by trace ID so one decision sees the whole trace.

Privacy and Security Design

Telemetry is a second data plane. Treat it as production data, not harmless debug text.

• Disable gen_ai.input.messages, gen_ai.output.messages, system instructions, tool definitions, arguments, and results by default. The GenAI conventions mark this content as sensitive and opt-in.
• Allowlist attributes at trust boundaries. Redaction patterns alone miss encoded, nested, or novel secrets.
• Never place credentials, authorization headers, cookies, raw PII, or payment data in spans, baggage, logs, or metric labels.
• Reject or sanitize trace context from untrusted ingress where your threat model requires it. Do not propagate internal baggage to external model providers.
• Encrypt transport and storage. Separate tenant access, operator roles, retention, deletion, legal holds, and audit logs.
• Hash identifiers only when stable linkage is needed and the privacy team accepts the re-identification risk. Hashing is pseudonymization, not anonymization.
• Rate-limit diagnostic capture. An attacker should not be able to force expensive or sensitive traces by manufacturing errors.

OpenTelemetry's baggage security guidance warns that baggage can reach unintended third parties through automatic propagation and has no built-in integrity guarantee.

Evaluation and User Feedback

Operational success and answer quality are independent. A complete design uses:

• deterministic online checks for schema validity, citations, policy, and required fields;
• sampled asynchronous evaluators for groundedness, relevance, safety, and task completion;
• human review for high-impact domains and disputed cases; and
• product feedback such as acceptance, edit distance, abandonment, escalation, and task reversal.

Record evaluator name, version, rubric version, threshold, dataset slice, and score. Do not label metrics with free-form feedback. Link late evaluations to the original trace and preserve the prompt, model, retrieval-index, and application release versions needed to explain a regression.

An evaluator model can drift or fail. Monitor evaluator latency, errors, agreement with human labels, and score distribution. Never silently substitute a new judge while comparing releases.

Debug Decision Tree

How to Reproduce and Debug a Failure

For a cost spike, choose one known route and replay a sanitized fixture in staging:

1. Pin application, prompt, retrieval index, model route, and evaluator versions.
2. Disable semantic cache so the first run exercises the full path.
3. Inject one provider timeout, allow exactly one retry, and record both attempts.
4. Cancel one streaming run after its first chunk.
5. Run one tool call that policy denies and one that succeeds.
6. Compare client usage, provider response usage, cost ledger, and trace hierarchy.
7. Repeat with cache enabled and confirm only the intended usage category changes.

A correct trace contains separate retry attempts, an explicit cancellation, policy and effect outcomes, and one root task cost. The aggregate metric count must include all runs even if the diagnostic trace sample keeps only some successful baselines.

Troubleshooting matrix

Verification Steps

Use this checklist before production rollout:

• A single test request produces the expected root, retrieval, inference, and tool span tree with one trace ID.
• Two retries produce two attempt spans and one logical outcome.
• Streaming TTFC is lower than total duration and absent for non-streaming calls.
• Provider-returned billable usage matches the recorded token categories.
• A known usage fixture resolves to the expected price-card version and cost.
• Daily estimated cost reconciles with provider billing within the approved tolerance.
• Metrics remain unchanged when trace sampling changes from 100% to 5%.
• Error, slow, high-cost, denied-tool, and random baseline traces are retained with all children.
• Prompts, outputs, secrets, user IDs, and tool payloads are absent from normal exported telemetry.
• Collector overload tests raise loss alerts before queues fill.
• A revoked operator cannot query another tenant's restricted traces.
• Dashboards compare releases using pinned prompt, route, model, index, and evaluator versions.

Platform-Specific Notes

Version-Specific Differences in 2026

The official OpenTelemetry JS 2.x migration guide documents the tracer-provider change. The main semantic-conventions index now points GenAI users to the dedicated repository. Treat copied blog snippets as dated until checked against the version you deploy.

Alternative Approaches

The practical answer is hybrid: OpenTelemetry for correlated operational signals, provider telemetry for authoritative provider-side detail, and dedicated ledgers for cost and evaluations.

Prevention

Prevent observability drift through operating discipline:

• Own one reviewed telemetry contract and compatibility test suite.
• Wrap model and tool SDKs behind shared instrumentation adapters.
• Pin Collector images, semantic-convention expectations, and dashboards.
• Require bounded-label review for every new metric dimension.
• Keep content capture off; test redaction with seeded secrets and adversarial nested payloads.
• Reconcile cost daily and publish the unexplained variance.
• Test timeouts, cancellation, fallbacks, retries, rate limits, and partial streams.
• Capacity-test collectors at peak spans per second and largest permitted payload.
• Run periodic access reviews and deletion drills for telemetry stores.
• Treat missing telemetry, evaluator drift, and stale price cards as production incidents with owners and runbooks.

Key Takeaways

The best LLM observability design starts at the user task and works inward. Trace every retrieval, model attempt, and tool effect; aggregate reliable metrics outside sampling; calculate cost from provider usage and effective-dated prices; and link quality evaluation back to the exact versions that produced the answer.

OpenTelemetry is the interoperability layer, not the whole solution. Schema governance, privacy controls, price reconciliation, trace-aware sampling, and telemetry health are what make it production-ready.

Suggested Internal Links

• Designing production RAG retrieval and reranking
• Building a secure multi-tenant MCP gateway
• AI agent security and tool authorization
• OpenTelemetry distributed tracing fundamentals
• Designing multi-window SLO alerts
• LLM evaluation datasets and regression testing
• GPU inference capacity planning
• Controlling metric cardinality in Prometheus

Official References

• OpenTelemetry GenAI span semantic conventions
• OpenTelemetry GenAI metric semantic conventions
• OpenTelemetry GenAI event semantic conventions
• OpenTelemetry Collector configuration
• OpenTelemetry Collector scaling
• OpenTelemetry sensitive-data guidance
• OpenTelemetry error-recording conventions
• OpenTelemetry JavaScript exporters
• Prometheus metric and label naming
• NVIDIA DCGM Exporter documentation

FAQs

What should an LLM observability system monitor?

Monitor task success, end-to-end and model latency, time to first chunk, tokens, cost, retries, retrieval, tools, safety decisions, and evaluation scores. Correlate them with one trace ID; keep labels bounded and content disabled by default.

How should OpenTelemetry spans be structured for an AI agent?

Create a root request or workflow span. Add child spans for each agent, retrieval, physical model request, and tool execution. Retries get separate spans; parallel work uses siblings. Link delayed evaluation to the source trace.

How do you calculate LLM cost accurately?

Use provider-reported billable usage and preserve each priced unit. Join it to an immutable price record selected by provider, actual model, region, tier, batch mode, and event time. Reconcile the estimate against billing exports.

Should prompts and responses be stored in traces?

No. They can contain credentials, personal data, proprietary text, or hostile content. Prefer size, hash, classification, and policy outcome. Approved diagnostic content belongs in a redacted, restricted, short-retention store.

What is the difference between time to first token and total latency?

Time to first token or chunk ends when streaming output starts. Model duration ends at the final chunk. End-to-end latency also includes queues, retrieval, tools, retries, and post-processing. Each boundary diagnoses different work.

Should LLM traces use head sampling or tail sampling?

Tail sampling can retain errors, high cost, long duration, denials, failed evaluations, and a random baseline. Keep metrics unsampled and route by trace ID so the stateful sampler receives complete traces.

How do you avoid double-counting tokens?

For hosted models, use provider-returned billable usage on the client span. For self-hosted inference, separate client demand, server admission, and engine compute. Compare these views; never sum them as independent billable usage.

How do you observe LLM tool calls safely?

Record bounded tool name, attempt, duration, outcome, policy decision, and idempotency result. Exclude unrestricted arguments and results. Trace proposal, authorization, execution, and business effect separately.