Observe the Decision Path, Not Just the Model Call

Start from the outcome a user or downstream system experienced. A useful trace links the request to eligibility, retrieval corpus and index revision, policy checks, prompt-template and model configuration, tool proposals and receipts, response validation, fallback, and final rendering. Each component still owns its local logs and metrics; the decision trace carries bounded correlation evidence rather than duplicating every payload.

Prefer stable version identifiers, enumerated statuses, counts, timing, costs, and keyed or collision-resistant digests over raw content. A digest is still sensitive when the input space is small or guessable, and it cannot prove the logged content was truthful. Tenant identifiers need purpose-specific pseudonyms and access controls; trace IDs crossing an untrusted boundary must be validated for format, cardinality, and abuse before being joined to internal data.

1. 01Classify fieldsName purpose, sensitivity, source, cardinality, allowed values, access, retention, deletion behavior, and whether downstream export is permitted.
2. 02Version the schemaMake producer and consumer compatibility explicit so a deployment cannot silently reinterpret stage, status, cost, or identity fields.
3. 03Separate content accessIf an authorized investigation truly needs payloads, retrieve them through a narrower audited system rather than placing them in default traces.
4. 04Delete coherentlyPropagate retention and deletion across exporters, warehouses, exemplars, alert attachments, and debugging copies, not just the primary span store.

1export function auditDecisionTrace(contractValue: unknown, traceValue: unknown): TraceReport {
2  const contract = validateContract(contractValue)
3  const trace = validateTrace(traceValue)
4  if (!sameStrings(trace.scope, contract.scope) || trace.contractContentId !== contract.contentId) {
5    throw new TypeError("trace belongs to another observability contract")
6  }
7  // Check declared stage presence, cardinality, order, status, latency, and cost.
8  return Object.freeze(report)
9}

example=illustrative_only
status=TRACE_COMPLETE
events=5;stages=retrieval,policy,model,tool,response
latency_ms=480;cost_microusd=1500
claim=HASHED_LOCAL_TRACE_NOT_PAYLOAD_CAPTURE

The fixture declares exactly five required stages and records invented version IDs, tenant surrogate, input and output SHA-256 values, stage status, latency, and cost. Exact plain objects and arrays, safe integers, scope length, event order, closed schemas, digest binding, constructor bypass, raw-payload mode, and total budgets are tested. The returned report and its collections are frozen.

The artifact neither emits OpenTelemetry spans nor authenticates an incoming trace header. It does not capture a prompt, reconstruct causal truth, score model quality, guarantee delivery, or prove that a digest corresponds to honest source data. Its complete status is a local schema result over invented events.

Use trace exemplars to move from an aggregate symptom to a bounded path, then verify source systems. Online evaluation needs treatment and version attribution. SLOs need end-to-end product latency and success, not just model availability. Incident response needs affected revisions, stage status, policy decisions, and downstream receipts. Offline datasets can sample privacy-reviewed failure traces, but the sampling decision and provenance must travel with each case.

• Monitor trace completion and export health separately from product success so telemetry loss is visible.
• Budget per-stage and end-to-end latency and cost; diagnose retries and fallbacks instead of summing only provider usage.
• Restrict high-cardinality fields and debug access with tenant-aware authorization and immutable audit trails.
• Test schema compatibility, deletion, sampling, incident joins, and fallback attribution during release rehearsals.