Design observability that stays useful under cardinality and cost pressure

• Identify operational questions, required aggregation, tenant support needs, retention, and privacy. Decide which dimensions belong in metrics, searchable logs, trace attributes, or external exemplars.

• Estimate series cardinality as the product of labels, event volume, trace sampling, ingestion peaks, and query patterns. Model incident bursts and multi-tenant fairness for telemetry itself.

• Core SLO and saturation signals survive incidents, teams can drill into exemplars, sensitive data is controlled, cardinality has budgets, and cost is attributable to producers.

• Use low-cardinality metrics, histogram exemplars pointing to sampled traces, structured logs with indexed-field limits, tenant-aware ingestion quotas, tiered retention, and tail sampling for errors and slow requests.

• Telemetry SDKs enforce schemas and resource identity; collectors redact, aggregate, sample, and route by criticality. High-cardinality IDs remain in traces or logs, not metric dimensions.

• Telemetry is lossy by design, but SLO counters and audit signals receive protected durable lanes. Sampling metadata lets analysts understand bias and approximate populations.

• Collectors buffer within bounds, shed debug data before golden signals, and isolate noisy producers. A telemetry backend outage must not block application requests.

• Redact secrets and personal data at source and collector, scope tenant access, and audit sensitive trace searches. Shorten retention for rich payloads.

• Observe the observability system: dropped spans, series growth, queue age, sampling rate, query latency, ingestion by team, and protected-signal completeness.

• Assign budgets by team and data class, show cost per useful query or incident, and tier old data to object storage. Optimize schemas before negotiating more backend capacity.

• Head sampling controls cost but misses rare slow outcomes; tail sampling captures them but requires buffering and distributed trace completion. Combine baseline head sampling with targeted tail rules.

• Freeze new unapproved metric labels, provide migration tooling and exemplars, then enforce schemas and budgets in CI and collectors. Remove unused indexes based on query telemetry.

• Staff candidates choose the right telemetry type for each question and protect the observability system as critical infrastructure. They balance debugging, bias, privacy, and cost.

Interviewer evaluation rubric