Interview prompt
Problem context
Skills being evaluated
Use the sequence below to surface constraints, choose boundaries, test failure behavior, and defend trade-offs. Concrete numbers are interview assumptions, not claims about a real production system.
Clarify the decision
- Classify which signals must alert locally, which need eventual central analysis, retention during isolation, and expected partition duration. Define trusted clock and sequence behavior.
Establish scale assumptions
- Estimate regional telemetry volume, protected-signal subset, buffer disk, compression, and replay bandwidth. A long partition plus recovery burst must not evict current incident data.
Functional and non-functional requirements
- Local customer SLO and safety alerts continue, core telemetry is durably buffered, loss is explicit, replay preserves provenance, and central views distinguish missing from healthy.
High-level architecture
- Run regional collectors and alert evaluators, tiered local buffers, protected lanes for SLO and audit signals, and multiple egress paths. Central ingestion deduplicates by source and sequence and accepts late data with original timestamps.
Data model and flow
- Every batch carries region, collector, boot epoch, sequence range, event and ingest time, sampling policy, and loss counters. Replay is rate-limited and interleaved with fresh telemetry.
Consistency and transaction boundaries
- Central aggregates are eventually corrected and mark incomplete regions. Do not retroactively pretend an alert fired on time; preserve evaluated-at and received-at semantics.
Failure modes and recovery
- When buffers fill, shed debug logs, then sampled traces, while protecting SLO and security signals. Local dashboards and out-of-band paging remain accessible to regional responders.
Security and privacy
- Encrypt buffers, restrict local access, preserve audit immutability, and avoid emergency exports that bypass residency. Alternate egress is authenticated and rate-limited.
Observability and SLOs
- Monitor buffer age, fill, loss by class, sequence gaps, clock skew, local-alert health, replay lag, and central completeness. Display gray unknown states rather than green absence.
Capacity and cost
- Retain compressed critical signals for the maximum planning partition and use cheaper local storage for debug data. Multiple egress and local alerting cost money but protect incident truth.
Alternatives and trade-offs
- Full local copies are expensive; sending everything centrally is fragile. Protect a small operational truth set locally and accept explicit loss for lower-value detail.
Evolution and migration
- Add source sequences and loss counters, regional buffering, completeness-aware dashboards, then local alerting. Exercise a telemetry-only partition during live traffic.
What Staff and Principal candidates should emphasize
- Principal candidates model telemetry as a distributed system and make unknown visible. They cover local decision capability, clocks, replay, and prioritization under buffer exhaustion.
Decision trade-offs
Alert evaluation
Option A
Central-only rules
Option B
Regional critical rules plus central global rules
Recommendation:Evaluate region-critical SLO and safety signals locally; centralize cross-region correlation and portfolio views.
Buffer pressure
Option A
Drop newest data
Option B
Priority-aware shedding with loss counters
Recommendation:Protect current critical signals and explicitly shed lower-value classes so silence can never be mistaken for health.
Follow-up interview questions
- 01How should a dashboard display a disconnected region?
- 02How are late events merged without duplicate counts?
- 03Which telemetry classes survive disk pressure?
- 04How does local paging work if central identity is unavailable?
Common weak answers and mistakes
- 01Assuming missing telemetry means zero errors.
- 02Buffering without priorities until disk fills and all signals disappear.
- 03Replaying backlog at full speed and overwhelming central ingestion.
- 04Using ingestion time as the incident timeline for late events.
Interviewer evaluation rubric
Adds local disk buffering but cannot preserve alerting, completeness, sequence, time, or controlled replay.
Uses regional collectors, priority buffers, local alerts, source sequences, loss counters, and rate-limited replay.
Adds completeness-aware views, clock semantics, protected audit lanes, out-of-band access, and recovery interleaving.
Defines a minimal operational truth architecture that remains trustworthy through regional, identity, storage, and central-backend failure.