Interview prompt
Problem context
Skills being evaluated
Use the sequence below to surface constraints, choose boundaries, test failure behavior, and defend trade-offs. Concrete numbers are interview assumptions, not claims about a real production system.
Clarify the decision
- Define whether users require point-in-time consistency across widgets or only per-entity freshness. Identify source-of-truth tables, event-log retention, rebuild duration, and acceptable visible staleness.
Establish scale assumptions
- Assume five billion derived rows, 50k source changes/second, and a rebuild lasting days. Estimate storage for parallel view versions and ensure the delta log remains available until every range catches up.
Functional and non-functional requirements
- Never mix incompatible view schemas in one response, publish a complete version atomically at a defined source position, and expose freshness. The old view remains serviceable until the new one is proven.
High-level architecture
- Create a versioned view namespace. Build from a consistent snapshot at position L, apply changes after L into the new version, validate by partition, then atomically move a routing pointer once global readiness criteria pass.
Data model and flow
- Backfill stable key ranges with source position and transform version, while a delta consumer upserts idempotently. Per-range watermarks and checksums let the controller know which partitions are complete and caught up.
Consistency and transaction boundaries
- A request reads one declared view version and its watermark. Multi-widget responses pin that version; monotonic routing prevents a user bouncing from new back to old except during an explicit rollback.
Failure modes and recovery
- Failed partitions retry from checkpoints, schema-poison events quarantine with visible gaps, and a lagging range blocks only publication. If post-cutover errors appear, move the pointer back while the old view remains current through dual consumption.
Security and privacy
- Apply row and tenant filters during both snapshot and stream paths, and validate that transformed fields do not broaden access. Temporary parallel stores inherit encryption, retention, and deletion requirements.
Observability and SLOs
- Measure range completion, delta lag, checksum and aggregate differences, query-result shadow comparisons, freshness, and version adoption. Users and support should see the data-as-of timestamp.
Capacity and cost
- Parallel versions can double storage and write amplification, so budget the overlap and clean only after rollback expiry. Throttle source scans and isolate rebuild compute from serving capacity.
Alternatives and trade-offs
- In-place mutation saves storage but makes mixed-schema results unavoidable. Versioned rebuilds cost temporary duplication but provide a clean publication and rollback boundary.
Evolution and migration
- Ship readers that understand version metadata before creating the new view. Shadow reads, compare aggregates and selected entities, canary internal users, then flip cohorts and finally the global pointer.
What Staff and Principal candidates should emphasize
- The core is a shared source position across snapshot and stream plus an atomic version switch. Strong candidates discuss user-visible freshness, cross-widget pinning, validation, and old-view maintenance during rollback.
Decision trade-offs
Rebuild target
Option A
Mutate the existing view in place
Option B
Build a parallel versioned view
Recommendation:Use a parallel version when logic or schema is incompatible; temporary duplication buys a provable cutover and rollback.
Publication
Option A
Expose ranges as they finish
Option B
Atomically route complete cohorts or the full dataset
Recommendation:Expose partial ranges only if product semantics tolerate mixed freshness; otherwise pin each request to a complete version.
Follow-up interview questions
- 01How do you join a days-long snapshot with a live event stream without a gap?
- 02What if one partition never catches up before log retention expires?
- 03How do you keep the old view current enough for rollback?
- 04Can publication be per tenant rather than global?
Common weak answers and mistakes
- 01Rebuilding in place and serving a mixture of old and new transform logic.
- 02Starting the stream without a snapshot position that defines the boundary.
- 03Validating only totals and missing per-tenant authorization errors.
- 04Dropping the old view before the new version survives real traffic.
Interviewer evaluation rubric
Runs a backfill and hopes eventual consistency converges, without a snapshot boundary or versioned serving plan.
Uses snapshot plus deltas, checkpoints, validation, and a parallel view with controlled cutover.
Pins reads to versions, defines per-range watermarks, preserves rollback currency, and exposes freshness and security proof.
Adapts publication granularity to product semantics and provides automated evidence that both data and access behavior remain equivalent.