Interview prompt
Problem context
Skills being evaluated
Use the sequence below to surface constraints, choose boundaries, test failure behavior, and defend trade-offs. Concrete numbers are interview assumptions, not claims about a real production system.
Clarify the decision
- Define completeness by business entity and time, tolerated lateness, correction policy, source authority, and whether records may be intentionally filtered. Separate pipeline availability from data correctness.
Establish scale assumptions
- Estimate events, partitions, tenants, lateness, retention, and reconciliation scan cost. Design summaries and checkpoints that make billions of records comparable without full joins every hour.
Functional and non-functional requirements
- Detect missing, duplicate, corrupt, late, and unauthorized data end to end; localize the loss stage; quarantine safely; repair idempotently; and report freshness and completeness to users.
High-level architecture
- Carry source sequence and lineage, emit per-partition count and hash manifests at each stage, maintain watermarks and invariant checks, and run independent source-to-sink reconciliation. A repair controller replays bounded ranges.
Data model and flow
- Each record carries event ID, source position, tenant, schema, event time, and transform version. Stage manifests summarize accepted, filtered, rejected, and emitted sets so intentional drops are distinguishable.
Consistency and transaction boundaries
- Use at-least-once processing with idempotent sinks and explicit finality windows. Financial reports publish certified snapshots only after required watermarks and reconciliation gates.
Failure modes and recovery
- Quarantine poison data with reason and owner while allowing unrelated partitions. If retention threatens a gap, snapshot the missing source range before log expiry and stop false certification.
Security and privacy
- Completeness manifests avoid sensitive payloads, tenant access is scoped, and rejected data follows secure retention. Repair cannot replay data into an unauthorized region or tenant.
Observability and SLOs
- Track source-to-stage deltas, sequence gaps, watermark age, intentional filters, schema rejects, manifest mismatch, repair outcomes, and certified-snapshot status. Page on correctness burn, not only worker health.
Capacity and cost
- Hierarchical hashes and sampled record comparison keep continuous checks affordable; full reconciliation runs by risk tier and accounting period. Correctness evidence is part of pipeline cost.
Alternatives and trade-offs
- Per-record lineage offers precise proof but is expensive; aggregate manifests localize most gaps cheaply. Combine manifests with targeted per-record tracing for high-risk domains.
Evolution and migration
- Add stable IDs and source positions, then manifests stage by stage, shadow reconciliation, and block certification on proven gates. Backfill historical gaps only with explicit provenance.
What Staff and Principal candidates should emphasize
- Distinguished candidates define a data SLO and certification boundary. They connect technical lineage to finance, tenant visibility, ownership, and a repair protocol that does not create duplicates.
Decision trade-offs
Completeness evidence
Option A
Trace every record through every stage
Option B
Hierarchical manifests plus targeted lineage
Recommendation:Use manifests for continuous scale and retain per-record lineage for exceptions and high-consequence flows.
Publication
Option A
Serve best-effort continuously
Option B
Publish provisional then certified snapshots
Recommendation:Expose provisional freshness for exploration and require reconciled certification for revenue or regulatory reporting.
Follow-up interview questions
- 01How do you distinguish an intentional filter from data loss?
- 02What if a gap is found after source retention expired?
- 03How can hashes compare records transformed into a new schema?
- 04Who owns the end-to-end data SLO?
Common weak answers and mistakes
- 01Treating green worker metrics as evidence of complete data.
- 02Comparing only total counts and hiding tenant or partition loss.
- 03Stopping a whole pipeline on one poison record without a quarantine contract.
- 04Replaying ranges without idempotent sinks and creating duplicates.
Interviewer evaluation rubric
Adds more service alerts but cannot define completeness, lineage, finality, or safe repair.
Uses stable IDs, watermarks, sequence gaps, manifests, quarantine, reconciliation, and replay.
Adds transform-aware hashes, certified states, tenant scope, retention rescue, correctness SLO, and repair evidence.
Creates an enterprise data-trust system aligning technical stages, financial certification, customer transparency, and accountable ownership.