Interview prompt
Problem context
Skills being evaluated
Use the sequence below to surface constraints, choose boundaries, test failure behavior, and defend trade-offs. Concrete numbers are interview assumptions, not claims about a real production system.
Clarify the decision
- Classify operations as intent mutation, reconciliation, serving, and telemetry. Identify what serving needs at request time and how stale that materialized state may be when the management plane is unavailable.
Establish scale assumptions
- Assume management writes are low QPS but expensive and bursty, while the data plane handles millions of requests per second. Size independent stores and pools around their distinct latency and availability profiles.
Functional and non-functional requirements
- Data-plane serving continues from validated local state during control-plane outage. Intent changes are durable, versioned, auditable, and eventually reflected in resources with visible convergence status.
High-level architecture
- The control plane stores desired state and emits versioned reconciliation work; controllers converge regional resources. The data plane consumes signed or versioned snapshots into local read-optimized stores and never makes a synchronous control-plane call on its hot path.
Data model and flow
- A mutation writes desired state and an outbox event atomically. Reconcilers perform idempotent steps, publish observed state and conditions, and only then produce a validated serving snapshot for atomic data-plane activation.
Consistency and transaction boundaries
- Control-plane APIs offer optimistic concurrency on resource versions, while convergence is asynchronous. Data planes expose the applied version; operations that require fresher policy can fail narrowly when a minimum version is unmet.
Failure modes and recovery
- Use independent databases, credentials, capacity pools, and deployment rings so a control-plane query cannot starve serving. Controllers use bounded retries and work queues; duplicate reconciliation is safe, and invalid snapshots never replace last-known-good state.
Security and privacy
- Separate identities and permissions for intent writers, reconcilers, and data-plane readers. Sign sensitive snapshots, redact tenant secrets from status, and make break-glass control-plane actions short-lived and audited.
Observability and SLOs
- Measure desired-versus-observed drift, reconcile age, failed conditions, snapshot distribution lag, applied version, and independent SLOs for management and serving. Give users a status model rather than returning success before convergence invisibly.
Capacity and cost
- The boundary duplicates some data and infrastructure, but protects the revenue path from bursty management work. Use compact snapshots and regional fan-out rather than per-instance control-plane reads.
Alternatives and trade-offs
- A shared store is simpler at small scale but couples saturation and schema changes. Full plane separation adds eventual consistency and reconciliation machinery; it is justified when serving availability and management workloads differ materially.
Evolution and migration
- First materialize the current serving reads and remove synchronous dependencies, then introduce desired and observed state, and finally split infrastructure. Shadow snapshots and compare decisions before switching the hot path.
What Staff and Principal candidates should emphasize
- Principal candidates articulate authority: the control plane owns desired state, controllers own convergence, and the data plane owns fast serving from a known version. They also make partial completion and drift visible to users.
Decision trade-offs
Serving dependency
Option A
Synchronous control-plane lookup
Option B
Versioned local data-plane snapshot
Recommendation:Use local snapshots for the hot path; synchronous authority checks belong only to rare operations whose risk exceeds availability needs.
API completion
Option A
Block until every resource converges
Option B
Return accepted with observable conditions
Recommendation:Return durable acceptance and expose convergence state; block only for small, bounded operations with a meaningful synchronous guarantee.
Follow-up interview questions
- 01How does a user know a successful policy update is not yet serving?
- 02When should a data plane refuse to serve stale policy?
- 03How do you roll back a desired-state schema that reconcilers already consumed?
- 04Which resources must be physically isolated versus logically isolated?
Common weak answers and mistakes
- 01Renaming services control plane and data plane while leaving the same database and pools.
- 02Making the data plane poll desired state on every request.
- 03Returning success without an observable convergence condition or applied version.
- 04Assuming reconcilers run exactly once and omitting idempotency.
Interviewer evaluation rubric
Draws two boxes but does not define authority, version flow, drift, or independent failure boundaries.
Removes synchronous management dependencies and introduces desired state, reconciliation, and cached serving data.
Defines versioned snapshots, optimistic APIs, conditions, isolated resources, bounded controllers, and stale-policy behavior.
Provides a low-risk extraction sequence and a precise authority model that remains understandable during partial convergence and rollback.