Coordinate safety-critical global configuration through network partitions

• Classify configuration by consequence and staleness tolerance: performance tuning, traffic movement, security revocation, and destructive enablement are not equivalent. Ask who may publish, how quickly each class must propagate, and what a disconnected data plane should do.

• Assume 60k clients across twenty regions, hundreds of updates per minute, and seconds-level propagation for emergency policy. Design for reconnect storms and many readers, not high write throughput.

• Request serving must continue from the last known valid snapshot, every update must be authenticated and auditable, and clients must reject corrupt or incompatible configuration. High-risk changes require stronger approval and freshness rules.

• Use a quorum-backed authoritative store, an append-only signed version log, regional distribution relays, and client-side cached snapshots. Clients evaluate locally and report applied version; a separate emergency channel can carry narrowly scoped revocations.

• Publish an immutable version with schema version, predecessor, author, approvals, validity window, and checksum. Relays stream deltas but clients can always fetch and verify a complete snapshot before an atomic local swap.

• The control plane is strongly consistent for publication order, while data planes are deliberately eventually consistent. Safety rules define minimum accepted version or expiry; a client beyond that boundary fails only the affected operation, not the whole process.

• Keep last-known-good state, cap relay retry storms, and use randomized reconnect. Bad configuration is contained with staged rollout, client validation, automatic rollback on health regression, and a break-glass revocation path independent of the primary publisher.

• Sign versions, authorize by policy scope, require multi-party approval for sensitive classes, and make rollback a new audited version rather than history mutation. Protect the signing path with hardware-backed keys and explicit rotation.

• Measure publish-to-apply latency percentiles, version skew, rejected configs, expired safety policy, relay backlog, and health changes by cohort. Operators need a global map of which instances are on which version.

• Distribute snapshots through regional relays or object storage and keep evaluation local to avoid a per-request control-plane dependency. Size for mass reconnect and retain enough history for forensic replay without keeping every delta forever.

• Push gives fast propagation but reconnect complexity; pull is robust but bounded by polling delay. Signed immutable versions and atomic swaps cost more storage than mutable rows but make rollback and forensics far safer.

• Introduce version envelopes and client reporting before changing transport. Run old and new evaluators in shadow, then migrate low-risk policy classes first; publish a deprecation horizon for clients that cannot enforce freshness.

• The important insight is that availability belongs to the data plane, while authority belongs to the control plane. Strong answers define risk classes, safe stale behavior, atomic application, and how an operator proves global convergence.

Interviewer evaluation rubric