Interview prompt
Problem context
Skills being evaluated
Use the sequence below to surface constraints, choose boundaries, test failure behavior, and defend trade-offs. Concrete numbers are interview assumptions, not claims about a real production system.
Clarify the decision
- Define business recovery tiers, corruption and compromise scenarios, maximum data loss, and clean-room requirements. Inventory data, configuration, keys, identities, binaries, external integrations, and people needed to restore.
Establish scale assumptions
- Measure backup creation lag, transfer and restore throughput, verification time, and dependency order. RTO includes provisioning, key recovery, replay, validation, traffic ramp, and backlog—not just database restore.
Functional and non-functional requirements
- Maintain immutable independently controlled recovery points, prove restoration regularly, detect corruption, and recover critical journeys without trusting compromised production control planes.
High-level architecture
- Use versioned full and incremental backups with checksums, cross-account immutable storage, separately escrowed keys, infrastructure and configuration artifacts, and an isolated recovery environment. A catalog maps recovery tiers and dependency order.
Data model and flow
- Record backup manifests with source positions, schema and key versions, object counts, and integrity roots. Restore to a quarantined environment, replay logs to a chosen point, validate invariants, then open traffic by journey.
Consistency and transaction boundaries
- Coordinate recovery points across databases, object stores, and event logs using manifest watermarks. Where atomic snapshots are impossible, document reconciliation and compensation needed after a point-in-time restore.
Failure modes and recovery
- Assume the newest backup may be corrupt or compromised; retain diverse points and test older generations. Recovery credentials and communication channels must work without production identity or DNS.
Security and privacy
- Separate backup write from delete authority, require multi-party break-glass access, scan restores before connection, and rotate credentials after compromise. Test that legal deletion and retention policies survive the backup design.
Observability and SLOs
- Track backup age, completion, integrity, restore duration by phase, invariant failures, and percentage of dependencies exercised. Publish last proven RPO and RTO, not merely configured targets.
Capacity and cost
- Tier retention and use incremental chains carefully because very long chains slow recovery. Reserve or pre-negotiate restore compute and bandwidth; a cheap backup that cannot restore within the objective is false economy.
Alternatives and trade-offs
- Frequent full backups simplify restore but cost more; incremental backups reduce storage while increasing chain risk. Mix periodic full, incremental, and log replay according to measured restore time.
Evolution and migration
- Begin with critical journey restore exercises, close missing artifacts, automate manifests and validation, then expand scope. Run surprise clean-room exercises and require teams to remediate findings as reliability work.
What Staff and Principal candidates should emphasize
- Principal answers treat recovery as a repeatedly demonstrated capability across data and control dependencies. They distinguish cyber recovery from regional failover and make business restoration order explicit.
Decision trade-offs
Backup topology
Option A
Production-account backups with shared administration
Option B
Immutable cross-account recovery vault
Recommendation:Use an independently administered immutable vault so compromised production credentials cannot erase the recovery path.
Restore strategy
Option A
Newest point automatically
Option B
Select a verified clean point with staged replay
Recommendation:Choose the newest point proven clean, then replay under inspection; freshness is secondary to avoiding reintroducing corruption.
Follow-up interview questions
- 01How do you recover if the key-management service is also unavailable or compromised?
- 02How do you choose a clean point during slow corruption?
- 03What belongs in a recovery manifest besides database files?
- 04How would you test recovery without risking production data exposure?
Common weak answers and mistakes
- 01Equating successful backup jobs with tested recoverability.
- 02Restoring data but omitting keys, configuration, identity, and external dependencies.
- 03Keeping delete authority in the same compromised account as production.
- 04Reporting configured RTO rather than the last measured end-to-end exercise.
Interviewer evaluation rubric
Adds more backups but cannot describe isolated restore, dependency order, integrity proof, or compromised-control-plane recovery.
Uses immutable backups, escrowed keys, manifests, clean restore environments, and scheduled end-to-end exercises.
Coordinates cross-store points, verifies business invariants, plans cyber isolation, reserves capacity, and publishes proven objectives.
Builds a risk-tiered organizational recovery program whose evidence changes public promises, investment, and architectural priorities.