Interview prompt
Problem context
Skills being evaluated
Use the sequence below to surface constraints, choose boundaries, test failure behavior, and defend trade-offs. Concrete numbers are interview assumptions, not claims about a real production system.
Clarify the decision
- Distinguish rotating root, wrapping, tenant, and data keys; only some require payload re-encryption. Define compromise scope, cryptoperiod, offline recovery, and acceptable cached-key lifetime.
Establish scale assumptions
- Estimate objects, key lookups, tenants, regions, rewrap throughput, and the window to meet compliance. Rewrapping compact data keys is far cheaper than rewriting large ciphertext.
Functional and non-functional requirements
- Support multiple key versions, rotate online, prove adoption, revoke compromised material, preserve least privilege, and retain a tested recovery path. Readers must never guess key version.
High-level architecture
- Use envelope encryption with versioned ciphertext metadata, per-tenant data-encryption keys, regional key-encryption keys in HSM-backed services, and a rotation controller. Cache unwrapped keys briefly in protected memory.
Data model and flow
- New writes use the current approved version; reads select exact metadata version and may lazily rewrap. A rate-limited bulk worker closes the long tail with conditional metadata updates and checkpoints.
Consistency and transaction boundaries
- Key metadata update is atomic with the new wrapped key reference; old versions remain decryptable until coverage and rollback gates pass. Revocation is a policy state, not deletion before rewrap proof.
Failure modes and recovery
- If rotation workers fail, old and new versions remain readable. If the key service is down, bounded caches serve low-risk reads until expiry; writes or sensitive operations fail narrowly.
Security and privacy
- Separate key administration, rotation, application use, and audit roles; never log plaintext keys. Emergency rotation prioritizes exposed scopes and may accept availability loss when compromise risk is higher.
Observability and SLOs
- Track coverage by tenant and version, decrypt errors, stale key use, cache age, HSM saturation, conditional conflicts, and privileged access. Periodically restore encrypted backups with historical keys.
Capacity and cost
- Lazy rewrap reduces spikes but leaves long-tail exposure; bulk rewrap provides deadlines at controlled cost. Reserve key-service capacity for emergency operations and reconnect storms.
Alternatives and trade-offs
- Immediate revocation contains compromise fastest but can make data unreadable; overlap preserves availability but extends exposure. Choose based on compromise evidence and isolate affected tenants or regions.
Evolution and migration
- Add explicit key version metadata and multi-version readers, then rotate low-risk wrapping keys, validate recovery, and automate. Test root recovery and emergency tenant-specific rotation separately.
What Staff and Principal candidates should emphasize
- Staff candidates distinguish rewrap from re-encrypt and define the exact moment old material may be revoked. They include backup restores, key-service outage, and emergency risk decisions.
Decision trade-offs
Rotation work
Option A
Re-encrypt every payload
Option B
Rewrap envelope data keys
Recommendation:Rewrap keys for routine hierarchy rotation; re-encrypt payload only when the data key or algorithm itself is compromised or retired.
Migration mode
Option A
Lazy rewrap on read
Option B
Rate-limited bulk rewrap
Recommendation:Combine lazy acceleration with bulk closure so rarely read objects still meet the compliance or compromise deadline.
Follow-up interview questions
- 01When must ciphertext itself be rewritten?
- 02How do encrypted backups remain recoverable after rotation?
- 03What happens during key-service outage?
- 04How would emergency rotation differ from annual rotation?
Common weak answers and mistakes
- 01Re-encrypting petabytes when only a wrapping key changed.
- 02Deleting the old key before proving every object and backup migrated.
- 03Using unversioned ciphertext and guessing which key decrypts it.
- 04Ignoring key-service capacity and cache behavior during fleet rollout.
Interviewer evaluation rubric
Says rotate keys annually but cannot define versions, overlap, rewrap, revocation, or recovery.
Uses envelope encryption, versioned metadata, multi-key reads, bulk checkpoints, and adoption metrics.
Adds lazy plus bulk rewrap, conditional updates, outage policy, HSM roles, backup restore, and compromise scope.
Makes rotation a risk-based operating capability balancing provable exposure reduction, availability, tenancy, and long-lived recovery.