Interview prompt
Problem context
Skills being evaluated
Use the sequence below to surface constraints, choose boundaries, test failure behavior, and defend trade-offs. Concrete numbers are interview assumptions, not claims about a real production system.
Clarify the decision
- Define quota dimensions, billing versus safety limits, acceptable overshoot, burst policy, reserved floors, and behavior during regional partition. Separate admission estimates from final usage accounting.
Establish scale assumptions
- Model millions of principals, six-region concurrency, tokens or compute units per request, and hot enterprise keys. Quantify worst-case overshoot as allocated regional leases plus in-flight work.
Functional and non-functional requirements
- Make admission local and fast, bound global overshoot, protect paid floors, borrow idle capacity, reconcile actual cost, and expose understandable limit state to clients.
High-level architecture
- A global quota control plane allocates time-bounded token leases to regional limiters by customer and class. Regions admit locally, reserve estimated work, adjust on completion, and request or return leases; global admission protects system safety.
Data model and flow
- Requests carry principal, plan, estimated cost, and operation ID. Regional ledgers record reservation and final usage idempotently; aggregated deltas update billing and inform the next lease allocation.
Consistency and transaction boundaries
- Quota enforcement is intentionally approximate within a proven bound, while billing converges from authoritative operation records. Hard safety caps retain non-borrowable global or regional reserves.
Failure modes and recovery
- During control-plane partition, regions spend only valid leases and then degrade by plan policy. Expired leases cannot be reused; reconciliation handles duplicated usage and revokes future allocation rather than denying completed work.
Security and privacy
- Principal and plan come from trusted identity, leases are signed and scoped, and clients cannot choose cost estimates. Cross-tenant usage data and enterprise reservation state remain isolated.
Observability and SLOs
- Track admitted, rejected, reserved, actual and borrowed units; overshoot bound, lease starvation, fairness, utilization, and client retry behavior by plan and region.
Capacity and cost
- Lease granularity trades global chatter against stranded tokens and overshoot. Allocate large predictable enterprise floors and smaller adaptive pools for volatile long-tail demand.
Alternatives and trade-offs
- Strong global counters provide precise limits but sacrifice latency and partition availability. Regional leases admit bounded approximation that can be explained, monitored, and priced.
Evolution and migration
- Start with shadow accounting and compare estimated to actual cost, then enforce soft regional budgets, enterprise reservations, and finally global borrowing. Publish client headers and retry contracts before hard rejection.
What Staff and Principal candidates should emphasize
- Distinguished candidates tie approximation math to contracts, partition policy, and unit economics. They distinguish quota, billing, and system overload even when all use similar counters.
Decision trade-offs
Counter topology
Option A
Synchronous globally consistent decrement
Option B
Regional time-bounded quota leases
Recommendation:Use regional leases for the hot path and reserve synchronous coordination for rare plan changes or absolute safety limits.
Borrowing
Option A
Fixed regional allocations
Option B
Adaptive borrowing from a global pool
Recommendation:Allow measured borrowing with revocable leases while preserving paid floors and a non-borrowable safety reserve.
Follow-up interview questions
- 01What is the mathematical maximum overshoot during a partition?
- 02How do you bill when actual compute exceeds the admission estimate?
- 03Can an idle region lend enterprise reserved capacity?
- 04What response helps clients back off intelligently?
Common weak answers and mistakes
- 01Using one global Redis counter and ignoring inter-region latency and partition availability.
- 02Conflating billable usage with admission estimates.
- 03Allowing unlimited offline regional operation during control-plane outage.
- 04Calling equal request counts fair when work cost varies by orders of magnitude.
Interviewer evaluation rubric
Centralizes counters or uses independent regional limits without an overshoot, reconciliation, or fairness model.
Uses regional token leases, trusted cost estimates, idempotent accounting, expiration, and plan-aware degradation.
Quantifies bounds, separates quota from billing and safety, supports borrowing, hot keys, and client contracts.
Aligns distributed approximation, commercial entitlements, capacity economics, and global outage policy in one defensible design.