Interview prompt
Problem context
A shared analytics service has 8,000 tenants. One large tenant runs bursty queries that expand into thousands of subtasks, consuming worker memory and delaying all queues. Enterprise contracts prohibit long outages, but the business cannot afford fully dedicated infrastructure for every customer.
Skills being evaluated
fair schedulingresource isolationnoisy-neighbor controltenant economics
A reasoning guide, not a memorized architecture
Use the sequence below to surface constraints, choose boundaries, test failure behavior, and defend trade-offs. Concrete numbers are interview assumptions, not claims about a real production system.
01
Clarify the decision
- Clarify whether fairness means equal capacity, plan-weighted shares, or protected minimums, and identify which resources saturate first. Separate legitimate paid bursts from abuse and decide which workloads may queue, degrade, or be rejected.
02
Establish scale assumptions
- Model tenant distributions, not a median: assume the largest tenant is 500 times the smallest and may submit 20k subtasks in seconds. Measure CPU, memory, I/O, concurrency, and queue-age amplification per parent query.
03
Functional and non-functional requirements
- Guarantee a minimum service floor for every tenant, honor purchased burst limits, and stop one workload from exhausting a shared pool. Operators need a fast quarantine path that preserves evidence and customer-visible status.
04
High-level architecture
- Put hierarchical admission control before fan-out: global safety limit, plan-level budget, tenant token bucket, and per-query expansion cap. Schedule weighted-fair queues into bulkheaded worker pools and offer a dedicated cell only for tenants whose isolation or economics justify it.
05
Data model and flow
- Carry tenant, workload class, parent query, cost estimate, and deadline through every subtask. Reserve budget before expansion, release it idempotently on completion, and aggregate child status so cancellation stops queued and running work.
06
Consistency and transaction boundaries
- Resource reservations need atomic conditional updates, but usage accounting may converge asynchronously with reconciliation. Treat duplicate completion and lost lease events as normal; a sweeper returns abandoned capacity after a fencing-safe timeout.
07
Failure modes and recovery
- Bound every queue and worker concurrency, spill low-priority tasks to a delayed lane, and shed before memory collapse. A tenant circuit breaker can pause new work while existing tasks drain; separate pools keep control and cancellation traffic available.
08
Security and privacy
- Tenant identity must come from authenticated context and survive every async hop. Resource metrics and operator tooling must not expose another tenant’s query text, and dedicated cells still require the same authorization boundary.
09
Observability and SLOs
- Track queue age, admitted versus rejected cost, resource seconds, cancellation latency, and SLO attainment by tenant and plan. Alert on fairness violations and resource saturation, not only total QPS.
10
Capacity and cost
- Price and provision by normalized work units rather than request counts because fan-out varies wildly. Use shared pools for the long tail, warm reserved capacity for premium floors, and dedicated cells where sustained demand covers fragmentation cost.
11
Alternatives and trade-offs
- Strict per-tenant quotas are predictable but waste idle capacity; borrowing raises utilization but needs rapid revocation. Weighted fair scheduling gives policy control, while dedicated cells buy isolation at the cost of stranded resources and operational copies.
12
Evolution and migration
- Start by tagging all work and measuring cost, then enforce soft warnings, hard expansion caps, weighted queues, and finally cell placement. Roll limits out per tenant with an override audit trail and customer-facing quota telemetry.
13
What Staff and Principal candidates should emphasize
- Staff candidates connect scheduling policy to contracts and unit economics. They explain admission before fan-out, bounded failure, cancellation, and how a support engineer safely quarantines one tenant without restarting the fleet.
Decision trade-offs
Isolation model
Option A
Shared weighted-fair pools
Option B
Dedicated tenant cells
Recommendation:Keep the long tail shared and graduate tenants to cells when regulatory isolation, sustained load, or contract value pays for fragmentation.
Idle capacity
Option A
Hard reservations with no borrowing
Option B
Borrow unused capacity with revocation
Recommendation:Allow bounded borrowing for utilization, but preserve protected floors and revoke before shared saturation becomes a customer incident.
Follow-up interview questions
- 01How do you estimate work before a query expands into subtasks?
- 02What happens when accounting events are duplicated or lost?
- 03How does cancellation preempt already running child tasks?
- 04When should a tenant move from shared capacity to a dedicated cell?
Common weak answers and mistakes
- 01Rate limiting request count while ignoring wildly different query cost.
- 02Applying limits after fan-out has already consumed memory and queue slots.
- 03Using one unbounded queue and calling priority labels isolation.
- 04Ignoring the commercial policy behind fairness and dedicated capacity.
Interviewer evaluation rubric
Weak
Adds more workers or a simple rate limit without modeling per-tenant work expansion and shared resource collapse.
Solid
Introduces tenant quotas, bounded queues, and basic resource tagging with a credible degradation path.
Strong
Designs hierarchical admission, fair scheduling, cancellation, bulkheads, reconciliation, and plan-aware capacity floors.
Exceptional
Links isolation tiers to contracts and unit economics, supplies migration criteria, and preserves an operable control plane during overload.