Interview prompt
Problem context
Skills being evaluated
Use the sequence below to surface constraints, choose boundaries, test failure behavior, and defend trade-offs. Concrete numbers are interview assumptions, not claims about a real production system.
Clarify the decision
- Separate tenfold peak throughput, stored data, tenant count, and geographic reach because each stresses a different boundary. Ask which customer journeys fund the business and what downtime, correctness, and delivery-date constraints are contractual.
Establish scale assumptions
- Use the current 40k requests/second as a measured baseline, then model 400k peak with a two-times safety margin. Break the request mix into read, write, background, and tenant percentiles; averages hide the one table, key, or dependency that actually saturates.
Functional and non-functional requirements
- Keep current availability and data guarantees while increasing headroom in stages. Require every scaling change to be observable, reversible, and deployable independently so the program can continue shipping customer value.
High-level architecture
- Retain the system of record and introduce pressure-relief boundaries: CDN and read caches for immutable reads, asynchronous queues for non-interactive work, connection pooling, read replicas, and stateless horizontal scaling. Split a service only where profiling shows a distinct scaling, ownership, or failure boundary.
An evolutionary pressure-relief architecture that keeps the existing system of record while moving independently scalable work behind measured boundaries.
Data model and flow
- Map the top three journeys end to end and assign a latency and resource budget to each hop. For new async paths, persist intent with an outbox, carry an idempotency key, and expose reconciliation state so delayed work is visible rather than silently inconsistent.
Consistency and transaction boundaries
- Preserve strong transactions around existing business invariants; do not distribute them merely to gain throughput. Move read models and secondary effects to eventual consistency only after product owners agree on staleness and repair semantics.
Failure modes and recovery
- Add admission control, bounded queues, per-dependency timeouts, retry budgets, and tenant bulkheads before increasing concurrency. Test each stage by exhausting a dependency, draining a region, and rolling back a mixed-version deployment.
Security and privacy
- Keep authorization at every new boundary and make cache keys tenant-aware. Scaling work must not copy regulated fields into new stores without retention, encryption, residency, and deletion ownership.
Observability and SLOs
- Define service-level indicators for the revenue journeys, then add saturation, queue age, pool utilization, and per-tenant heat maps. A capacity milestone is complete only when a load test and production canary demonstrate headroom against an agreed SLO.
Capacity and cost
- Build a demand model from current units per request and validate it with replay or synthetic load. Reserve predictable base capacity, autoscale burstable stateless tiers, and track cost per successful business transaction so performance fixes do not create unbounded spend.
Alternatives and trade-offs
- A rewrite may improve conceptual integrity but defers learning and creates a single cutover cliff. An evolutionary program leaves some duplication temporarily, yet produces evidence after every step and lets leadership stop when sufficient headroom exists.
Evolution and migration
- Sequence work as instrumentation, cheap configuration and query fixes, horizontal headroom, async decoupling, then selective data partitioning. Use shadow reads, dual-run comparisons, and feature flags for risky paths; publish explicit exit criteria for temporary bridges.
What Staff and Principal candidates should emphasize
- A strong candidate turns “10×” into a portfolio of verified constraints, not a diagram containing more boxes. They explain organizational ownership, rollout order, decision checkpoints, and which changes they deliberately refuse to make until evidence demands them.
Decision trade-offs
Evolution path
Option A
Incremental pressure-relief changes
Option B
Greenfield rewrite and flag-day cutover
Recommendation:Choose incremental changes by default; authorize a rewrite only for a bounded domain with a measurable strangler path and an owner-funded migration.
Data scale
Option A
Optimize and vertically scale the current database
Option B
Partition data immediately
Recommendation:Buy simple headroom first, then partition only after access patterns, hot keys, operational tooling, and shard-move procedures are proven.
Follow-up interview questions
- 01What evidence would make you stop the scaling program before completing every proposed change?
- 02How would you isolate a tenant whose workload grows one hundred times faster than the median?
- 03Which change would you make first if the database is at 70% CPU but connection waits dominate?
- 04How do you keep eight teams aligned on temporary compatibility layers and their removal dates?
Common weak answers and mistakes
- 01Treating 10× as one scalar instead of decomposing traffic, data, tenants, and regions.
- 02Starting with microservices or sharding before measuring the limiting resource.
- 03Presenting a final architecture without a safe sequence from the system that exists today.
- 04Ignoring delivery capacity and assuming every team can migrate at the same time.
Interviewer evaluation rubric
Names generic scaling components but cannot connect them to a measured constraint, invariant, or safe rollout.
Identifies likely bottlenecks, proposes capacity tests, and preserves the current transactional core while adding headroom.
Sequences reversible changes with SLO gates, tenant-aware failure isolation, cost units, and explicit migration exit criteria.
Frames the work as an evidence-driven portfolio, includes organization and ownership constraints, and knows when not to redesign.