Diagnose a p99 regression that every average says does not exist

• Define the slow journey, percentile window, payload and tenant cohorts, rollout versions, and whether eight seconds includes queueing and retries. Check if measurement drops canceled or timed-out requests.

• Model fan-out probability: many individually rare slow calls make a request tail common. Examine concurrency, queue wait, connection acquisition, garbage collection, lock contention, and cold paths.

• Reproduce or capture the slow cohort, preserve exemplars, isolate rollout causality, and fix without moving latency elsewhere. Measurement must include offered load and abandoned work.

• Use end-to-end traces with tail-based sampling, high-resolution histograms, request cohort tags, queue and pool timing, and continuous production-like load. Compare control and rollout cohorts under matched demand.

• Carry request, tenant tier, payload bucket, route, version, fan-out count, and deadline. Record time spent waiting, executing, retrying, and serializing at each boundary.

• Performance fixes cannot weaken business correctness silently; cache or async changes require explicit freshness and failure semantics. Compare result parity during experiments.

• A slow request should respect deadlines and cancel downstream work so tail latency does not become resource leakage. Protect diagnostics during overload and cap high-cardinality labels.

• Use opaque cohort identifiers and scrub trace payloads. Tail sampling must not retain sensitive enterprise inputs simply because they are slow.

• Measure conditional percentiles by cohort, queue, pool, fan-out, retry, GC and dependency; use exemplars linking histograms to traces. Detect coordinated omission in load generation.

• Tail tracing and detailed profiles are expensive, so sample intelligently and enable focused temporary diagnostics. Fix utilization cliffs before buying capacity whose average appears idle.

• Hedging can reduce read tails but adds load and cost; use only for idempotent operations with spare capacity and cancellation. Increasing timeouts usually worsens saturation.

• First add missing wait-time instrumentation, replay representative slow shapes, canary one hypothesis, and compare distribution plus correctness. Keep a regression workload after the incident.

• Staff candidates decompose latency into wait and work, reason about fan-out probabilities, and distrust averages and synthetic tests that stop sending load during pauses.

Interviewer evaluation rubric