Data Science Interview Knowledge Hub

Report the median when the distribution is skewed or has outliers — income, latency, session length. The mean is pulled toward the tail, so it stops describing a typical value; the median (the 50th percentile) is robust to extremes. Use the mean when the distribution is roughly symmetric and you need a quantity that sums (e.g. total revenue = mean × count).

Variance is the average squared deviation from the mean; standard deviation is its square root. We usually report the standard deviation because it's in the same units as the data (dollars, ms), so it's interpretable, and it's what shows up in the normal distribution's 68/95/99.7 rule. Variance is the natural quantity to work with algebraically because variances of independent variables add.

Both measure how two variables move together, but covariance is unbounded and unit-dependent, so its magnitude is uninterpretable. Correlation is covariance normalized by the two standard deviations, giving a unitless value in [-1, 1] you can actually compare across pairs. Use correlation to judge strength of a linear relationship; use covariance mainly as an intermediate quantity (e.g. in PCA or portfolio math).

The CLT says the distribution of the sample mean approaches a normal distribution as the sample size grows, regardless of the population's shape — provided the variance is finite. It's why we can build confidence intervals and run t-tests on means even when the underlying data (clicks, revenue) is nowhere near normal: it's the mean's sampling distribution that's normal, not the data.

It's a range produced by a procedure that, over many repeated samples, captures the true parameter 95% of the time. For your one interval, the parameter is either in it or not — the 95% describes the method's long-run reliability, not the probability the truth is in this particular interval. Practically: a 95% CI is the set of parameter values your data wouldn't reject at the 5% level.

Standard deviation measures spread in the data; standard error measures spread in an estimate. The standard error of the mean is std / √n — it shrinks as you collect more data, because your estimate of the mean gets more precise, even though the data's own spread doesn't change. Confidence intervals and test statistics are built from standard errors, not standard deviations.

Bias is how far an estimator's expected value sits from the true parameter; variance is how much it bounces around from sample to sample. A good estimator trades them off. The sample mean is unbiased; dividing sample variance by n (instead of n-1) is biased low. Sometimes a slightly biased estimator with much lower variance is preferable — the mean-squared error (bias² + variance) is what you actually minimize.

Prefer simple random sampling for an unbiased estimate; use stratified sampling when subgroups matter or vary a lot (sample within each segment, then combine) to cut variance and guarantee coverage. The biases to name: selection bias (who's eligible to be sampled), survivorship bias (only observing what survived), and self-selection (opt-in surveys). The fix is to define the frame first — who could be sampled — and check it matches the population you want to describe.

Bayes updates a prior with evidence: P(disease | positive) = P(positive | disease)·P(disease) / P(positive). The trap is the base rate. Even a 99%-accurate test for a disease that affects 1 in 1,000 people produces mostly false positives, because the 999 healthy people generate ~10 false positives for every 1 true positive — so a positive result means roughly a 9% chance of disease, not 99%.

Expectation is linear always: E[aX + bY] = aE[X] + bE[Y], even when X and Y are dependent. Variance is not: Var(aX) = a²Var(X), and Var(X + Y) = Var(X) + Var(Y) only when X and Y are independent (otherwise add 2·Cov(X,Y)). Knowing linearity of expectation lets you solve many 'expected number of…' problems without touching the full distribution.

Bernoulli/Binomial for a fixed number of yes/no trials (conversion, click). Poisson for counts of rare events in a fixed window (arrivals, errors per minute). Exponential for the time between such events. Normal for sums/averages of many small effects (via the CLT). Log-normal for multiplicative, right-skewed quantities (income, session length). Naming the generative story — how the data is produced — is what picks the distribution.

No. Independence means knowing one variable tells you nothing about the other — a strong, all-relationships condition. Uncorrelated only means zero linear association. Independence implies uncorrelated, but not the reverse: variables can be perfectly dependent yet uncorrelated (e.g. Y = X² with X symmetric around 0 has correlation 0). The exception is the joint-normal case, where uncorrelated does imply independent.

The Law of Large Numbers says the sample mean converges to the true mean as n grows — it's about where the average lands. The CLT says how the sample mean fluctuates around that true mean — its distribution is approximately normal with standard error std/√n. LLN gives you consistency (you'll get the right answer eventually); CLT gives you the error bars (how wrong you might be right now).

A p-value is the probability of observing data at least as extreme as yours, assuming the null hypothesis is true. Small p means your data would be surprising if there were truly no effect, so you reject the null. It is not the probability the null is true, and not the probability your result happened by chance — both are common misreadings.

A Type I error (false positive, rate α) is rejecting a true null — shipping a change that does nothing. A Type II error (false negative, rate β) is failing to detect a real effect. Power = 1 − β is the probability of detecting an effect that truly exists. You buy power with a bigger sample, a larger true effect, lower variance, or a higher α — the four levers of experiment sizing.

It means the effect is real but too small to matter. With a large enough sample, even a trivial difference becomes statistically significant, because significance measures 'distinguishable from zero,' not 'big enough to act on.' The decision should hinge on the effect size and its confidence interval versus a pre-agreed minimum meaningful effect — not on the p-value crossing 0.05.

A two-tailed test asks 'is there any difference?'; a one-tailed test asks 'is it better in this specific direction?'. Two-tailed is the safe default for experiments, because a change can help or hurt and you want to catch harm. A one-tailed test has more power for the same α but only by ignoring the other direction — rarely worth it in product, where a negative surprise is exactly what you must not miss.

Each test at α = 0.05 has a 5% false-positive rate, so across 20 independent metrics you'd expect about one 'significant' result by pure chance even if nothing changed. Control it by correcting for multiplicity: Bonferroni (divide α by the number of tests — simple, conservative) or Benjamini-Hochberg (control the false discovery rate — more powerful when you have many metrics). Better yet, pre-register one primary metric and treat the rest as secondary/exploratory.

Reach for a non-parametric test (Mann-Whitney, permutation, bootstrap) when the assumptions behind a t-test are shaky — heavy skew, extreme outliers, tiny samples, or ordinal data — because those break the t-test's reliance on approximate normality of the mean. The trade-off is that non-parametric tests often answer a slightly different question (about ranks/medians, not means) and can be less powerful when the parametric assumptions actually hold.

Orient before analyzing. Establish the grain (what one row represents) and the size, then check each column's type, range, and missingness. Look at distributions for skew and outliers, and at a few key relationships. Crucially, sanity-check against reality — do the totals match a known number, are timestamps sane, are IDs unique — because most 'insights' from a dirty dataset are artifacts. Only once you trust the data do you chase the actual question.

Start with why it's missing, because the mechanism dictates the fix. Missing completely at random can be dropped or simply imputed with low risk. Missing at random (missingness depends on observed variables) can be imputed conditionally. Missing not at random — where the missingness depends on the unseen value itself (high earners hiding income) — biases any naive imputation, and often the fact of missingness is itself signal worth encoding as a flag. Match the method to the mechanism, not to convenience.

Not reflexively — decide by cause. An outlier that's a data error (a negative age, a $10^9 order) should be fixed or removed; an outlier that's a real extreme (a whale customer, a viral spike) is often the most important data and must stay. Choose the treatment for the goal: robust statistics or winsorizing for a summary, keep-and-model for prediction, investigate individually for anomaly detection. Never delete inconvenient points to make a result look cleaner.

Leakage is when information that wouldn't be available at prediction time sneaks into training, producing amazing offline metrics that collapse in production. Classic forms: a feature that's a proxy for the label (a 'was_refunded' flag when predicting fraud), fitting a scaler or imputer on the full dataset before splitting, or using future data in a time-series split. Catch it by asking of every feature 'would I truly know this at prediction time?', splitting before any fitting, and treating a suspiciously perfect model as a leak until proven otherwise.

Bisect the pipeline and compare distributions before vs after the change. For each feature, compare summary stats and distributions across the two versions (a KS test or population-stability index flags shifts), and check missingness and cardinality — a broken join or an encoding change shows up as a spike in nulls or a new category. Separate training-serving skew (features computed differently in prod than in training) from genuine data drift (the world changed). Reproduce on a fixed sample to confirm the change is the cause, not a coincidence.

Prefer the simple model when the marginal accuracy doesn't pay for its costs: when you need interpretability (regulated decisions, stakeholder trust), when data or latency is constrained, when the complex model is fragile to drift, or when the maintenance burden outweighs a small lift. A logistic regression you can explain and monitor often beats a gradient-boosted forest that's 1% better but a black box nobody trusts. The right model is a business decision, not a leaderboard result.

Let the cost of each error decide. Optimize recall when a miss is expensive and a false alarm is cheap — disease screening, fraud you can afford to review. Optimize precision when acting on a positive is costly or annoying — blocking a legitimate transaction, spamming users with alerts. Then set the threshold, not just the model, to the operating point that balances those costs at your actual base rate. The metric follows the business, not the other way round.

Offline and online can diverge for several reasons: the offline metric isn't aligned with the KPI (better AUC that doesn't change user behavior), the offline data is stale or biased by the old model's own logging (feedback loops), the improvement lands where it doesn't matter (already-easy cases), or the change shifts the system in ways offline data can't see (users adapt, other components react). The resolution is an online experiment — offline evaluation ranks candidates, but only an A/B test measures product impact.

Start from the decision, not the test. State one primary metric tied to the goal (checkout conversion), pick guardrails you must not harm (revenue, latency, refunds), and define the minimum effect worth shipping. From that effect, α, and power, compute the sample size and therefore the duration. Randomize at the right unit (user, not request), split traffic, and only read the result at the pre-committed stopping point.

Randomize at the level at which the treatment is experienced and at which you want to generalize — usually the user (or account), not the page-view or request. If a user can see both variants across sessions, the treatment contaminates the control and the comparison is meaningless. Coarser units (user, cluster) reduce contamination but increase variance, because you have fewer independent units; that trade-off drives your sample size.

Four inputs set the sample size: the baseline rate, the minimum detectable effect (the smallest lift worth catching), α (false-positive rate), and power (1 − β). Smaller effects and higher power need quadratically more users — halving the MDE roughly quadruples the sample. Convert the required users into duration using your daily eligible traffic, and always run at least one or two full weekly cycles to cover day-of-week effects.

A fixed-horizon test controls the false-positive rate only if you look once, at the planned end. If you check repeatedly and stop as soon as p < 0.05, you give yourself many chances to cross the line by luck, and the real false-positive rate balloons well past 5%. The fixes are to commit to the sample size and look once, or to use methods designed for continuous monitoring — sequential tests (e.g. mSPRT), group-sequential boundaries (O'Brien-Fleming), or always-valid confidence sequences.

That pattern is usually a novelty effect: users click the new thing because it's new, not because it's better, and the lift decays as novelty wears off. Its opposite, a primacy effect, is when users initially resist a change and the true benefit only appears later. Handle both by running long enough for behavior to stabilize, and by analyzing new users separately from tenured ones — new users have no 'old' to react to, so their behavior is a cleaner read of the steady state.

Standard A/B tests assume one user's treatment doesn't affect another's outcome (SUTVA). In marketplaces, social graphs, or shared-resource systems, that breaks: treating a seller changes buyers in control, ride-pricing changes the pool for everyone, a social feature leaks across friends. The measured effect is then biased — often understated — because control is contaminated by treatment. You mitigate with cluster randomization (assign whole graphs/regions), switchback tests (toggle the whole system on and off over time), or ego-cluster designs.

Guardrail metrics are the things a change must not harm even while it improves the primary metric — latency, crash rate, revenue, unsubscribes, support tickets. They protect against local wins that cause global damage: a change can lift clicks while tanking revenue or page speed. You watch guardrails for regressions, and a significant harm to one can block a launch that 'won' on the primary metric.

Pick one metric that is sensitive (moves when the feature works), aligned with the long-term goal, and hard to game. Prefer a metric close to user value over a vanity proxy. Where the true goal is long-term (retention, LTV) but too slow to measure in a two-week test, choose a validated short-term surrogate that's known to correlate with it — and say explicitly that it's a surrogate. One primary metric keeps the decision unambiguous; everything else is secondary.

Reduce the metric's variance, since required sample scales with variance. The standard technique is CUPED: use each user's pre-experiment value of the metric as a covariate to subtract out the variation that has nothing to do with the treatment, which can cut variance 30-50% and shorten the test proportionally. Other levers: a less noisy metric, stratification/regression adjustment on pre-experiment covariates, winsorizing extreme outliers, or triggered analysis (only include users who could be affected).

The catch with ratio metrics (clicks ÷ sessions, where a user has many sessions) is that the numerator and denominator are correlated and the unit of analysis (session) is nested within the randomization unit (user), so naive variance is wrong and too small. Use the delta method to get the variance of the ratio, or bootstrap at the user level, so the standard error reflects that users — not sessions — are the independent units.

That's a possible sample-ratio mismatch (SRM): the observed split differs from the intended one by more than chance allows (a simple chi-square test flags it). SRM means the randomization or logging is broken — a redirect that drops users, a bot filter applied to one arm, differential opt-outs — and it invalidates the whole result, because the groups are no longer comparable. Don't interpret the metrics until the SRM is explained and fixed.

Trust the process over either intuition. First validate the test itself — SRM, instrumentation, no peeking, the right unit — because a 'clean 2%' from a broken test is nothing. Then check whether the 2% is on the metric that matters and whether guardrails held. Look for heterogeneity: is the lift concentrated in a segment that makes sense, or diffuse and suspicious? Consider novelty by segmenting new vs tenured users. If it survives all that, a real, guardrail-safe 2% usually ships — the team's disbelief is a hypothesis to test, not a veto.

A good north-star metric captures the value users get, moves ahead of revenue, and is hard to game. Anchor it to the core action that predicts retention and monetization — 'nights booked' for lodging, 'messages sent' for chat — not a vanity count like signups. Pair it with a small set of input metrics that teams can actually move, and guardrails so nobody games the star at the cost of trust or revenue.

Two metrics disagreeing usually means the mix changed. A 10% DAU jump with falling retention points to an influx of low-intent users — a marketing push or a viral loop bringing people who bounce. Separate new from returning users and look at retention by acquisition cohort: if new-cohort retention is low and it's just diluting the average, the product is fine but acquisition quality dropped. If existing users' retention fell too, that's a real product regression to chase.

Localize before theorizing. Break the funnel into steps and find which transition dropped — add-to-cart → checkout → payment → confirmation — so you're debugging one stage, not the whole flow. Then slice that stage by dimensions that isolate a cause: platform/browser (a bug), geography (a payment provider), new vs returning, and time (a deploy). A drop concentrated in one slice is a bug or outage; a broad, uniform drop is more likely demand or a pricing/UX change.

Plot the fraction of a cohort still active over time since signup, one line per cohort. Health is about the shape, not the level: a curve that declines then flattens onto a plateau means you've found a core group for whom the product sticks — that plateau height is your long-term retained fraction. A curve that keeps sliding toward zero has no product-market fit yet. Compare cohorts to see whether recent changes lifted the plateau.

Lagging metrics (revenue, retention, LTV) tell you the outcome but arrive too late to steer by. Leading metrics (activation, early engagement, a validated surrogate) move first and predict the lagging ones, so teams optimize them day to day. The craft is validating that the leading metric actually causes the lagging one — otherwise you're steering by a dial connected to nothing.

That's Simpson's paradox: the aggregate is a mix, and the mix of segments shifted. Each segment can improve while the overall number falls because a larger share of traffic moved into a lower-performing segment. The resolution is to decide which view answers the question — segment-level is usually the causal truth for 'did the change help?', while the aggregate reflects composition — and to report the mix shift explicitly rather than letting the paradox mislead.

Segment along dimensions with a prior reason to differ — platform, tenure, geography, acquisition channel, power vs casual users — chosen before looking, so you're testing hypotheses rather than mining for any split that shows an effect. Treat post-hoc segment discoveries as exploratory hypotheses to confirm, not conclusions, and correct for the many comparisons you're implicitly making. A segment effect you can't explain mechanistically is a lead, not a finding.

Correlation means two things move together; causation means changing one changes the other. You can only claim causation cleanly from a randomized experiment, because randomization balances all other factors — known and unknown — so the only systematic difference between groups is the treatment. Without randomization you're stuck with confounding: some third variable drives both, and the observed association may vanish or reverse once you account for it.

A confounder is a variable that influences both the treatment and the outcome, creating a spurious association between them. It biases any naive comparison because the treated and untreated groups differ on that variable, not just on treatment. You address it by design (randomize, so confounders are balanced) or, in observational data, by adjustment — stratification, regression, matching, or propensity scores — but only for confounders you can measure. Unmeasured confounding is the fundamental limit of observational causal claims.

Lean on quasi-experimental designs that approximate randomization. Difference-in-differences compares the change over time in a treated group vs a control group, cancelling out fixed differences (assuming parallel trends). A regression-discontinuity design uses an arbitrary threshold (a cutoff score) as a local randomizer. Instrumental variables exploit a source of variation that affects treatment but not the outcome directly. Each rests on an assumption you must argue for and, ideally, test.

Selection bias is when the data you analyze isn't representative because of how it was selected. Analyzing only users who completed onboarding to judge onboarding overstates success — the failures selected themselves out. Survivorship bias (studying only surviving accounts) and self-selection (opt-in surveys) are common forms. Guard against it by defining the population and the eligibility frame up front, and by including the units that dropped out rather than conditioning on the outcome.

Translate the statistics into a decision, not a lecture. Instead of 'p = 0.06, fail to reject,' say 'the data leans positive but we can't rule out that it does nothing — the true lift is somewhere between −1% and +4%.' Give the range and what it means for the choice: whether it's safe to ship, needs more data, or the downside is tolerable. Anchor uncertainty to the action the PM has to take, and be honest about what you don't know without freezing the decision.

Distinguish 'no effect' from 'no evidence of an effect,' because they're different and the audience will conflate them. Report the confidence interval: a tight interval around zero is real evidence the change does little; a wide interval means the test was underpowered and you simply can't tell. Then make a recommendation — ship anyway for other reasons, iterate, or invest in a larger test — so a null result still drives a decision instead of being filed as a failure.

Lead with the shared goal, not a contradiction. Acknowledge the metric moved, then show the fuller picture with evidence — the segment or guardrail that tells a different story, or the composition shift behind the number — and propose the measure that better reflects the actual objective. The aim is to protect the decision, not win the argument, so frame it as 'here's a risk this number hides' rather than 'you're wrong.' Bring the data, and offer the better metric to rally around.

Don't average them into false confidence or hide one. Reconcile first — usually the conflict comes from different populations, time windows, or definitions, and naming that difference often resolves it. If it genuinely can't be resolved, present both with their assumptions and your assessment of which is more credible and why, plus what evidence would settle it. Leadership makes better calls with an honest 'here's the disagreement and my read' than with a tidy answer that papers over real uncertainty.