Teaching with the labs

“Once n is above about 30, the sampling distribution is normal.”

The threshold is a rule of thumb for mildly skewed, finite-variance populations. Log-normal data at n = 30 is still visibly skewed, and Cauchy data is never normal at any n because the CLT's finite-variance hypothesis fails.

“If I impute the missing values, I have recovered the missing data.”

Imputation is a device for using the rows you kept, not for recreating information you never observed. Mean imputation fabricates a spike at the centre and shrinks the variance; under MNAR, even a perfectly specified conditional model remains biased because the deletion depends on the value that was deleted.

“Collinearity biases my regression coefficients, so I should drop one predictor.”

Least squares stays unbiased at any ρ < 1; only the variance inflates. Dropping a genuinely relevant predictor introduces real omitted-variable bias, trading a variance problem for a bias problem. If prediction is the goal, collinearity may be harmless; if a single coefficient must be interpreted, the fix is better data or a regularized/identifiable target such as the sum.

“PCA finds the intrinsic structure of my data, whatever units I measure in.”

PCA on the covariance matrix is not invariant to rescaling a single variable. Changing metres to millimetres inflates that variable's variance by a factor of a million and drags PC1 onto it. Only decomposition of the correlation matrix (standardizing first) is invariant, which is why it is the default for variables in different units.

“My classifier is 99% accurate, so it works well.”

With 1% prevalence, predicting “negative” for everyone is 99% accurate and useless. Accuracy weights both error types by their frequency, which hides failures on the rare class that usually matters most. Report sensitivity and precision at a stated threshold, plus AUC or average precision for the ranking, and always give the majority-class baseline for comparison.

“The elbow in the inertia curve tells me the right number of clusters.”

Inertia falls monotonically with k by construction, and on data with no clusters at all it still produces a smooth curve with a visible bend. An elbow is a heuristic about diminishing returns, not evidence about the number of groups. Use it alongside stability across resamples, a validation index appropriate to the cluster shape, and — decisively — subject-matter meaning.

“My model has an AUC of 0.85, so its predicted probabilities are reliable.”

AUC only measures whether positives are ranked above negatives; it is invariant to every monotone rescaling of the scores. A model can achieve any AUC while reporting probabilities that are systematically double the truth. Whenever probabilities feed a decision — treatment thresholds, expected loss, resource allocation — you must report calibration (reliability diagram, ECE, Brier, log loss) alongside discrimination, and recalibrate if needed.

“The variables selected at the cross-validated λ are the true predictors.”

Cross-validation optimizes out-of-sample prediction error, and prediction tolerates extra weakly correlated variables far more happily than it tolerates dropping a real one. So the CV-optimal model habitually includes false discoveries, and it also under-reports the size of true effects because of shrinkage. For inferential claims about which variables matter, use methods designed for that target — knockoffs, stability selection, or debiased inference — rather than reading off the LASSO's support.

“If I have plenty of data, a few strange values cannot really matter.”

Sample size does not protect a non-robust estimator. The mean is a linear functional of the data, so one observation at distance d shifts it by d/n — and an adversarial or instrument-error value can have unbounded d. Robustness is a property of the estimator, not of the sample size. Conversely, trimming is not free: a 10% trimmed mean breaks down at exactly 10% contamination.

“Each of my tests uses α = 0.05, so I have a 5% chance of a false conclusion.”

α = 0.05 is a per-test guarantee. Across m₀ independent nulls, the probability of at least one false positive is 1 − 0.95^m₀ — 40% at m₀ = 10 and above 99% at m₀ = 100. Only a multiplicity procedure restores a study-level guarantee, and you must choose which guarantee you want: FWER (no false positives at all) or FDR (a bounded fraction of your reported discoveries).

“The regression line and the first principal component are two names for the line of best fit.”

They minimize different distances. OLS minimizes the squared vertical gaps, so it depends on which variable you call the response — swapping x and y gives a different line. PC1 minimizes perpendicular gaps and is symmetric in the two variables. The gap between them grows with the noise-to-signal ratio, and with large noise OLS visibly under-estimates the slope while PC1 does not.

“My classifier got every training case right, so it will do well on new patients.”

A 1-nearest-neighbour rule achieves zero training error on any dataset, including randomly labelled noise, because each training point is its own nearest neighbour. Training error measures memorization capacity, not generalization. Only an estimate that never uses a case in its own prediction — leave-one-out, k-fold, or a held-out test set — measures the quantity you care about, and even that estimate is bounded below by the Bayes error implied by class overlap.

“Documents are points in space, so Euclidean distance tells me which are alike.”

With bag-of-words counts, vector length encodes document length rather than content. A 60-token clinical note and a 20-token clinical note differ mostly in norm, so Euclidean distance ranks them far apart while cosine — which only reads the angle — ranks them together. Text retrieval therefore compares directions, using L2 normalization or cosine, and re-weights terms by IDF so that words appearing everywhere stop dominating the direction.

“I split the data 70/30, so the number I got is the model's accuracy.”

A single split gives one draw from a sampling distribution that can be very wide when n is small. In this lab, at n = 40 the holdout estimate routinely lands 30–50% above or below the true prediction error, so two competent analysts with identical data and identical models report different accuracies. k-fold averages over k splits and materially reduces that variability, and the estimate should always be reported with its uncertainty rather than as a point value.

“My loss exploded, so the loss surface must have a bad region or a saddle.”

Divergence on this lab's objective happens on a strictly convex quadratic with a unique minimum, purely because η exceeded 2/L. Step size interacts with the largest curvature, and no property of the landscape is needed to explain the failure. Symmetrically, slow progress on an ill-conditioned bowl looks like a plateau but is just the (κ−1)/(κ+1) rate — the fixes are scaling, preconditioning, and momentum, not a different loss function.

“I have 200 measurements, so my standard error is s/√200.”

That formula assumes independence. With AR(1) dependence the variance of the mean is inflated by (1 + φ)/(1 − φ): at φ = 0.85 the 200 observations carry the information of about 16 independent ones, and the honest standard error is roughly 3.5 times the naive one. Confidence intervals and p-values computed as if time-ordered data were a random sample are systematically overconfident — this is one of the most common errors in applied longitudinal work.

“These formulas are algebraically identical, so it does not matter which one I code.”

Algebra says nothing about rounding. The one-pass formula subtracts two large nearly equal quantities, so its relative error grows like ε·(μ/σ)² and it can return a negative variance. The centred and Welford forms compute the same quantity with error near machine epsilon regardless of location. Choosing the numerically stable form is part of the statistics, not an implementation detail.

“As long as the determinant is not zero, the matrix is safe to invert.”

The determinant is a product, so it can stay comfortably away from zero while one singular value is tiny — scale a matrix up and det grows by the square. What controls stability is the ratio σ₁/σ₂: an almost-collapsed ellipse means almost-collinear columns, and solving with such a matrix amplifies noise by κ regardless of what det reports.

“If my accuracy is stuck at 82%, I need a stronger model.”

First compute the floor. If the classes overlap enough that the Bayes error is 18%, then 82% is optimal and every hour spent on a bigger model is wasted; the gain must come from better features that separate the densities. Conversely, an accuracy below the majority-class baseline means the model is worse than a constant. Always locate your result between the baseline and the ceiling before interpreting it.

“This rule has 92% confidence, so buying A strongly predicts buying B.”

Compare it with P(B). If B is in 90% of baskets, a 92% confidence rule is barely distinguishable from independence and its lift is 1.02 — the rule tells you almost nothing you did not know from B's marginal frequency. High confidence is necessary but not sufficient; report lift or leverage alongside it, and remember all three are symmetric associations rather than causal claims.

“The model has AUC 0.85, so it is ready to guide treatment decisions.”

AUC is a threshold-free ranking measure: it says nothing about whether acting on the model at any particular cutoff beats treating everyone or nobody. Under low prevalence a well-ranked model can still sit below the treat-all line for every threshold a clinician would accept. Decision curve analysis makes usefulness explicit, and it demands one thing AUC does not: that you say what a false positive costs relative to a false negative.

“Our pipeline takes six hours on one machine, so on sixty machines it will take six minutes.”

Only if every step parallelises perfectly and coordination is free. Profile first: if 8% of the runtime is serial — reading configuration, writing a single output file, a final sort — the ceiling is 12.5×, so six hours becomes about 29 minutes at best, and any communication cost makes it worse. The honest engineering question is not how many workers to add but which serial fraction to remove.

“Forty patients were lost to follow-up, so we removed them from the survival analysis.”

Those forty were event-free for as long as they were observed, and that is real information about the survival curve up to their last visit. Removing them biases the estimate upward (the remaining sample is enriched for people observed long enough to have events) while counting them as events biases it sharply downward. Kaplan–Meier keeps them in the risk set until their censoring time and drops them afterwards, which is why it is the standard estimator — provided censoring is independent of prognosis.

“Convolutional layers work by smoothing the image, so they should always help.”

Smoothing is one kernel among infinitely many, and it is the one a network rarely learns. A convolution is a local template match whose nine weights are learned; zero-sum kernels ignore brightness entirely and respond only to change. The layer helps only when its built-in assumptions hold — that nearby inputs are related and absolute position does not matter. On tabular features, where column order is arbitrary, those assumptions are false and a dense layer is the right choice.

“Randomness means a computation can't be reproduced, and any seed is as good as any other.”

A pseudo-random generator is deterministic given its seed: the same seed always produces the same stream, so recording the seed makes a 'random' study exactly repeatable. But not all seeds are interchangeable in practice — seeding parallel workers naively by index (1, 2, 3, ...) can start highly correlated streams, which breaks the independence a parallel simulation relies on even though every individual worker is running the identical deterministic algorithm.

“Increasing the number of retained components always makes the summary better.”

It always makes the training reconstruction error weakly better, since each extra term can only add energy — but past the point where the retained components represent the true signal, the extra components are reconstructing noise. That is expensive (more storage per component) and does not generalize: it fits idiosyncrasies of this particular noisy draw, not the underlying pattern.

“As long as I evaluate on a held-out test set — or use k-fold cross-validation — my accuracy number is trustworthy.”

A held-out set only protects you if nothing about it influenced the model before that final evaluation. Feature selection, scaling, or imputation fit on the full dataset, or a split that fails to respect subjects or time, all let the test set quietly inform training. The reported number is then measuring memorization, not generalization.

“Clustering means finding groups that are round blobs in space, so k-means is 'the' clustering algorithm.”

k-means partitions space by Euclidean distance to centroids, which only recovers convex, roughly spherical groups. Spectral clustering instead asks which points are locally connected on a similarity graph, so it recovers clusters defined by connectivity and shape — moons, rings, spirals — that are not separable by any set of centroids.

“The variable with the highest importance score is the one that causes the outcome.”

Every measure here is a statistical summary of association or predictive loss under a specific procedure, not an estimate of a causal effect. Correlated duplicates, refit-vs-fixed-model differences, and even the number of categories in a variable all move these scores without touching the underlying data-generating process. Ask 'important for what question, computed how' before comparing rankings across variables or across methods.

“If you have the memory for exact statistics, computing them exactly is always the right choice.”

Exact statistics require O(n) memory to support order statistics like the median, and a naive one-pass variance formula is numerically unsafe once values are large or shifted. Streaming estimators are not merely a fallback for scale: Welford's update is strictly more accurate than the naive formula at any n, and sketches expose an explicit, tunable memory-for-accuracy trade that exact recomputation hides until it runs out of RAM.

“Cross-validation is cross-validation — a random k-fold split gives the same honest error estimate for a time series as it does for i.i.d. data.”

Random k-fold assumes exchangeability: any observation could equally well be used to predict any other. Time series break that assumption by construction — the future depends on the past, not vice versa. Letting a fold's training set include points after the target time leaks information a real forecaster would never have, and the reported error is optimistic, sometimes dramatically so. Rolling-origin or expanding-window evaluation, which only ever trains on the past, is the honest analogue of cross-validation for sequential data.

“Backpropagation is the algorithm that trains a neural network.”

Backpropagation only computes ∂L/∂w for every weight, using the chain rule layer by layer. Training also requires an update rule — plain gradient descent, momentum, Adam, or anything else — that decides how to use those derivatives. The gradients in this lab do not change if you imagine using a different optimizer; only the weight-update arithmetic would.

“A fair model satisfies every fairness definition at once.”

Demographic parity, equal opportunity, and calibration are each defensible on their own, but they are mathematically incompatible whenever the two groups have different base rates and the classifier is imperfect. Picking a 'fair' model means choosing which criterion to prioritize, not finding a model that satisfies all of them — this is a provable impossibility result, not a modeling failure.

“If a t-SNE plot shows separated blobs, the data has that many real clusters, and the distance between blobs tells you how different those clusters are.”

t-SNE's objective only tries to preserve small pairwise probabilities — who is a near neighbour of whom — while an aggressive early-exaggeration phase and a heavy-tailed embedding kernel actively encourage tight, well-separated visual clumps regardless of whether such structure exists upstream. Global quantities like the number of blobs, their sizes, and the distances between them are not optimized at all and can change completely with the random seed, the perplexity, or the learning rate on identical data. PCA, which has no such mechanism, shows the same noise or single-blob data as one undifferentiated cloud.

“A kernel like RBF works by first computing the coordinates of each point in a high-dimensional feature space.”

It never does. The dual SVM only needs inner products K(xᵢ,xⱼ), and the kernel function computes that scalar directly from the two low-dimensional inputs — for RBF, exp(−γ‖x−y‖²) is one exponential of one distance, not a dot product taken after mapping to an infinite-dimensional space. The 'explicit feature dimension' readout in this lab shows what you would need to compute by hand if you insisted on doing it the slow way; the kernel-matrix cost readout shows what the SVM actually pays, and it never depends on that dimension.

“EM finds the maximum likelihood solution — its guaranteed convergence means it finds the best fit.”

EM guarantees that the log-likelihood never decreases from one iteration to the next, and that the sequence converges to a stationary point. It does not guarantee that this stationary point is the global maximum. With bad initial parameters, EM converges to a local optimum with a strictly lower log-likelihood than another run from a better start on the identical data — exactly what the compare-seeds panel and the worst-case-corner initialization demonstrate.

“Test error is U-shaped in model capacity: more parameters than data always means overfitting and gets worse forever.”

Test error is worst exactly at the interpolation threshold p = n, not beyond it. Once the model has enough capacity to interpolate the training data in many different ways, the minimum-norm (or otherwise regularized) choice among those solutions gets smoother — not wilder — as capacity keeps growing, producing a second descent in test risk.

“A chart is a neutral window onto the data — it just shows what's there.”

Every chart embeds choices that a viewer cannot see from the picture alone: bin width, axis limits, aspect ratio, linear-vs-log scale, level of aggregation, and colour mapping. None of those choices touch the underlying numbers, but all of them change the story a viewer walks away with. Reading a chart honestly means asking what was chosen before asking what it shows.

“Bagging and boosting are basically the same idea — averaging trees — so it doesn't matter which one you pick, and more trees eventually overfits any forest.”

Bagging averages independent trees fit to resampled data; it reduces variance at roughly fixed bias and its test error plateaus as trees accumulate, because independently-fit trees eventually stop adding new information. Boosting fits trees sequentially to residuals with a learning rate; it reduces bias fast, but with a high learning rate and enough rounds it starts fitting training noise, and test error can rise again. The two curves in the error panel above have visibly different shapes for exactly this reason.

“If I rank features by importance (or fit a model and read off p-values), the top-ranked ones are the discoveries I can trust.”

A ranking, by construction, always returns an answer — the top k features exist whether or not any of them are real, and an uncorrected p-value threshold ignores how many tests were run. Neither carries a false discovery rate guarantee. The knockoff filter instead builds a synthetic negative control for every feature and uses the sign pattern of real-vs-decoy competition to certify, before you ever see the data twice, that the expected fraction of false discoveries stays below q.

“The constrained optimum is just the unconstrained minimum, clipped into the feasible region.”

Clipping projects a single point — the unconstrained minimum — onto the boundary using only the direction from the center of the constraint, ignoring the objective's gradient entirely at the projection step. The true constrained optimum is where the objective's gradient is anti-parallel to the constraint's gradient, which is generally a different point on the boundary. The gap readout in this lab makes that difference numeric, not just visual.

“Grid search and Bayesian optimization differ only in speed.”

They differ in what each query depends on. Grid search fixes every location in advance regardless of the data; Bayesian optimization recomputes a posterior after each observation and lets the acquisition function choose the next location adaptively. The efficiency panel shows BO reaching a given best-value threshold in far fewer evaluations on average — not because it evaluates faster, but because it evaluates smarter. And the same adaptivity that helps it can hurt it: an over-smooth surrogate or an overly exploitative acquisition can make BO commit early and get stuck, something grid search's blind uniform coverage never does.