Controlled Selection with Knockoffs

Manufacture decoy variables, then use them to bound the false discovery rate.

Ch 11 11.6

Explorer

Validation & Feature Selection

25 mindifficulty 5/5

Preparing…

Predict before you look

Averaged over many repeated draws, which selection rule keeps its realized FDR at or below the target q?

Scenarios

Teacher-ready starting points.

Documentation

What this shows

A synthetic Gaussian design with AR(1) correlation ρ across p features, k of which truly drive the response. For each feature a knockoff twin is constructed from the known covariance — a decoy statistically indistinguishable from the original under the null but independent of the response given the original. A lasso path over the doubled design yields entry-lambda statistics Wⱼ, and the knockoff/knockoff+ threshold turns those into a selection set with a guaranteed bound on the expected false discovery rate. A repeated-simulation panel contrasts the realized FDR of the filter against a naive importance-ranking or uncorrected p-value rule.

Exchangeability, not significance

Model-X knockoffs need one thing: the joint distribution of (X, X̃) is invariant under swapping any subset of the (Xⱼ, X̃ⱼ) pairs, while X̃ is constructed without ever looking at y. For a Gaussian design with covariance Σ, equicorrelated knockoffs draw X̃ = X(I − Σ⁻¹diag(s)) + Z̃Cᵀ, where diag(s) = sI for a scalar s ≤ 2λmin(Σ) and CCᵀ = 2diag(s) − diag(s)Σ⁻¹diag(s). This construction guarantees Cov(Xⱼ, Xₖ) = Cov(X̃ⱼ, X̃ₖ) for all j,k, and Cov(Xⱼ, X̃ₖ) = Cov(Xⱼ, Xₖ) off the diagonal.

For each feature, the statistic Wⱼ compares evidence for the original against evidence for its knockoff — here, the lasso regularization value at which each first enters the model. A large positive Wⱼ means the real feature beat its decoy; under the null, Wⱼ is equally likely to be positive or negative by exchangeability, which is exactly the coin-flip structure the threshold exploits.

The knockoff+ threshold T controls E[FDP] ≤ q exactly by counting one extra false discovery in the numerator; the plain knockoff threshold controls a slightly looser modified FDR. Neither guarantee has anything to do with how features are ranked — a top-k or uncorrected-p-value rule has no decoy to compare against and so has no such control.

X̃ = X(I − Σ⁻¹diag(s)) + Z̃Cᵀ,   CCᵀ = 2diag(s) − diag(s)Σ⁻¹diag(s)

Wⱼ = sign(Zⱼ − Z̃ⱼ)·max(Zⱼ, Z̃ⱼ),  Zⱼ = entry-λ of feature j

T = min{ t : (1 + #{j : Wⱼ ≤ −t}) / max(#{j : Wⱼ ≥ t}, 1) ≤ q }  (knockoff+)

Select { j : Wⱼ ≥ T }  ⟹  E[FDP] ≤ q

Try this

Set amplitude to 0 (no real signal) and compare the knockoff selection set — usually empty — against the naive top-k rule, which always reports exactly k features.
Raise ρ from 0 toward 0.9 and watch correlated null features climb the importance ranking while the knockoff FDR readout stays anchored near q.
Switch between knockoff and knockoff+ at small q and few discoveries; note how knockoff+ is more conservative but its guarantee is exact even then.
Increase p and k together and watch power rise for the knockoff filter as more true signal becomes available to distinguish from noise.
Inspect the correlation heatmap at high ρ: the swapped off-diagonal block should visually match Σ, the empirical signature of exchangeability.

Common misconception

“If I rank features by importance (or fit a model and read off p-values), the top-ranked ones are the discoveries I can trust.”

A ranking, by construction, always returns an answer — the top k features exist whether or not any of them are real, and an uncorrected p-value threshold ignores how many tests were run. Neither carries a false discovery rate guarantee. The knockoff filter instead builds a synthetic negative control for every feature and uses the sign pattern of real-vs-decoy competition to certify, before you ever see the data twice, that the expected fraction of false discoveries stays below q.

DSPA3 §11.6 — Knockoffs and controlled variable selection