Importance Depends on the Question
Impurity, permutation, and SHAP-style attributions disagree — especially under correlation.
Controls
Correlation between x1 and its duplicate; the twin has no direct effect on y.
Shrinkage applied when fitting the standardized model.
Categories of the impurity-splitting demo below, independent of y.
Every result on this page is a deterministic function of the seed and the controls.
Four importance measures, per variable
Bars are normalized within each measure (column) to its own maximum, so widths compare rank, not raw scale.
Rank table
1 = most important under that measure. Disagreement is the normalized mean pairwise footrule distance.
| variable | marginal | coef | permutation | LOCO |
|---|---|---|---|---|
| x1 (true) | 1 | 1 | 1 | 2 |
| x2 (true) | 3 | 2 | 2 | 1 |
| x1 twin | 2 | 3 | 3 | 3 |
| noise 1 | 4 | 5 | 4 | 4 |
| noise 2 | 5 | 4 | 5 | 5 |
High-cardinality noise wins
A purely random label independent of y still explains variance in y just by chance, more so as the number of categories grows relative to n.
Marginal correlation asks "how associated is x with y, ignoring everything else" and inflates both members of a correlated pair. The standardized coefficient asks "how much does the fitted model lean on x once the others are in the equation," and with two near-identical columns it can lean on either one arbitrarily and split credit unstably as ρ rises. Permutation importance asks "how much worse do predictions get if x is scrambled," which is small for a twin because the model can lean on its partner instead. LOCO asks "how much worse do predictions get if x is unavailable at refit time," which is smaller still for redundant variables because the refit model routes around the missing column. None of the four is measuring causal contribution.