Accuracy Is a Trap: Metrics Under Class Imbalance

Move a threshold, move the prevalence, and watch which performance numbers stay honest and which quietly lie.

Ch 5 5.6

Comparator

Supervised Learning

20 mindifficulty 2/5

Controls

Prevalenceπ2.0%

Share of positives in the population.

Signal separationΔ1.600

How far apart the two latent score distributions sit.

Decision thresholdt0.500

Changes the confusion matrix only — not the ranking.

Sample sizen2000

Random seed

Every result on this page is a deterministic function of the seed and the controls.

Score distributions by true class

The threshold is a vertical cut.

NegativesPositives

ROC

AUC = 0.936 — threshold-free.

Precision–recall

Average precision = 0.520 — baseline 0.021.

Confusion matrix at the current threshold

Pred +

Pred −

True +

True −

1957

Accuracy

98.10%

always-negative: 97.90%

Balanced accuracy

55.93%

Sensitivity / recall

11.90%

Specificity

99.95%

Precision (PPV)

83.33%

depends on prevalence

0.208

MCC

0.311

robust to imbalance

AUC

0.936

independent of threshold

Predict before you look

A disease affects 1% of patients. A model always predicts “no disease”. What is its accuracy, and what is its AUC?

Scenarios

Teacher-ready starting points.

Documentation

What this shows

A simulated screening problem where you control how rare the positive class is, how separable the classes are, and where the decision threshold sits. Every metric is recomputed live from one confusion matrix, alongside the two curves that summarize all thresholds at once.

One matrix, many summaries

A threshold t turns scores into predictions and yields counts TP, FP, FN, TN. Sensitivity and specificity condition on the true class, so they do not depend on prevalence. Precision conditions on the prediction, so it does.

Bayes' rule makes the dependence explicit: precision rises with prevalence even when sensitivity and specificity are fixed. This is why a 99%-specific test for a 1-in-1000 disease still produces mostly false positives.

The ROC curve traces (FPR, TPR) over all thresholds and its area equals the probability that a random positive outranks a random negative — a property of the ranking, invariant to both threshold and prevalence. The precision–recall curve has a baseline equal to the prevalence, so it is the more informative summary for rare events.

sens = TP/(TP+FN),  spec = TN/(TN+FP),  prec = TP/(TP+FP)

prec = (sens·π) / (sens·π + (1−spec)(1−π))

AUC = P(score(X⁺) > score(X⁻))

Try this

Set prevalence to 1% and push the threshold to 1.0: accuracy stays near 99% with zero positives detected.
Return the threshold to 0.5 and compare accuracy with balanced accuracy and MCC.
Hold separation fixed and slide prevalence from 50% to 1%; watch AUC hold steady while precision falls.
Slide the threshold across its whole range and confirm the ROC and PR curves never move.
Set separation to 0 — AUC drops to 0.5 and the ROC collapses onto the diagonal.

Common misconception

“My classifier is 99% accurate, so it works well.”

With 1% prevalence, predicting “negative” for everyone is 99% accurate and useless. Accuracy weights both error types by their frequency, which hides failures on the rare class that usually matters most. Report sensitivity and precision at a stated threshold, plus AUC or average precision for the ranking, and always give the majority-class baseline for comparison.

DSPA3 Chapter 5 — Supervised classification