Calibration Is Not Discrimination

Distort a model's probabilities without touching its ranking: AUC does not move a decimal place while every probability becomes wrong.

Ch 9 9.5

Misconception

Validation & Feature Selection

18 mindifficulty 3/5

Controls

Signal

Predictor strengthβ1.400

Controls discrimination — and only discrimination.

Distortion (rank-preserving)

Logit slopea2.400

a > 1 makes the model overconfident; a < 1 underconfident.

Logit shiftb0.600

A systematic bias in reported risk.

Calibration bins10

Sample sizen2000

Random seed

Every result on this page is a deterministic function of the seed and the controls.

Reliability diagram

Observed event rate against reported probability. Point size is bin count.

Reported modelTrue probabilitiesPerfect calibration

Reported versus true probability

A monotone curve: the ranking is identical, the numbers are not.

AUC (reported)

0.7963

true model 0.7963 — identical

Expected calibration error

0.1384

true model 0.0092

Brier score

0.2080

true model 0.1842 — lower is better

Log loss

0.6893

true model 0.5468

Mean reported risk

55.72%

Observed event rate

49.45%

calibration-in-the-large compares these two

Predict before you look

You take a well-calibrated model and replace every predicted probability p with σ(3·logit(p)), making it far more confident. What happens to the AUC?

Scenarios

Teacher-ready starting points.

Documentation

What this shows

A logistic model whose true event probabilities are known, plus a distortion you apply to the reported probabilities on the logit scale. Because the distortion is strictly increasing, it cannot change the ranking — so it isolates calibration from discrimination.

Rank invariance and proper scoring rules

AUC is a functional of the ordering of the scores alone. Any strictly increasing transformation g leaves it exactly unchanged, so reporting g(p) instead of p is invisible to ROC analysis.

Calibration asks a different question: among cases assigned probability about q, does the event occur about q of the time? The reliability diagram plots that directly and perfect calibration is the diagonal.

Proper scoring rules such as the Brier score and log loss are minimized only by the true conditional probability, so they detect the distortion. This is why validation should report a discrimination measure and a calibration measure together — the standard decomposition splits the Brier score into calibration, refinement, and irreducible uncertainty.

reported = σ(a·logit(p) + b),  strictly increasing in p ⇒ AUC unchanged

Brier = (1/n) Σ (pᵢ − yᵢ)²,   LogLoss = −(1/n) Σ [yᵢ log pᵢ + (1−yᵢ) log(1−pᵢ)]

ECE = Σ_b (n_b/n) · |observed_b − predicted_b|

Try this

Set slope 1 and shift 0: the reliability curve lies on the diagonal and the scores match the true model.
Raise the slope to 4 — an overconfident model. Watch AUC hold and log loss climb.
Set the shift to +2 with slope 1: every risk is systematically too high, a pure calibration-in-the-large failure.
Lower the predictor strength to 0.2. Now discrimination is genuinely poor but a slope-1 model is still well calibrated.
Compare the mean reported risk with the observed event rate at various shifts.

Common misconception

“My model has an AUC of 0.85, so its predicted probabilities are reliable.”

AUC only measures whether positives are ranked above negatives; it is invariant to every monotone rescaling of the scores. A model can achieve any AUC while reporting probabilities that are systematically double the truth. Whenever probabilities feed a decision — treatment thresholds, expected loss, resource allocation — you must report calibration (reliability diagram, ECE, Brier, log loss) alongside discrimination, and recalibrate if needed.

DSPA3 Chapter 9 — Model performance and validation