Net Benefit and Threshold Choice

Accuracy cannot pick a threshold. A stated harm-to-benefit ratio can — and the decision curve shows where a model earns its keep.

Ch 9 9.7

Explorer

Validation & Feature Selection

20 mindifficulty 3/5

Controls

Outcome prevalence12.0%

Model signalβ1.600

β = 0 is a useless model; large β approaches perfect separation.

Decision thresholdt50%

Harm : benefit ratio10

Treating 10 unnecessary cases is as bad as missing one. Implied threshold 9.1%.

Sample sizen1500

Show treat-all strategy

The benchmark every model must beat.

Random seed

Every result on this page is a deterministic function of the seed and the controls.

Decision curve: net benefit against threshold

A model is only useful where its curve is above both the treat-all line and zero.

modeltreat alltreat none (zero)your thresholdthreshold from your loss ratio

Net benefit at your t

0.0407

true positives per patient, net of harm

Treat-all at the same t

-0.6160

model wins here

Interventions avoided

65.7per 100

versus treating everyone, same detection rate

AUC

0.847

threshold-free discrimination

At your threshold

t = 50% on 1500 cases.

treated151 (10.1%)

true positives106

false positives45

sensitivity36.8%

specificity96.3%

accuracy84.9%

accuracy of treat-none80.8%

Accuracy prefers treat-none whenever prevalence is low. Net benefit does not, because it prices a missed case against an unnecessary treatment.

Choosing t from a loss ratio

harm : benefit1 : 10

implied threshold9.09%

net benefit there0.1391

best t on this curve1.0%

observed prevalence19.20%

The threshold is a statement about consequences: t/(1−t) is exactly the number of false positives you will tolerate for one true positive.

0.5 is not a natural cutoff

Thresholding a probability at 0.5 asserts that a false positive and a false negative cost the same. In screening they almost never do. Fix the harm-to-benefit ratio you are willing to defend, and the threshold follows: t = 1/(1 + harm:benefit). Every point on the decision curve corresponds to one such ratio.

Predict before you look

A calibrated model for a 5%-prevalence outcome is used at the default cutoff of 0.5. What is the most likely outcome?

Scenarios

Teacher-ready starting points.

Documentation

What this shows

A calibrated risk model on a simulated cohort with adjustable prevalence and signal strength. The decision curve plots net benefit against threshold probability for the model, for treating everyone, and for treating nobody, so you can read off the range of thresholds where the model is the best of the three strategies.

Net benefit and the exchange rate

Choosing a threshold t is equivalent to declaring an exchange rate between error types. If you are willing to treat w unnecessary cases to catch one real one, then you should treat whenever the predicted risk exceeds t = 1/(1 + w); equivalently w = t/(1 − t) is the odds of the threshold.

Net benefit puts both error types in units of true positives per person: NB(t) = TP/n − (FP/n)·t/(1−t). Treating nobody has NB = 0 by construction, and treating everybody has NB = prevalence − (1 − prevalence)·t/(1−t), which falls steeply as t rises.

A model is clinically useful over the range of t where its curve sits above both benchmarks. Because the weight is the threshold's own odds, the curve requires no external cost figures: the reader supplies their own t and reads the answer. Net interventions avoided per 100, (NB_model − NB_all)/(t/(1−t))·100, re-expresses the same gap as treatments saved at equal detection.

t = 1/(1 + harm:benefit),   w = t/(1−t)

NB(t) = TP/n − (FP/n)·t/(1−t)

NB_none = 0

NB_all(t) = prevalence − (1 − prevalence)·t/(1−t)

avoided per 100 = 100 · (NB_model − NB_all) / (t/(1−t))

Try this

Set prevalence to 5% and leave t at 0.5. Compare net benefit with the accuracy readout — do they agree about the model's value?
Set the harm:benefit ratio to 20 and move your threshold to the implied value. How much net benefit did you gain?
Set β = 0 and confirm the model curve never rises above the two benchmarks.
Raise prevalence to 40% and find the threshold at which treat-all stops being the best strategy.
Find a setting where the model has AUC above 0.8 yet no useful threshold range at all.

Common misconception

“The model has AUC 0.85, so it is ready to guide treatment decisions.”

AUC is a threshold-free ranking measure: it says nothing about whether acting on the model at any particular cutoff beats treating everyone or nobody. Under low prevalence a well-ranked model can still sit below the treat-all line for every threshold a clinician would accept. Decision curve analysis makes usefulness explicit, and it demands one thing AUC does not: that you say what a false positive costs relative to a false negative.

DSPA3 Chapter 9 — Model performance assessment