Testing Many Things at Once

Run 200 tests where nothing is real and still publish a finding — then watch Bonferroni and Benjamini–Hochberg trade power for error control.

Ch 2 2.9

Simulator

Foundations & Data Quality

18 mindifficulty 3/5

Controls

Tests in the studym200

Tests with a real effectm₁10

The rest are exactly null.

Effect sizeδ3

Mean of the z-statistic under the alternative.

Nominal levelα0.050

Multiplicity procedure

Replicate studies200

Long-run error rates are averaged over this many independent studies.

Random seed

Every result on this page is a deterministic function of the seed and the controls.

One study, sorted p-values

Bonferroni is a flat line; Benjamini–Hochberg is a ramp — the largest crossing wins.

true discoveryfalse discoveryα (uncorrected)α/m (Bonferroni)αk/m (BH ramp)

Null p-values are uniform

Only the truly null tests, this study.

False discoveries per study

Across 200 replicate studies under No correction.

Family-wise error rate

100.0%

uncorrected theory 100.0%

False discovery rate

51.6%

mean false share of discoveries

Power

84.5%

share of real effects found

False positives / study

9.33

α·m₀ = 9.50

This study: discoveries

This study: false share

60.0%

12 false of 20

Read the two rates apart

FWER is the probability of at least one false positive anywhere in the study. FDR is the expected fraction of the discoveries you report that are false. Bonferroni controls the first and is conservative; BH controls the second and keeps far more power.

Predict before you look

You run 100 independent tests at α = 0.05 on data where every null hypothesis is true. What is the chance you report at least one significant result?

Scenarios

Teacher-ready starting points.

Documentation

What this shows

A screening study of m independent tests, of which m₁ carry a real effect. One study is shown as a sorted p-value plot with each procedure's threshold drawn on it, and the long-run FWER, FDR, and power are averaged over hundreds of replicate studies.

Two different error rates

Under the null a two-sided p-value is uniform on [0,1], so P(p ≤ α) = α for each test. With m₀ independent nulls, the chance of at least one false positive is 1 − (1 − α)^m₀, which is already 99.4% at α = 0.05 and m₀ = 100.

Bonferroni tests each hypothesis at α/m. The union bound gives FWER ≤ α with no independence assumption, at the cost of power that falls roughly like the reduced per-test level.

Benjamini–Hochberg sorts the p-values, finds the largest rank k with p₍k₎ ≤ αk/m, and rejects everything up to it. That controls the expected false discovery proportion at α·m₀/m. Because the threshold adapts to how many small p-values there are, it keeps most of the power when real effects exist.

FWER = P(V ≥ 1),   FDR = E[V / max(R,1)]

FWER_uncorrected = 1 − (1 − α)^{m₀}

Bonferroni: reject pᵢ ≤ α/m   ⟹ FWER ≤ α

BH: k* = max{k : p₍k₎ ≤ αk/m},  reject p₍1₎ … p₍k*₎

Try this

Set m₁ = 0 and no correction. Read the FWER: how often does a study of pure noise report a finding?
Keep m₁ = 0 and raise m from 20 to 2000. Compare the measured FWER with 1 − (1 − α)^m.
Set m₁ = 10, δ = 3. Compare power under no correction, Bonferroni, and BH.
With BH selected, lower δ to 1.5 and watch FDR stay near α while power collapses — control is not detection.
Set m = 2000, m₁ = 100 and compare Bonferroni's power with BH's. Which would you use for a genome scan?

Common misconception

“Each of my tests uses α = 0.05, so I have a 5% chance of a false conclusion.”

α = 0.05 is a per-test guarantee. Across m₀ independent nulls, the probability of at least one false positive is 1 − 0.95^m₀ — 40% at m₀ = 10 and above 99% at m₀ = 100. Only a multiplicity procedure restores a study-level guarantee, and you must choose which guarantee you want: FWER (no false positives at all) or FDR (a bounded fraction of your reported discoveries).

DSPA3 Chapter 2 — Multiple testing and inference