Holm-Bonferroni method

In statistics, the Holm-Bonferroni method performs more than one hypothesis test simultaneously. It is named after Sture Holm and Carlo Emilio Bonferroni.

Suppose there are k hypotheses to be tested and the overall type 1 error rate is α. Start by ordering the p-values and comparing the smallest p-value to α/k. If that p-value is less than α/k, then reject that hypothesis and start all over with the same α and test the remaining k - 1 hypothesis, i.e. order the k - 1 remaining p-values and compare the smallest one to α/(k - 1). Continue doing this until the hypothesis with the smallest p-value cannot be rejected. At that point, stop and accept all hypotheses that have not been rejected at previous steps.

Here is an example. Four hypotheses are tested with α = 0.05. The four unadjusted p-values are 0.01, 0.03, 0.04, and 0.005. The smallest of these is 0.005. Since this is less than 0.05/4, hypothesis four is rejected. The next smallest p-value is 0.01, which is smaller than 0.05/3. So, hypothesis one is also rejected. The next smallest p-value is 0.03. This is not smaller than 0.05/2. Therefore, hypotheses one and four are rejected while hypotheses two and three are not rejected.

The Holm-Bonferroni method is an example of a closed test procedure. As such, it controls the familywise error rate for all the k hypotheses at level α in the strong sense. Each intersection is tested using the simple Bonferroni test.

It is also possible to define a weighted version. Let p1,..., pk be the unadjusted p-values and let w1,..., wk be a set of corresponding positive weights that add to 1. Without loss of generality, assume the p-values and the weights are all ordered such that p1/w1 ≤ p2/w2 ≤ ... ≤ pk/wk. The adjusted p-value for the first hypothesis is q1 = min{1,p1/w1}. Inductively, define the adjusted p-value for hypothesis i by qi=min{1,max{qi-1,(wi + ... + wk)×pi/wi}}. A hypothesis is rejected at level α if and only if its adjusted p-value is less than α. In the earlier example using equal weights, the adjusted p-values are 0.03, 0.06, 0.06, and 0.02. This is another way to see that using α = 0.05, only hypotheses one and four are rejected by this procedure.