What does "Bonferroni Correction" mean?

What is Bonferroni Correction?

The Bonferroni Correction is one of the earliest procedures for controlling the Family-Wise Error Rate (FWER) which is the error rate across a set of tightly related significance tests. The goal of the correction is to maintain the overall type I error rate which is computed under the null hypothesis that all nulls tested in the significance tests are false. The correction provides a conservative bound on alpha_m.

Bonferroni derived the calculation that the overall α of performing m significance tests is equal to 1 - (1 - α_{per test})^m which is the probability that one of them will result in a statistically significant outcome. The simple Bonferroni correction would suggest performing each test at level α/m to maintain the Family-Wise Error Rate (FWER) fixed at α, but this is a conservative adjustment when the comparisons are not independent.

The Bonferroni correction can be applied when there is more than one primary KPI in an A/B test and finding any of them to be statistically significant would result in deciding against the control. Still, it is a bit conservative in the presence of positive dependence so the Sidak Correction is usually slightly more powerful and thus preferred.

While the Bonferroni correction can also be applied to a multivariate test (A/B/n test) it is not the best choice as it takes no account of the dependency present between the tests due to the fact that they are all against a common control. In such cases the Dunnett’s correction provides a significantly more powerful method that also controls FWER as defined above.

Like this glossary entry? For an in-depth and comprehensive reading on A/B testing stats, check out the book "Statistical Methods in Online A/B Testing" by the author of this glossary, Georgi Georgiev.

Articles on Bonferroni Correction

Multivariate Testing – Best Practices & Tools for MVT (A/B/n) Tests
blog.analytics-toolkit.com