What does "Dunnett's Correction" mean?

What is the Dunnett's Correction?

The Dunnett's Correction has been developed in the 1960s in order to provide a powerful procedure for controlling the Family-Wise Error Rate (FWER) in the case of comparing multiple treatments to a common control group which is the case in a multivariate test. Such a scenario is a case of multiple comparisons / multiple testing. The goal of the correction is to maintain the overall type I error rate which is computed under the null hypothesis that there is no difference between the control group and any of the test groups. In this sense it is an adjustment of the Bonferroni Correction to that specific case.

Dunnett's test is the most-powerful method that maintains control of the FWER in such a scenario as it makes use of the positive dependence between the comparisons.

While the correction is a p-value adjustment after a test is completed, when planning an MVT one needs to perform sample size computations and power analysis by taking the p-value correction into account.

The correction is usually applied as a step-down procedure whereas the sample size calculations require integrating over multivariate normal distributions, often through simulations.

Like this glossary entry? For an in-depth and comprehensive reading on A/B testing stats, check out the book "Statistical Methods in Online A/B Testing" by the author of this glossary, Georgi Georgiev.

Articles on Dunnett's Correction

Multivariate Testing – Best Practices & Tools for MVT (A/B/n) Tests
blog.analytics-toolkit.com