What does "Multiple Comparisons" mean?

Definition of Multiple Comparisons in the context of A/B testing (online controlled experiments).

What is Multiple Comparisons?

There is no strict definition of multiple comparisons in the statistics literature: it sometimes refers to comparing multiple groups between each other or versus a shared control group while in other cases it refers to comparing only two groups but based on multiple characteristics of theirs. In A/B testing multiple comparisons most often refers to the former case and especially to the situation of significance testing multiple test groups versus a common control group, known as multivariate testing (MVT).

Regardless of the precise definition multiple statistical comparisons only one of which is enough to lead to the rejection of the null hypothesis lead to the need to control the Family-Wise Error Rate (FWER) where "family" refers to a set of logically connected significance tests and "error rate" refers to the type I error rate.

When one performs multiple comparisons in the above sense the Dunnett's test is the most powerful (as in statistical power) procedure one can use. It is essentially a p-value adjustment and so is often referred to as Dunnett's Correction. Using a FWER-correcting procedure also has consequences during the planning stage, when the statistical design is decided on (with or without a risk-reward analysis). sample size calculations need to take into account the multiple comparisons correction which will be applied after the data is gathered, otherwise one is likely to end up with an underpowered test.

Articles on Multiple Comparisons

Like this glossary entry? For an in-depth and comprehensive reading on A/B testing stats, check out the book "Statistical Methods in Online A/B Testing" by the author of this glossary, Georgi Georgiev.

Purchase Statistical Methods in Online A/B Testing

Glossary Index by Letter


Select a letter to see all A/B testing terms starting with that letter or visit the Glossary homepage to see all.