What does "Superiority Test" mean?

Definition of Superiority Test in the context of A/B testing (online controlled experiments).

What is a Superiority Test?

A superiority test is one in which the null hypothesis is that of no difference or no or negative difference in the variable of interest (e.g. means, proportions) while the alternative is that of some a positive change in that variable in the treatment group(s). In a superiority A/B test we require there to be some positive discrepancy between the treatment and control groups in order to proceed with implementing the treatment.

A sub-case of that is a so-called "strong-superiority" test in which the positive discrepancy needs to be greater than zero for the change to be worth impementing.

In a superiority test H0 is defined as μ ≤ 0 while H1 as μ > 0. In a strong-superiority test these become H0 = μ ≤ ε and H1 = μ > ε.

You can contrast that to a non-inferiority test in which we would implement the tested variant as long as it is not substantially worse than the control.

Like this glossary entry? For an in-depth and comprehensive reading on A/B testing stats, check out the book "Statistical Methods in Online A/B Testing" by the author of this glossary, Georgi Georgiev.

Purchase Statistical Methods in Online A/B Testing

Statistical Methods in Online A/B Testing

Take your A/B testing program to the next level with the most comprehensive book on user testing statistics in e-commerce.

Learn more

Glossary index by letter

Select a letter to see all A/B testing terms starting with that letter or visit the Glossary homepage to see all.