What does "Superiority Test" mean?

What is a Superiority Test?

A superiority test is one in which the null hypothesis is that of no difference or no or negative difference in the variable of interest (e.g. means, proportions) while the alternative is that of some a positive change in that variable in the treatment group(s). In a superiority A/B test we require there to be some positive discrepancy between the treatment and control groups in order to proceed with implementing the treatment.

A sub-case of that is a so-called "strong-superiority" test in which the positive discrepancy needs to be greater than zero for the change to be worth impementing.

In a superiority test H₀ is defined as μ ≤ 0 while H₁ as μ > 0. In a strong-superiority test these become H₀ = μ ≤ ε and H₁ = μ > ε.

You can contrast that to a non-inferiority test in which we would implement the tested variant as long as it is not substantially worse than the control.

Like this glossary entry? For an in-depth and comprehensive reading on A/B testing stats, check out the book "Statistical Methods in Online A/B Testing" by the author of this glossary, Georgi Georgiev.