What does "Non-Inferiority Test" mean?

Definition of Non-Inferiority Test in the context of A/B testing (online controlled experiments).

What is a Non-Inferiority Test?

A non-inferiority test is such an online experiment in which we have reason to implement a tested variant as long as it is not substantially worse than the control. In particular, we would consider it a "winner" so long as it is no worse than a specified non-inferiority margin m.

This might be due to different reasons such as testing speed requirements, side benefits such as lower maintenance costs or improvements in other metrics of interest. Non-inferiority tests are often performed for secondary KPIs such as site speed, email unsubscribe rate, etc.

In contrast, in the more common superiority test we would be happy only if we can reject a null of μ ≤ 0 or even μ ≤ ε where ε is a difference that we must surpass in order to justify implementing the tested variant.

Articles on Non-Inferiority Test

Like this glossary entry? For an in-depth and comprehensive reading on A/B testing stats, check out the book "Statistical Methods in Online A/B Testing" by the author of this glossary, Georgi Georgiev.

Purchase Statistical Methods in Online A/B Testing

Glossary Index by Letter


Select a letter to see all A/B testing terms starting with that letter or visit the Glossary homepage to see all.