What does "Uniformly Most Powerful Test" mean?

Definition of Uniformly Most Powerful Test in the context of A/B testing (online controlled experiments).

What is Uniformly Most Powerful Test?

Aliases: UMP

In hypothesis testing using statistical methods a uniformly most powerful (UMP) test is a test of a specified null hypothesis which has the greatest statistical power (1-β) among all possible tests with a specified significance threshold α. The Neyman–Pearson lemma proved the presence of a UMP test for two point hypotheses: the likelihood ratio test, while an extension by the Karlin-Rubin theorem extended the lemma to cover composite hypotheses.

For the lemma to apply the parameter space Θ must be partitioned into two disjoint sets Θ0 and Θ10 ∪ Θ1 = Θ, Θ0 ∩ Θ1 = ∅).

A UMP test does not always exist with a trivial case being a two-sided test in which the alternative lies on both sides of the null. In such a case the uniformly most powerful test of a certain sample size for a value of the parameter on one side of the null is different than that for a value of the parameter on the other side of the null. This is not a major concern in A/B testing where most of the time we perform a one-sided test.

In practice, having a UMP test simply means that one is using the statistical method which makes the most efficient use of the available sample size and there does not exist a more efficient test for this data set.

Like this glossary entry? For an in-depth and comprehensive reading on A/B testing stats, check out the book "Statistical Methods in Online A/B Testing" by the author of this glossary, Georgi Georgiev.

Purchase Statistical Methods in Online A/B Testing

Glossary Index by Letter


Select a letter to see all A/B testing terms starting with that letter or visit the Glossary homepage to see all.