What does "Uniformly Most Powerful Test" mean?

What is Uniformly Most Powerful Test?

Alias: UMP

In hypothesis testing using statistical methods a uniformly most powerful (UMP) test is a test of a specified null hypothesis which has the greatest statistical power (1-β) among all possible tests with a specified significance threshold α. The Neyman–Pearson lemma proved the presence of a UMP test for two point hypotheses: the likelihood ratio test, while an extension by the Karlin-Rubin theorem extended the lemma to cover composite hypotheses.

For the lemma to apply the parameter space Θ must be partitioned into two disjoint sets Θ₀ and Θ₁ (Θ₀ ∪ Θ₁ = Θ, Θ₀ ∩ Θ₁ = ∅).

A UMP test does not always exist with a trivial case being a two-sided test in which the alternative lies on both sides of the null. In such a case the uniformly most powerful test of a certain sample size for a value of the parameter on one side of the null is different than that for a value of the parameter on the other side of the null. This is not a major concern in A/B testing where most of the time we perform a one-sided test.

In practice, having a UMP test simply means that one is using the statistical method which makes the most efficient use of the available sample size and there does not exist a more efficient test for this data set.

Like this glossary entry? For an in-depth and comprehensive reading on A/B testing stats, check out the book "Statistical Methods in Online A/B Testing" by the author of this glossary, Georgi Georgiev.