What does "Overpowered Test" mean?

Definition of Overpowered Test in the context of A/B testing (online controlled experiments).

What is an Overpowered Test?

An overpowered A/B or MVT test is one which has relatively poor probability for detecting a specified minimum effect of interest (MEI). Getting a statistically significant result from such a test is a poor evidence for large discrepancies from the null hypothesis versus observing the same p-value from a less powerful test. This becomes very obvious when examining the upper confidence limit of a one-sided or two-sided confidence interval.

A test can be overpowered with respect to larger alternatives while being properly powered or even underpowered versus alternatives closer to the null hypothesis since power is calculated at a point. By definition in frequentist statistics the smaller the magnitude of the effect of interest, the less powerful the test, all else being fixed.

Related A/B Testing terms

Statistical Power

Type II Error

beta

Articles on Overpowered Test


Glossary Index by Letter

ABCDEFGHIKLMNOPRSTUVZ

Select a letter to see all A/B testing terms starting with that letter or visit the Glossary homepage to see all.