What does "Underpowered Test" mean?

Definition of Underpowered Test in the context of A/B testing (online controlled experiments).

What is an Underpowered Test?

An underpowered A/B or MVT test is a test which had relatively poor probability for detecting a specified effect size of interest (MEI). Getting a non-significant outcome from such a test is a poor evidence for the null hypothesis versus the specified alternative hypothesis for which statistical power is calculated.

A test can be underpowered with respect to some possible alternatives while being properly powered or overpowered versus others since power is calculated at a point. By definition in frequentist statistics the larger the magnitude of the effect of interest, the more powerful the test.

Related A/B Testing terms

Statistical Power

Type II Error

beta

Articles on Underpowered Test


Like this glossary entry? For an in-depth and comprehensive reading on A/B testing stats, check out the book "Statistical Methods in Online A/B Testing" by the author of this glossary, Georgi Georgiev.


Purchase Statistical Methods in Online A/B Testing


Glossary Index by Letter

ABCDEFGHIKLMNOPRSTUVZ

Select a letter to see all A/B testing terms starting with that letter or visit the Glossary homepage to see all.