What does "Underpowered Test" mean?

Definition of Underpowered Test in the context of A/B testing (online controlled experiments).

What is an Underpowered Test?

An underpowered A/B or MVT test is a test which had relatively poor probability for detecting a specified effect size of interest (MEI). Getting a non-significant outcome from such a test is a poor evidence for the null hypothesis versus the specified alternative hypothesis for which statistical power is calculated.

A test can be underpowered with respect to some possible alternatives while being properly powered or overpowered versus others since power is calculated at a point. By definition in frequentist statistics the larger the magnitude of the effect of interest, the more powerful the test.

Like this glossary entry? For an in-depth and comprehensive reading on A/B testing stats, check out the book "Statistical Methods in Online A/B Testing" by the author of this glossary, Georgi Georgiev.

About the author

Georgi Z. Georgiev

Georgi has over twenty years of experience in online marketing, web analytics, statistics, and design of business experiments.

Author of the book "Statistical Methods in Online A/B Testing", white papers on statistical analysis of A/B tests, and a speaker, he has been distinguished as a winner in the Data & Analytics category of the 2024 Experimentation Thought Leadership Awards.

Statistical Methods in Online A/B Testing

Take your A/B testing program to the next level with the most comprehensive book on user testing statistics in e-commerce.

Learn more

Glossary index by letter

A B C D E F G H I K L M N O P R S T U V Z

Select a letter to see all A/B testing terms starting with that letter or visit the Glossary homepage to see all.

What does "Underpowered Test" mean?

What is an Underpowered Test?

Articles on Underpowered Test

Related A/B Testing terms

See this in action

About the author

Glossary index by letter