What does "Overpowered Test" mean?
Definition of Overpowered Test in the context of A/B testing (online controlled experiments).
What is an Overpowered Test?
An overpowered A/B or MVT test is one which has relatively poor probability for detecting a specified minimum effect of interest (MEI). Getting a statistically significant result from such a test is a poor evidence for large discrepancies from the null hypothesis versus observing the same p-value from a less powerful test. This becomes very obvious when examining the upper confidence limit of a one-sided or two-sided confidence interval.
A test can be overpowered with respect to larger alternatives while being properly powered or even underpowered versus alternatives closer to the null hypothesis since power is calculated at a point. By definition in frequentist statistics the smaller the magnitude of the effect of interest, the less powerful the test, all else being fixed.
Like this glossary entry? For an in-depth and comprehensive reading on A/B testing stats, check out the book "Statistical Methods in Online A/B Testing" by the author of this glossary, Georgi Georgiev.
Articles on Overpowered Test
The Importance of Statistical Power in Online A/B Testing
blog.analytics-toolkit.com
Statistical Power, MDE, and Designing Statistical Tests
blog.analytics-toolkit.com
Related A/B Testing terms
Statistical PowerType II ErrorbetaSee this in action
Power & MDE Calculator
Sample Size Calculator
About the author
Georgi has over twenty years of experience in online marketing, web analytics, statistics, and design of business experiments.
Author of the book "Statistical Methods in Online A/B Testing", white papers on statistical analysis of A/B tests, and a speaker, he has been distinguished as a winner in the Data & Analytics category of the 2024 Experimentation Thought Leadership Awards.
Statistical Methods in Online A/B Testing
Take your A/B testing program to the next level with the most comprehensive book on user testing statistics in e-commerce.
Learn moreGlossary index by letter
Select a letter to see all A/B testing terms starting with that letter or visit the Glossary homepage to see all.