## What is an Overpowered Test?

An overpowered A/B or MVT test is one which has relatively poor probability for detecting a specified minimum effect of interest (MEI). Getting a statistically significant result from such a test is a poor evidence for large discrepancies from the null hypothesis versus observing the same p-value from a less powerful test. This becomes very obvious when examining the upper confidence limit of a one-sided or two-sided confidence interval.

A test can be overpowered with respect to larger alternatives while being properly powered or even underpowered versus alternatives closer to the null hypothesis since power is calculated at a point. By definition in frequentist statistics the smaller the magnitude of the effect of interest, the less powerful the test, all else being fixed.

## Articles on Overpowered Test

Like this glossary entry? For an in-depth and comprehensive reading on A/B testing stats, check out the book "Statistical Methods in Online A/B Testing" by the author of this glossary, Georgi Georgiev.