What does "Power Analysis" mean?

What is Power Analysis?

Power analysis is defined as the practice of estimating the statistical power of a test with regards to one or more effect sizes of interest. It can involve analytical computations or calculation through simulation for complex cases.

During the analysis a power function can be presented over all possible values of the alternative hypothesis. Usually calculations for key points from the space of the alternative hypothesis are also presented.

If there is no pre-determined time limit for the duration of the A/B or the MVT a power analysis can result in a calculation of the required sample size and thus inform the expected test duration of the experiment. This is most often the case with online controlled experiments in conversion rate optimization.

If there is a time limit on the A/B test the duration and thus the sample size of the test is known and so a power analysis can only serve as a measure of whether the test is to be undertaken (if there is too little power for any reasonably expected effect size, it could be fuitile to run it).

Like this glossary entry? For an in-depth and comprehensive reading on A/B testing stats, check out the book "Statistical Methods in Online A/B Testing" by the author of this glossary, Georgi Georgiev.

Articles on Power Analysis

The Importance of Statistical Power in Online A/B Testing
blog.analytics-toolkit.com