What does "Observed Power" mean?
Definition of Observed Power in the context of A/B testing (online controlled experiments).
What is Observed Power?
Aliases: post hoc power, retrospective power
Observed power is the level of statistical power of a statistical test calculated for the observed effect size. It is the probability of observing a statistically significant result at level alpha (α) if a true effect equal to the observed effect is in fact present. It is also known as post hoc power or retrospective power.
Observed is a direct function of the p-value and the chosen significance threshold so it adds nothing that these concepts do not already convey. That is just one of the reasons why there are no legitimate uses for observed power. It is misused almost by definition.
Common mistakes include concluding a test is underpowered or overpowered, extending a test until it achieves a particular threshold (which is peeking), or concluding there is a true effect which the test was not sensitive enough to pick up. From a different angle, it may lead to a desire for the observed effect size to be larger than the MDE used in planning the test. All of these are discussed in detail in the references below.
Like this glossary entry? For an in-depth and comprehensive reading on A/B testing stats, check out the book "Statistical Methods in Online A/B Testing" by the author of this glossary, Georgi Georgiev.
Articles on Observed Power
A Comprehensive Guide to Observed Power (Post Hoc Power)
blog.analytics-toolkit.com
Using Observed Power in Online A/B Tests
blog.analytics-toolkit.com
What if the Observed Effect is Smaller Than the MDE?
blog.analytics-toolkit.com
Underpowered A/B Tests – Confusions, Myths, and Reality
blog.analytics-toolkit.com
Related A/B Testing terms
Statistical PowerType II ErrorbetaOverpowered TestUnderpowered TestHypothesis TestingSee this in action
Power & MDE Calculator
About the author
Georgi has over twenty years of experience in online marketing, web analytics, statistics, and design of business experiments.
Author of the book "Statistical Methods in Online A/B Testing", white papers on statistical analysis of A/B tests, and a speaker, he has been distinguished as a winner in the Data & Analytics category of the 2024 Experimentation Thought Leadership Awards.
Statistical Methods in Online A/B Testing
Take your A/B testing program to the next level with the most comprehensive book on user testing statistics in e-commerce.
Learn moreGlossary index by letter
Select a letter to see all A/B testing terms starting with that letter or visit the Glossary homepage to see all.