What does "Optional Stopping" mean?
Definition of Optional Stopping in the context of A/B testing (online controlled experiments).
What is Optional Stopping?
Optional stopping is the practice of stopping an online controlled experiment (a.k.a. A/B test) based on its outcome at some interim analysis time t. For example, one can stop an experiment if at time t a certain effect size has been observed. The result of optional stopping is that the stopping time of a test becomes a random variable itself and thus proper sequential testing procedures are required in order to keep the error probability of the test intact.
Optional stopping, when done with appropriate adjustments that often come in the form of an efficacy boundary and futility boundary can be a powerful method that significantly decreases test duration and thus makes more more efficient tests and better ROI.
On the other hand, unaccounted optional stopping with intent is called peeking and is a serious violation of the assumption of the statistical model / the null hypothesis. In such a case the nominal p-value would be significantly different than the actual p-value.
Like this glossary entry? For an in-depth and comprehensive reading on A/B testing stats, check out the book "Statistical Methods in Online A/B Testing" by the author of this glossary, Georgi Georgiev.
Articles on Optional Stopping
Error Spending in Sequential Testing Explained
blog.analytics-toolkit.com
20-80% Faster A/B Tests? Is it real?
blog.analytics-toolkit.com
The Bane of AB Testing: Reaching Statistical Significance
blog.analytics-toolkit.com
Related A/B Testing terms
PeekingSequential TestingAbout the author
Georgi has over twenty years of experience in online marketing, web analytics, statistics, and design of business experiments.
Author of the book "Statistical Methods in Online A/B Testing", white papers on statistical analysis of A/B tests, and a speaker, he has been distinguished as a winner in the Data & Analytics category of the 2024 Experimentation Thought Leadership Awards.
Statistical Methods in Online A/B Testing
Take your A/B testing program to the next level with the most comprehensive book on user testing statistics in e-commerce.
Learn moreGlossary index by letter
Select a letter to see all A/B testing terms starting with that letter or visit the Glossary homepage to see all.