## What is Optional Stopping?

Optional stopping is the practice of stopping an online controlled experiment (a.k.a. A/B test) based on its outcome at some interim analysis time t. For example, one can stop an experiment if at time t a certain effect size has been observed. The result of optional stopping is that the stopping time of a test becomes a random variable itself and thus proper sequential testing procedures are required in order to keep the error probability of the test intact.

Optional stopping, when done with appropriate adjustments that often come in the form of an efficacy boundary and futility boundary can be a powerful method that significantly decreases test duration and thus makes more more efficient tests and better ROI.

On the other hand, unaccounted optional stopping with intent is called peeking and is a serious violation of the assumption of the statistical model / the null hypothesis. In such a case the nominal p-value would be significantly different than the actual p-value.

## Articles on Optional Stopping

- Error Spending in Sequential Testing Explained
- 20-80% Faster A/B Tests? Is it real?
- The Bane of AB Testing: Reaching Statistical Significance

Like this glossary entry? For an in-depth and comprehensive reading on A/B testing stats, check out the book "Statistical Methods in Online A/B Testing" by the author of this glossary, Georgi Georgiev.