What does "Optional Stopping" mean?

Definition of Optional Stopping in the context of A/B testing (online controlled experiments).

What is Optional Stopping?

Optional stopping is the practice of stopping an online controlled experiment (a.k.a. A/B test) based on its outcome at some interim analysis time t. For example, one can stop an experiment if at time t a certain effect size has been observed. The result of optional stopping is that the stopping time of a test becomes a random variable itself and thus proper sequential testing procedures are required in order to keep the error probability of the test intact.

Optional stopping, when done with appropriate adjustments that often come in the form of an efficacy boundary and futility boundary can be a powerful method that significantly decreases test duration and thus makes more more efficient tests and better ROI.

On the other hand, unaccounted optional stopping with intent is called peeking and is a serious violation of the assumption of the statistical model / the null hypothesis. In such a case the nominal p-value would be significantly different than the actual p-value.

Related A/B Testing terms

Peeking

Sequential Testing

Articles on Optional Stopping


Glossary Index by Letter

ABCDEFGHIKLMNOPRSTUVZ

Select a letter to see all A/B testing terms starting with that letter or visit the Glossary homepage to see all.