What does "Efficacy Boundary" mean?

Definition of Efficacy Boundary in the context of A/B testing (online controlled experiments).

What is an Efficacy Boundary?

An efficacy boundary (efficacy stopping boundary, rarely efficacy limit) is a statistical decision boundary used in sequential testing such as an AGILE A/B test. It is constructed so it maintains the type I error probability (alpha α) at a specified level, on average. It is usually computed using an alpha spending function. Crossing the boundary means that the statistical significance threshold has been crossed.

An efficacy boundary always assumes that the A/B test is stopped immediately after a boundary crossing is detected. The use of an error-spending function instead of fixed analyses times allows for significant flexibility in the timing and number of interim analyses, although very significant departures from the initial design may lead to the efficacy and futility boundaries not crossing even at the maximum planned sample size, resulting in an indecision region at the end of the test which is undesirable.

The use of an efficacy boundary results in sample proportions that are biased conditional on the stopping stage: significant positive bias is present early on while somewhat significant negative bias is in effect if a test stops in its latest analysis stages. Bias-corrected estimators are required on order to get a near-unbiased estimator from such a test.

Articles on Efficacy Boundary


Glossary Index by Letter

ABCDEFGHIKLMNOPRSTUVZ

Select a letter to see all A/B testing terms starting with that letter or visit the Glossary homepage to see all.