What does "Efficacy Boundary" mean?

What is an Efficacy Boundary?

An efficacy boundary (efficacy stopping boundary, rarely efficacy limit) is a statistical decision boundary used in sequential testing such as an AGILE A/B test. It is constructed so it maintains the type I error probability (alpha α) at a specified level, on average. It is usually computed using an alpha spending function. Crossing the boundary means that the statistical significance threshold has been crossed.

An efficacy boundary always assumes that the A/B test is stopped immediately after a boundary crossing is detected. The use of an error-spending function instead of fixed analyses times allows for significant flexibility in the timing and number of interim analyses, although very significant departures from the initial design may lead to the efficacy and futility boundaries not crossing even at the maximum planned sample size, resulting in an indecision region at the end of the test which is undesirable.

The use of an efficacy boundary results in sample proportions that are biased conditional on the stopping stage: significant positive bias is present early on while somewhat significant negative bias is in effect if a test stops in its latest analysis stages. Bias-corrected estimators are required on order to get a near-unbiased estimator from such a test.

Like this glossary entry? For an in-depth and comprehensive reading on A/B testing stats, check out the book "Statistical Methods in Online A/B Testing" by the author of this glossary, Georgi Georgiev.

Articles on Efficacy Boundary

Improving ROI in A/B Testing: the AGILE AB Testing Approach
blog.analytics-toolkit.com

Error Spending in Sequential Testing Explained
blog.analytics-toolkit.com