What does "Futility Boundary" mean?

Definition of Futility Boundary in the context of A/B testing (online controlled experiments).

What is a Futility Boundary?

A futility boundary (futility stopping boundary, rarely futility limit) is a statistical decision boundary used in sequential testing such as an AGILE A/B test. It is built in such a way that it maintains the type II error probability (beta β) larger than a specified level, on average. It is usually computed using a beta spending function. Crossing the boundary means that the probability of detecting a statistically significant outcome has fallen below the desired.

There are two types of futility bounds: binding and non-binding. A binding futility bound is calculated in such a way, that the error guarantees hold if any boundary cross results in an immediate termination of the A/B test. Failure to do so will affect the error probabilities. Since statistical power and type II errors are more complex than the type I error, it is often desirable to have the option to continue running the test. External reasons can also impact the decision to continue a test even though the futility boundary has been crossed. Non-binding futility boundaries are constructed in cases where the above cases are concerned.

Similarly to efficacy boundaries the error-spending function allows for significant flexibility in the timing and number of analyses, although very significant departures from the initial design may lead to the two boundaries not crossing, resulting in an indecision region at the end of the test.

Articles on Futility Boundary


Like this glossary entry? For an in-depth and comprehensive reading on A/B testing stats, check out the book "Statistical Methods in Online A/B Testing" by the author of this glossary, Georgi Georgiev.


Purchase Statistical Methods in Online A/B Testing


Glossary Index by Letter

ABCDEFGHIKLMNOPRSTUVZ

Select a letter to see all A/B testing terms starting with that letter or visit the Glossary homepage to see all.