What does "Futility Boundary" mean?

What is a Futility Boundary?

A futility boundary (futility stopping boundary, rarely futility limit) is a statistical decision boundary used in sequential testing such as an AGILE A/B test. It is built in such a way that it maintains the type II error probability (beta β) larger than a specified level, on average. It is usually computed using a beta spending function. Crossing the boundary means that the probability of detecting a statistically significant outcome has fallen below the desired.

There are two types of futility bounds: binding and non-binding. A binding futility bound is calculated in such a way, that the error guarantees hold if any boundary cross results in an immediate termination of the A/B test. Failure to do so will affect the error probabilities. Since statistical power and type II errors are more complex than the type I error, it is often desirable to have the option to continue running the test. External reasons can also impact the decision to continue a test even though the futility boundary has been crossed. Non-binding futility boundaries are constructed in cases where the above cases are concerned.

Similarly to efficacy boundaries the error-spending function allows for significant flexibility in the timing and number of analyses, although very significant departures from the initial design may lead to the two boundaries not crossing, resulting in an indecision region at the end of the test.

Like this glossary entry? For an in-depth and comprehensive reading on A/B testing stats, check out the book "Statistical Methods in Online A/B Testing" by the author of this glossary, Georgi Georgiev.

Articles on Futility Boundary

Futility Stopping Rules in AGILE A/B Testing
blog.analytics-toolkit.com

Error Spending in Sequential Testing Explained
blog.analytics-toolkit.com