What does "Beta-Spending" mean?

What is Beta-Spending?

Alias: beta-spending function

Beta-spending is an extension of the alpha-spending approach to the distribution (spending) of the type II error (denoted beta) over the duration of a sequential A/B test. Beta-spending makes it possible to perform sequential testing with early stopping for futility: when continuing the test is unlikely to produce a statistically significant outcome. This allows a conversion rate optimization practitioner to fail fast and move on to other experiments while exposing as little of the user base to inferior or non-superior interventions as possible.

Usually an error-spending function is employed which is an increasing function of the proportion of the maximum sample size of the experiment. A beta-spending function calculates the cumulative type II error spent up until the time of a observation and thus governs the allocation of beta at that particular point in time.

From the point of generalizability and acquiring representative samples a power spending function should be convex: starting to spend slowly in the early stages of a test, then spending more rapidly around mid-way through and finally slowing down spending towards the end. This way the test has the highest probability of correctly rejecting a false null hypothesis when the sample is representative enough and external validity issues are less likely.

An AGILE A/B test is an example for a testing method that includes both Alpha- and Beta-Spending functions.

Like this glossary entry? For an in-depth and comprehensive reading on A/B testing stats, check out the book "Statistical Methods in Online A/B Testing" by the author of this glossary, Georgi Georgiev.

Articles on Beta-Spending

Error Spending in Sequential Testing Explained
blog.analytics-toolkit.com

Improving ROI in A/B Testing: the AGILE AB Testing Approach
blog.analytics-toolkit.com

Efficient AB Testing with the AGILE Statistical Method
blog.analytics-toolkit.com