What does "Beta-Spending" mean?

Definition of Beta-Spending in the context of A/B testing (online controlled experiments).

What is Beta-Spending?

Alias: beta-spending function

Beta-spending is an extension of the alpha-spending approach to the distribution (spending) of the type II error (denoted beta) over the duration of a sequential A/B test. Beta-spending makes it possible to perform sequential testing with early stopping for futility: when continuing the test is unlikely to produce a statistically significant outcome. This allows a conversion rate optimization practitioner to fail fast and move on to other experiments while exposing as little of the user base to inferior or non-superior interventions as possible.

Usually an error-spending function is employed which is an increasing function of the proportion of the maximum sample size of the experiment. A beta-spending function calculates the cumulative type II error spent up until the time of a observation and thus governs the allocation of beta at that particular point in time.

From the point of generalizability and acquiring representative samples a power spending function should be convex: starting to spend slowly in the early stages of a test, then spending more rapidly around mid-way through and finally slowing down spending towards the end. This way the test has the highest probability of correctly rejecting a false null hypothesis when the sample is representative enough and external validity issues are less likely.

An AGILE A/B test is an example for a testing method that includes both Alpha- and Beta-Spending functions.

Like this glossary entry? For an in-depth and comprehensive reading on A/B testing stats, check out the book "Statistical Methods in Online A/B Testing" by the author of this glossary, Georgi Georgiev.

Articles on Beta-Spending

Error Spending in Sequential Testing Explained

Improving ROI in A/B Testing: the AGILE AB Testing Approach

Efficient AB Testing with the AGILE Statistical Method

Purchase Statistical Methods in Online A/B Testing

Statistical Methods in Online A/B Testing

Take your A/B testing program to the next level with the most comprehensive book on user testing statistics in e-commerce.

Learn more

Glossary index by letter

Select a letter to see all A/B testing terms starting with that letter or visit the Glossary homepage to see all.