What does "Beta-Spending" mean?
Definition of Beta-Spending in the context of A/B testing (online controlled experiments).
What is Beta-Spending?
Alias: beta-spending function
Beta-spending is an extension of the alpha-spending approach to the distribution (spending) of the type II error (denoted beta) over the duration of a sequential A/B test. Beta-spending makes it possible to perform sequential testing with early stopping for futility: when continuing the test is unlikely to produce a statistically significant outcome. This allows a conversion rate optimization practitioner to fail fast and move on to other experiments while exposing as little of the user base to inferior or non-superior interventions as possible.
Usually an error-spending function is employed which is an increasing function of the proportion of the maximum sample size of the experiment. A beta-spending function calculates the cumulative type II error spent up until the time of a observation and thus governs the allocation of beta at that particular point in time.
From the point of generalizability and acquiring representative samples a power spending function should be convex: starting to spend slowly in the early stages of a test, then spending more rapidly around mid-way through and finally slowing down spending towards the end. This way the test has the highest probability of correctly rejecting a false null hypothesis when the sample is representative enough and external validity issues are less likely.
An AGILE A/B test is an example for a testing method that includes both Alpha- and Beta-Spending functions.
Like this glossary entry? For an in-depth and comprehensive reading on A/B testing stats, check out the book "Statistical Methods in Online A/B Testing" by the author of this glossary, Georgi Georgiev.
Articles on Beta-Spending
Error Spending in Sequential Testing Explained
Improving ROI in A/B Testing: the AGILE AB Testing Approach
Efficient AB Testing with the AGILE Statistical Method
Related A/B Testing termsbetaFutility BoundarySequential TestingAlpha-Spending
See this in actionA/B Testing Calculator
Statistical Methods in Online A/B Testing
Take your A/B testing program to the next level with the most comprehensive book on user testing statistics in e-commerce.Learn more
Glossary index by letter
Select a letter to see all A/B testing terms starting with that letter or visit the Glossary homepage to see all.