What does "Alpha-Spending" mean?
Definition of Alpha-Spending in the context of A/B testing (online controlled experiments).
What is Alpha-Spending?
Alias: alpha-spending function
Alpha-spending is an approach of distributing (spending) the type I error (denoted alpha) over the duration of a sequential A/B test. Alpha-spending makes it possible to perform sequential testing while maintaining the overall error probability of the procedure. The ability to perform sequential monitoring of the data and to make decisions in case of significant departures from the minimum effect of interest it was planned for is a desirable quality as it allows for faster decision making.
Usually an error-spending function is employed which is an increasing function of the proportion of the maximum sample size of the experiment. An alpha-spending function calculates the cumulative type I error spent up until the time of a observation and thus governs the allocation of alpha at that particular point in time. Notable examples of such functions include O'Brien-Fleming type, Pocock-type, Kim & DeMets power functions and the Shih, Hwang & De Cani function.
From the point of generalizability and acquiring representative samples a spending function should be convex: starting to spend slowly in the early stages of a test, then spending more rapidly around mid-way through and finally slowing down spending towards the end. This way only very extreme outcomes would result in the test being terminated very early with a low sample size that might have some external validity issues.
An AGILE A/B test is an example for a testing method that includes both Alpha- and Beta-Spending functions.
Like this glossary entry? For an in-depth and comprehensive reading on A/B testing stats, check out the book "Statistical Methods in Online A/B Testing" by the author of this glossary, Georgi Georgiev.
Articles on Alpha-Spending
Error Spending in Sequential Testing Explained
blog.analytics-toolkit.com
Improving ROI in A/B Testing: the AGILE AB Testing Approach
blog.analytics-toolkit.com
Efficient AB Testing with the AGILE Statistical Method
blog.analytics-toolkit.com
Related A/B Testing terms
alphaEfficacy BoundarySequential TestingSee this in action
A/B Testing Calculator
About the author
Georgi has over twenty years of experience in online marketing, web analytics, statistics, and design of business experiments.
Author of the book "Statistical Methods in Online A/B Testing", white papers on statistical analysis of A/B tests, and a speaker, he has been distinguished as a winner in the Data & Analytics category of the 2024 Experimentation Thought Leadership Awards.
Statistical Methods in Online A/B Testing
Take your A/B testing program to the next level with the most comprehensive book on user testing statistics in e-commerce.
Learn moreGlossary index by letter
Select a letter to see all A/B testing terms starting with that letter or visit the Glossary homepage to see all.