What does "Alpha-Spending" mean?

Definition of Alpha-Spending in the context of A/B testing (online controlled experiments).

What is Alpha-Spending?

Aliases: alpha-spending function

Alpha-spending is an approach of distributing (spending) the type I error (denoted alpha) over the duration of a sequential A/B test. Alpha-spending makes it possible to perform sequential testing while maintaining the overall error probability of the procedure. The ability to perform sequential monitoring of the data and to make decisions in case of significant departures from the minimum effect of interest it was planned for is a desirable quality as it allows for faster decision making.

Usually an error-spending function is employed which is an increasing function of the proportion of the maximum sample size of the experiment. An alpha-spending function calculates the cumulative type I error spent up until the time of a observation and thus governs the allocation of alpha at that particular point in time. Notable examples of such functions include O'Brien-Fleming type, Pocock-type, Kim & DeMets power functions and the Shih, Hwang & De Cani function.

From the point of generalizability and acquiring representative samples a spending function should be convex: starting to spend slowly in the early stages of a test, then spending more rapidly around mid-way through and finally slowing down spending towards the end. This way only very extreme outcomes would result in the test being terminated very early with a low sample size that might have some external validity issues.

An AGILE A/B test is an example for a testing method that includes both Alpha- and Beta-Spending functions.

Related A/B Testing terms

alpha

Efficacy Boundary

Sequential Testing

Articles on Alpha-Spending


Glossary Index by Letter

ABCDEFGHIKLMNOPRSTUVZ

Select a letter to see all A/B testing terms starting with that letter or visit the Glossary homepage to see all.