What does "Error-Spending Function" mean?

What is an Error-Spending Function?

An error-spending function is a function that governs the cumulative type I or type II error that is "spent" at each analysis time during a sequential testing online experiment. A function that governs the rate at which the type I error (alpha) is spent is called and Alpha-Spending while one which controls the rate at which the type II error (beta) is spent is called Beta-Spending. An AGILE A/B test is an example for an approach that employs both Alpha- and Beta-Spending functions.

Error spending functions were developed in the 1980s by Lan, Kim and DeMets to address the issue of approaches that relied on fixed data evaluation times that were too inflexible to meet real-world demands in most practical situations. The advantage of having a spending function over the whole duration of an experiment is that an evaluation can be performed pretty much at will with no or minimum impact on the relevant error rate.

While there are spending functions that mimic established procedures for fixed-time analyses: O'Brien-Fleming type, Pocock-type, etc., other spending functions have also been developed to cover all possible needs, such as the Kim & DeMets power function and the Shih, Hwang & De Cani function.

Like this glossary entry? For an in-depth and comprehensive reading on A/B testing stats, check out the book "Statistical Methods in Online A/B Testing" by the author of this glossary, Georgi Georgiev.

Articles on Error-Spending Function

Error Spending in Sequential Testing Explained
blog.analytics-toolkit.com

20-80% Faster A/B Tests? Is it real?
blog.analytics-toolkit.com

Efficient A/B Testing in Conversion Rate Optimization: The AGILE Statistical Method
www.analytics-toolkit.com