What does "Error-Spending Function" mean?

Definition of Error-Spending Function in the context of A/B testing (online controlled experiments).

What is an Error-Spending Function?

An error-spending function is a function that governs the cumulative type I or type II error that is "spent" at each analysis time during a sequential testing online experiment. A function that governs the rate at which the type I error (alpha) is spent is called and Alpha-Spending while one which controls the rate at which the type II error (beta) is spent is called Beta-Spending. An AGILE A/B test is an example for an approach that employs both Alpha- and Beta-Spending functions.

Error spending functions were developed in the 1980s by Lan, Kim and DeMets to address the issue of approaches that relied on fixed data evaluation times that were too inflexible to meet real-world demands in most practical situations. The advantage of having a spending function over the whole duration of an experiment is that an evaluation can be performed pretty much at will with no or minimum impact on the relevant error rate.

While there are spending functions that mimic established procedures for fixed-time analyses: O'Brien-Fleming type, Pocock-type, etc., other spending functions have also been developed to cover all possible needs, such as the Kim & DeMets power function and the Shih, Hwang & De Cani function.

Articles on Error-Spending Function

Like this glossary entry? For an in-depth and comprehensive reading on A/B testing stats, check out the book "Statistical Methods in Online A/B Testing" by the author of this glossary, Georgi Georgiev.

Purchase Statistical Methods in Online A/B Testing

Glossary Index by Letter


Select a letter to see all A/B testing terms starting with that letter or visit the Glossary homepage to see all.