What does "AGILE A/B Test" mean?

Definition of AGILE A/B Test in the context of A/B testing (online controlled experiments).

What is an AGILE A/B Test?

An AGILE A/B test is an online controlled experiment conducted following the AGILE method as described in the paper "Efficient A/B Testing in Conversion Rate Optimization: The AGILE Statistical Method". It includes a group sequential testing approach with alpha and beta spending functions that define stopping/monitoring boundaries for efficacy and futility. Among its main advantages is the fact that tests conducted under this method can be stopped early due to overwhelmingly positive effects or very low probability of a positive finding (futility). The ability to monitor tests as they go while maintaining type I error control is a highly sought quality among CRO practitioners and stakeholders alike.

The "AGILE" part comes from the fact that the method based on error-spending functions instead of fixed evaluation points allows for great flexibility in monitoring the data as it gathers.

The method also includes prescriptions for dealing with multiple testing in the form of both multiple comparisons as well as using more than one metric to evaluate the test (usually a key performance indicator, KPI) so that the Family-Wise Error Rate is kept in check. Part of the method is the use of conditionally near-unbiased estimators (confidence intervals, point estimators and p-values).

The easiest way to conduct an AGILE A/B test is through using the AGILE A/B testing calculator

Articles on AGILE A/B Test

Glossary Index by Letter


Select a letter to see all A/B testing terms starting with that letter or visit the Glossary homepage to see all.