What does "AGILE A/B Test" mean?

Definition of AGILE A/B Test in the context of A/B testing (online controlled experiments).

What is an AGILE A/B Test?

An AGILE A/B test is an online controlled experiment conducted following the AGILE method as described in the paper "Efficient A/B Testing in Conversion Rate Optimization: The AGILE Statistical Method". It includes a group sequential testing approach with alpha and beta spending functions that define stopping/monitoring boundaries for efficacy and futility. Among its main advantages is the fact that tests conducted under this method can be stopped early due to overwhelmingly positive effects or very low probability of a positive finding (futility). The ability to monitor tests as they go while maintaining type I error control is a highly sought quality among CRO practitioners and stakeholders alike.

The "AGILE" part comes from the fact that the method based on error-spending functions instead of fixed evaluation points allows for great flexibility in monitoring the data as it gathers.

The method also includes prescriptions for dealing with multiple testing in the form of both multiple comparisons as well as using more than one metric to evaluate the test (usually a key performance indicator, KPI) so that the Family-Wise Error Rate is kept in check. Part of the method is the use of conditionally near-unbiased estimators (confidence intervals, point estimators and p-values).

The easiest way to conduct an AGILE A/B test is through using the AGILE A/B testing calculator

Like this glossary entry? For an in-depth and comprehensive reading on A/B testing stats, check out the book "Statistical Methods in Online A/B Testing" by the author of this glossary, Georgi Georgiev.

Articles on AGILE A/B Test

Improving ROI in A/B Testing: the AGILE AB Testing Approach

Efficient AB Testing with the AGILE Statistical Method

20-80% Faster A/B Tests? Is it real?

Efficient A/B Testing in Conversion Rate Optimization: The AGILE Statistical Method

Error Spending in Sequential Testing Explained

Purchase Statistical Methods in Online A/B Testing

Statistical Methods in Online A/B Testing

Take your A/B testing program to the next level with the most comprehensive book on user testing statistics in e-commerce.

Learn more

Glossary index by letter

Select a letter to see all A/B testing terms starting with that letter or visit the Glossary homepage to see all.