Efficient A/B Testing in Conversion Rate Optimization: The AGILE Statistical Method

Author
Georgi Z. Georgiev

Abstract

This paper presents AGILE - an improved A/B testing statistical methodology and accompanying software tool that allows for running conversion rate optimization (CRO) experiments that reach conclusions 20% to 80% faster than traditional methods and solutions while providing the same or better statistical guarantees.

The AGILE A/B Testing method also allows for significant flexibility in monitoring and acting on accruing data by providing rules for early stopping for both efficacy and futility. The introduction of futility stopping is a significant improvement as it allows for early termination of tests with very little statistical chance of proving themselves a success.

The paper outlines current statistical issues and pains in A/B testing for CRO such as data peeking and unwarranted stopping, underpowered tests, multiplicity testing and a brief discussion on the drawbacks and limitations of the currently employed Bayesian methods.

It proceeds with an overview of the statistical foundations for AGILE. Then the method is introduced in detail, followed by thorough guidelines for its application in conducting A/B tests. Throughout the paper conversion rate optimization is used as an example application, but the method can just as easily be applied to experiments in landing page optimization, email marketing optimization, CTR optimization in SEO & PPC, and others.

Finally, full-scale simulations with the AGILE A/B Testing Calculator software for applying this improved statistical method are presented and discussed.

Keywords

a/b testing, split testing, conversion rate optimization, landing page optimization, sequential testing, statistical design of online experiments, alpha spending functions, beta spending functions

Selected visuals