Issues with Current Bayesian Approaches to A/B Testing in Conversion Rate Optimization

Author
Georgi Z. Georgiev

Abstract

This paper covers what the author perceives as major issues with the current (as of late 2016) mainstream approaches to statistical design and statistical analysis of A/B testing experiments, mostly as applied in fields of Conversion Rate Optimization (CRO) and Landing Page Optimization (LPO).

Fundamental issues with applying Bayesian inferential procedures to the case of A/B testing are presented and examined alongside concrete issues with particular implementations – most notably those by current industry leaders VWO and Optimizely.

The paper is not meant to be the final word on the matter, especially when it comes to the particular implementations, not least due to details about those implementations are often lacking or insufficient for complete evaluation. This in itself, is a non-negligible issue when it comes to Bayesian approaches, as is argued in the paper.

Keywords

bayesian inference, a/b testing, split testing, conversion rate optimization, landing page optimization, bayesian testing, statistical design of online experiments