What does "A/B Test" mean?
Definition of A/B Test in the context of A/B testing (online controlled experiments).
What is an A/B Test?
Alias: split test
An A/B test, a.k.a. a split test is a simple form of an online controlled experiment in which one user experience is tested versus another. The name comes from the convention of labeling one experience "A" and the other "B". One, usually the default, current or preferred experience is designated as the control (A) and it is then tested against another, hopefully better in some way with randomization being used to assign users/sessions/pageviews/etc. to one or the other.
The A/B test is a simple yet powerful tool for making causal claims in the toolbox of online marketing and user experience specialists so it is no wonder that over the years it has become a bit detached from its strict meaning and is often used to refer to any kind of online experimentation like a Multivariate Test (A/B/n) or even a Factorial Design.
We can distinguish between two main goals of performing an A/B test: controlling the business risk associated with a given proposed change (hypothesis testing) as well as estimating the magnitude and direction of effect (estimation). Often one seeks to achieve both and usually the former is the primary goal and the latter the secondary.
The business value of running randomized controlled experiments comes from its ability to block poor solutions from being implemented while green-lighting beneficial changes, resulting in multiple times better business outcomes compared to releasing changes without testing (see reference 2).
Like this glossary entry? For an in-depth and comprehensive reading on A/B testing stats, check out the book "Statistical Methods in Online A/B Testing" by the author of this glossary, Georgi Georgiev.
Articles on A/B Test
A guide to A/B testing statistics
blog.analytics-toolkit.com
The Business Value of A/B Testing
blog.analytics-toolkit.com
Risk vs. Reward in A/B Tests: A/B testing as Risk Management
blog.analytics-toolkit.com
A/B Testing with a Small Sample Size
blog.analytics-toolkit.com
The Cost of Not A/B Testing – a Case Study
blog.analytics-toolkit.com
Related A/B Testing terms
Multivariate TestOnline Controlled ExperimentRandomizationA/B TestingSee this in action
A/B Testing Calculator
Statistical Significance Calculator
About the author
Georgi has over twenty years of experience in online marketing, web analytics, statistics, and design of business experiments.
Author of the book "Statistical Methods in Online A/B Testing", white papers on statistical analysis of A/B tests, and a speaker, he has been distinguished as a winner in the Data & Analytics category of the 2024 Experimentation Thought Leadership Awards.
Statistical Methods in Online A/B Testing
Take your A/B testing program to the next level with the most comprehensive book on user testing statistics in e-commerce.
Learn moreGlossary index by letter
Select a letter to see all A/B testing terms starting with that letter or visit the Glossary homepage to see all.