What does "alpha" mean?
Definition of alpha in the context of A/B testing (online controlled experiments).
What is alpha?
The greek leter alpha (α) is most-often used to denote the pre-test significance threshold deemed acceptable for the test at hand.
In proper notation α = P(d(X) > c(α); H0) where H0 stands for the null hypothesis of interest, c(α) is the significance threshold and d(X) is a test statistic (distance function).
In case the significance level after the test is complete is equal to α the two coincide. In case the observed significance (p) is different, the test can be interpreted as rejecting the null hypothesis at all levels α for which p < α.
The error rates alpha (α) and beta (β) are inversely related: increasing one decreases the other, assuming fixed variance, sample size and minimum effect of interest.
Like this glossary entry? For an in-depth and comprehensive reading on A/B testing stats, check out the book "Statistical Methods in Online A/B Testing" by the author of this glossary, Georgi Georgiev.
Articles on alpha
Statistical Significance in A/B Testing – a Complete Guide
blog.analytics-toolkit.com
Related A/B Testing terms
Statistical SignificanceType I ErrorSignificance LevelSee this in action
Statistical Significance Calculator
About the author
Georgi has over twenty years of experience in online marketing, web analytics, statistics, and design of business experiments.
Author of the book "Statistical Methods in Online A/B Testing", white papers on statistical analysis of A/B tests, and a speaker, he has been distinguished as a winner in the Data & Analytics category of the 2024 Experimentation Thought Leadership Awards.
Statistical Methods in Online A/B Testing
Take your A/B testing program to the next level with the most comprehensive book on user testing statistics in e-commerce.
Learn moreGlossary index by letter
Select a letter to see all A/B testing terms starting with that letter or visit the Glossary homepage to see all.