What does "alpha" mean?

Definition of alpha in the context of A/B testing (online controlled experiments).

What is alpha?

The greek leter alpha (α) is most-often used to denote the pre-test significance threshold deemed acceptable for the test at hand.

In proper notation α = P(d(X) > c(&alpha); H0) where H0 stands for the null hypothesis of interest, c(&alpha) is the significance threshold and d(X) is a test statistic (distance function).

In case the significance level after the test is complete is equal to α the two coincide. In case the observed significance (p) is different, the test can be interpreted as rejecting the null hypothesis at all levels α for which p < α.

The error rates alpha (α) and beta (β) are inversely related: increasing one decreases the other, assuming fixed variance, sample size and minimum effect of interest.

Articles on alpha

Like this glossary entry? For an in-depth and comprehensive reading on A/B testing stats, check out the book "Statistical Methods in Online A/B Testing" by the author of this glossary, Georgi Georgiev.

Purchase Statistical Methods in Online A/B Testing

Glossary Index by Letter


Select a letter to see all A/B testing terms starting with that letter or visit the Glossary homepage to see all.