What does "Error Probability" mean?
Definition of Error Probability in the context of A/B testing (online controlled experiments).
What is an Error Probability?
Alias: error rate
In statistics, an error probability is the frequency with which a certain probabilistic testing procedure will lead to a type I error or a type II error. In other words, it is the rate of occurrence of an error in a hypothetical infinite repetition of the procedure. In frequentist hypothesis testing the worst case error rates are controlled by the selected levels of alpha and beta.
Using procedures with known error probabilities gives one the ability to employ modus tollens or the argument from coincidence when arguing against a hypothesis. Error probabilities are cornerstone in the error statistics approach to causal inference.
Like this glossary entry? For an in-depth and comprehensive reading on A/B testing stats, check out the book "Statistical Methods in Online A/B Testing" by the author of this glossary, Georgi Georgiev.
About the author
Georgi has over twenty years of experience in online marketing, web analytics, statistics, and design of business experiments.
Author of the book "Statistical Methods in Online A/B Testing", white papers on statistical analysis of A/B tests, and a speaker, he has been distinguished as a winner in the Data & Analytics category of the 2024 Experimentation Thought Leadership Awards.
Statistical Methods in Online A/B Testing
Take your A/B testing program to the next level with the most comprehensive book on user testing statistics in e-commerce.
Learn moreGlossary index by letter
Select a letter to see all A/B testing terms starting with that letter or visit the Glossary homepage to see all.