What does "Error Probability" mean?

Definition of Error Probability in the context of A/B testing (online controlled experiments).

What is an Error Probability?

Aliases: error rate

In statistics, an error probability is the frequency with which a certain probabilistic testing procedure will lead to a type I error or a type II error. In other words, it is the rate of occurrence of an error in a hypothetical infinite repetition of the procedure. In frequentist hypothesis testing the worst case error rates are controlled by the selected levels of alpha and beta.

Using procedures with known error probabilities gives one the ability to employ modus tollens or the argument from coincidence when arguing against a hypothesis. Error probabilities are cornerstone in the error statistics approach to causal inference.

Like this glossary entry? For an in-depth and comprehensive reading on A/B testing stats, check out the book "Statistical Methods in Online A/B Testing" by the author of this glossary, Georgi Georgiev.

Purchase Statistical Methods in Online A/B Testing

Glossary Index by Letter


Select a letter to see all A/B testing terms starting with that letter or visit the Glossary homepage to see all.