What does "Frequentist Inference" mean?
Definition of Frequentist Inference in the context of A/B testing (online controlled experiments).
What is Frequentist Inference?
Frequentist inference is a collection of error probabilistic methods which allows us to learn from data about the true state of nature in the presence of uncertainty by using model-based inference. It's core goal involves providing error control in the face of uncertainty. It was developed in the early 20-th century by Fisher, Neyman & Pearson, and others, largely replacing the present approaches to statistical inference, among them Bayesian inference.
The core of frequentist inference can best be summed up as such: "Putting forward a statistical model and interpreting the observed data as a realization of the 'idealized' stochastic mechanism constitutes the cornerstone of modern statistical inference.". Frequentist inference is factual: it operates with the goal of pinpointing the true state of nature regarding a certain phenomenon and all its measures of optimality are based on that goal. The importance of defining a statistical model based on which one can assign probabilities of observing a certain sample value is key in hypothesis testing. Since these result in a well-defined sample space freqentist statistics are also sometimes called sampling methods and they are the gold standard in scientific and applied causal inference and estimation.
In a frequentist framework the finite-sample performance of hypothesis tests and estimators is key, bringing rise to estimators which are finite-sample unbiased, efficient and sufficient, as well as asymptotically consistent. That is: we want to be able to have reliable estimation of the error associated with a piece of data, in finite time, thus decisions based on such data can enjoy certain worst-case error-guarantees. This is in contrast with Bayesian inference methods which exclusively rely on asymptotic performance and offer no finite-sample guarantees.
Frequentist inference is the basis of much of the applied statistics in A/B testing / online controlled experiments, but some vendors and practitioners employ Bayesian inference instead. This is often due to misunderstood or simply false arguments of superiority: no-penalty peeking and better alignment with the issue at hand being the two most prominent ones.
From a philosophical standpoint frequentist error statistics can be viewed as applying the general idea of Popper's falsificationism in probabilistic terms.
 Zellner A. (2002) "Simplicity, Inference and Modelling - Keeping it Sophisticatedly Simple"
Like this glossary entry? For an in-depth and comprehensive reading on A/B testing stats, check out the book "Statistical Methods in Online A/B Testing" by the author of this glossary, Georgi Georgiev.
Articles on Frequentist Inference
Frequentist vs Bayesian Inference
Statistical Methods in Online A/B Testing
Take your A/B testing program to the next level with the most comprehensive book on user testing statistics in e-commerce.Learn more
Glossary index by letter
Select a letter to see all A/B testing terms starting with that letter or visit the Glossary homepage to see all.