What does "Frequentist Inference" mean?

What is Frequentist Inference?

Frequentist inference is a collection of error probabilistic methods which allows us to learn from data about the true state of nature in the presence of uncertainty by using model-based inference. It's core goal involves providing error control in the face of uncertainty. It was developed in the early 20-th century by Fisher, Neyman & Pearson, and others, largely replacing the present approaches to statistical inference, among them Bayesian inference.

The core of frequentist inference can best be summed up as such: "Putting forward a statistical model and interpreting the observed data as a realization of the 'idealized' stochastic mechanism constitutes the cornerstone of modern statistical inference."^[1]. Frequentist inference is factual: it operates with the goal of pinpointing the true state of nature regarding a certain phenomenon and all its measures of optimality are based on that goal. The importance of defining a statistical model based on which one can assign probabilities of observing a certain sample value is key in hypothesis testing. Since these result in a well-defined sample space freqentist statistics are also sometimes called sampling methods and they are the gold standard in scientific and applied causal inference and estimation.

In a frequentist framework the finite-sample performance of hypothesis tests and estimators is key, bringing rise to estimators which are finite-sample unbiased, efficient and sufficient, as well as asymptotically consistent. That is: we want to be able to have reliable estimation of the error associated with a piece of data, in finite time, thus decisions based on such data can enjoy certain worst-case error-guarantees. This is in contrast with Bayesian inference methods which exclusively rely on asymptotic performance and offer no finite-sample guarantees.

Frequentist inference is the basis of much of the applied statistics in A/B testing / online controlled experiments, but some vendors and practitioners employ Bayesian inference instead. This is often due to misunderstood or simply false arguments of superiority: no-penalty peeking and better alignment with the issue at hand being the two most prominent ones.

From a philosophical standpoint frequentist error statistics can be viewed as applying the general idea of Popper's falsificationism in probabilistic terms.

References:
[1] Zellner A. (2002) "Simplicity, Inference and Modelling - Keeping it Sophisticatedly Simple"

Like this glossary entry? For an in-depth and comprehensive reading on A/B testing stats, check out the book "Statistical Methods in Online A/B Testing" by the author of this glossary, Georgi Georgiev.

Articles on Frequentist Inference

Frequentist vs Bayesian Inference
blog.analytics-toolkit.com