What does "Causal Inference" mean?

What is Causal Inference?

Causal inference is a process by which a causal connection is established based on evidence. In A/B testing this happens through hypothesis testing, usually in the form of a Null Hypothesis Statistical Test. Putting forward a statistical model and interpreting the observed data as a realization of the 'idealized' stochastic mechanism constitutes the cornerstone of modern statistical inference ^[1]. While there are other methods of inferring causal links based on data like Bayesian inference from experimental data as well as quasi-experimental methods based on observational data, frequentist inference remains the gold standard due its entirely factual account of the data (more precisely due to the minimal assumptions involved).

In frequentist inference one relies on the probabilistic version of modus tollens logic: a valid deductive logical device of the form: from hypothesis H follows evidence e, observe not-e, from this follows that H is false. Alternatively, when communicating results one can frame it as a strong argument from coincidence: had the true state of nature been H, then it is an incredible coincidence that we did not observe e. Of course "incredible" is a relative term that must be defined numerically i.e. via a significance threshold.

Causal inference of the frequentist kind shifts the burden of proof on the person arguing for the null hypothesis to the extent to which the data contradicts it, thus data takes central position in the decision-making process.

The frequentist account conforms very well with the general idea of Popper's falsificationism. Prior philosophies of inference include confirmation theory which stemmed from logical positivism and prior epistemological accounts: in essence on justifies the validity of a theory based on the amount of evidence in favor of that theory. Contrast that to faslificationism in which no amount of evidence can ever confirm a theory while a single valid observation in opposition can refute it, therefore passing a test with a higher capacity to produce an observation opposing a hypothesis presents a more rigorous test of that hypothesis if it fails to do so. A test with low such capacity is barely a test at all.

A deeper understanding of causal inference can help a conversion rate optimization specialist to better understand his job and hence to better communicate its results to stakeholders.

References:
[1] Zellner A. (2002) "Simplicity, Inference and Modelling - Keeping it Sophisticatedly Simple"

Like this glossary entry? For an in-depth and comprehensive reading on A/B testing stats, check out the book "Statistical Methods in Online A/B Testing" by the author of this glossary, Georgi Georgiev.