What does "Effect Size" mean?

What is Effect Size?

Effect size is the magnitude of the difference between a hypothesized and an observed value of a parameter. The parameter is usually an unknown variable theta (θ) and can be either a primary KPI or secondary KPI. In power analysis we speak of the minimum effect of interest which is the effect size and the magnitude of the minimum change that would be interesting to be detected (from a business perspective).

Since in an A/B test we are comparing one or more treatment groups with a control group the effect size is equivalent to the magnitude of the difference between a treatment group and the control group (delta δ). The difference we are usually interested is percent change (a.k.a. percentage lift, percentage difference and rarely relative differnce) but sometimes we might prefer to do a significance test for the absolute difference.

The observed effect size in an A/B test has a tight relationship to the p-value and its sample size: very large effect sizes can lead to very small p-values even with moderate-to-small sample sizes, while a very large sample size would allow us to get a statistically significant outcome even with a very small effect size. In A/B testing and hypothesis testing in general failure to understand the above leads to fallacies of rejection or fallacies of acceptance.

Like this glossary entry? For an in-depth and comprehensive reading on A/B testing stats, check out the book "Statistical Methods in Online A/B Testing" by the author of this glossary, Georgi Georgiev.

Articles on Effect Size

What if the Observed Effect is Smaller Than the MDE?
blog.analytics-toolkit.com