What does "Bias" mean?

What is Bias?

Bias in the colloquial sense means having a systemic preference towards something, regardless of the true state of nature, facts, evidence and possibly even regardless of the cost of having this preference versus not having it. There a ton of cognitive biases such as confirmation bias, pro-innovation bias, selection bias, sample size insensitivity, the bandwagon effect, and so on. Understanding of these biases can factor into decisions around what to test in a conversion rate optimization program.

In the statistical sense a bias is the difference between an estimator's expected value and the true value of the parameter being estimated. It is a systemic departure from the true value. A statistical estimator is biased when it shows systemic bias away from the true value (θ*), on average, for a given sample size n or even asymptotically. In other words, if we perform infinitely many estimation procedures with a given sample size n, the arithmetic mean of the estimate from those will over or under-estimate the the true value θ*.

Unbiasedness is thus a desirable property of most statistical estimators, although sometimes a strictly unbiased estimator does not exist, but there might exist a near-unbiased one.

Bias is inversely related to variance: the smaller the bias of an estimator, the larger its variance becomes. A perfectly unbiased estimator thus has the highest variance.

Like this glossary entry? For an in-depth and comprehensive reading on A/B testing stats, check out the book "Statistical Methods in Online A/B Testing" by the author of this glossary, Georgi Georgiev.