What does "Selection Bias" mean?

Definition of Selection Bias in the context of A/B testing (online controlled experiments).

What is Selection Bias?

Selection bias is a systematic under or overestimation of an effect on a parameter of interest due to the difference between the population that is exposed to some of the treatment groups and the rest. In extreme cases a result can be biased to the point that the whole observed effect size is due to it and there is no genuine effect at all.

An example for selection bias in an A/B test is if the A/B testing software that randomizes users into the groups and delivers the experiences is not giving each user an equal chance to end up in all of the test groups. A/B testing software based on JavaScript repainting of the page on the client side is known to introduce "load flicker": a visible reordering of the page after it has been loaded for the user in case of a delay in delivering an experimental intervention. This naturally alters the user experience beyond the intended intervention and the result is no longer testing the substantive hypothesis - the statistical model becomes inadequate vis-a-vis the actual experiment being performed. Such an experiment will be biased against any intervention.

The opposite selection bias is present in some commercial software where in an attempt to reduce "load flicker". In case of a significant the delay in displaying the treatment that would have caused flicker the user is instead shown the control and re-assigned to the control group. By acting as if the user has been randomized to the control group when they were not, the software is biasing the experimental outcome against the control group since, for example, a user who experiences a slower load time is more likely to have a slower machine, slower connection, be on a mobile device, be in a remote geographical region relative to the server infrastructure, and so on: all factors that are known to be somewhat negatively correlated with metrics like average revenue per user and conversion rate.

Related A/B Testing terms


Survivorship Bias

Like this glossary entry? For an in-depth and comprehensive reading on A/B testing stats, check out the book "Statistical Methods in Online A/B Testing" by the author of this glossary, Georgi Georgiev.

Purchase Statistical Methods in Online A/B Testing

Glossary Index by Letter


Select a letter to see all A/B testing terms starting with that letter or visit the Glossary homepage to see all.