What does "Holdout Group" mean?

What is a Holdout Group?

A holdout group is a group of users who are "held back" from entering into any experiments performed on a website and kept as a global control group. The purpose of a holdout group is to compare the performance of a website, app or other software as it would be without conversion rate optimization efforts versus the effects of those efforts. To serve its purpose it must also not be subject to any changes introduced to the website.

Usually the cost of maintaining such a group are prohibitive and thus its sample size is kept small (<5% of all users) which also makes it hard to draw reliable conclusions from its data.

Hold-out groups are very prone to "pollution" due to cookie churn if there is no persistent user identity across devices and over time and cookies are the only way to differentiate one user from another. This can introduce significant bias in any estimate based on such groups. A further issue concerns the survivorship bias and selection bias that are introduced by not adding new members to the group which cannot be countered easily by adding new users to the holdout group over time as this would negate the logic of its existence.

Like this glossary entry? For an in-depth and comprehensive reading on A/B testing stats, check out the book "Statistical Methods in Online A/B Testing" by the author of this glossary, Georgi Georgiev.