What does "Holdout Group" mean?

Definition of Holdout Group in the context of A/B testing (online controlled experiments).

What is a Holdout Group?

A holdout group is a group of users who are "held back" from entering into any experiments performed on a website and kept as a global control group. The purpose of a holdout group is to compare the performance of a website, app or other software as it would be without conversion rate optimization efforts versus the effects of those efforts. To serve its purpose it must also not be subject to any changes introduced to the website.

Usually the cost of maintaining such a group are prohibitive and thus its sample size is kept small (<5% of all users) which also makes it hard to draw reliable conclusions from its data.

Hold-out groups are very prone to "pollution" due to cookie churn if there is no persistent user identity across devices and over time and cookies are the only way to differentiate one user from another. This can introduce significant bias in any estimate based on such groups. A further issue concerns the survivorship bias and selection bias that are introduced by not adding new members to the group which cannot be countered easily by adding new users to the holdout group over time as this would negate the logic of its existence.

Related A/B Testing terms

Control Group

Like this glossary entry? For an in-depth and comprehensive reading on A/B testing stats, check out the book "Statistical Methods in Online A/B Testing" by the author of this glossary, Georgi Georgiev.

Purchase Statistical Methods in Online A/B Testing

Glossary Index by Letter


Select a letter to see all A/B testing terms starting with that letter or visit the Glossary homepage to see all.