What does "Holdout Group" mean?

Definition of Holdout Group in the context of A/B testing (online controlled experiments).

What is a Holdout Group?

A holdout group is a group of users who are "held back" from entering into any experiments performed on a website and kep as a global control group. The purpose of a holdout group is to compare the performance of a website, app or other software as it would be without conversion rate optimization efforts versus the effects of those efforts. Usually the cost of maintaining such a group are prohibitive and thus its sample size is kept small (<5% of all users) which also makes it hard to draw reliable conclusions from its data.

Hold-out groups are very prone to "pollution" due to cookie churn if there is no persistent user identity across devices and over time and cookies are the only way to differentiate one user from another. This can introduce significant bias in any estimate based on such groups. A further issue concerns the survivorship bias and selection bias that are introduced by not adding new members to the group which cannot be countered easily by adding new users to the holdout group over time as this would negate the logic of its existence.

Related A/B Testing terms

Control Group


Glossary Index by Letter

ABCDEFGHIKLMNOPRSTUVZ

Select a letter to see all A/B testing terms starting with that letter or visit the Glossary homepage to see all.