What does "Control Group" mean?

What is a Control Group?

The Control group in an online controlled experiment is a group of (usually randomly assigned) users/sessions/pageviews/etc. who will not be exposed to the experimental treatment(s). The performance of the Test Group(s) is compared to that of the Control group to check for discrepancies large enough to reject the null hypothesis.

A randomly assigned control group is essential to any A/B test since it allows one to model the effect of an unlimited number of influences (covariates, confounders) as random noise with a known distribution (typically normal) which is a prerequisite for the use of many statistical methods. Put simply, the control group is what allows one to separate the effect of the tested intervention from the effects of everything else that might be happening at the same time to the experimental population. Randomization is key in the above process.

In most tests the control and test groups are of equal size as it facilitates easier interpretation due to lack of generalizability issues, is easier to compute and is usually an optimal design. In an adaptive sequential design or a multi-armed bandit design the weights of the control and test groups are usually different and vary across the test duration.

Like this glossary entry? For an in-depth and comprehensive reading on A/B testing stats, check out the book "Statistical Methods in Online A/B Testing" by the author of this glossary, Georgi Georgiev.