What does "Test Group" mean?

What is a Test Group?

A Test Groups is a group of (usually randomly assigned) users/sessions/pageviews/etc. that are exposed to a certain treatment. The test groups are then compared to the Control Group to check for discrepancies large enough to reject the null hypothesis of interest.

In most tests the control and test groups are of equal sizes as it facilitates easier interpretation due to lack of generalizability issues, is easier to compute and is usually an optimal design. In an adaptive sequential design or a multi-armed bandit design the weights of the control and test groups are usually different and vary across the test duration.

Adding or dropping test groups mid-experiment should only be done using proper adaptive sequential design methods, otherwise the error probabilities of the A/B test will suffer.

Like this glossary entry? For an in-depth and comprehensive reading on A/B testing stats, check out the book "Statistical Methods in Online A/B Testing" by the author of this glossary, Georgi Georgiev.