What does "Test Group" mean?

Definition of Test Group in the context of A/B testing (online controlled experiments).

What is a Test Group?

A Test Groups is a group of (usually randomly assigned) users/sessions/pageviews/etc. that are exposed to a certain treatment. The test groups are then compared to the Control Group to check for discrepancies large enough to reject the null hypothesis of interest.

In most tests the control and test groups are of equal sizes as it facilitates easier interpretation due to lack of generalizability issues, is easier to compute and is usually an optimal design. In an adaptive sequential design or a multi-armed bandit design the weights of the control and test groups are usually different and vary across the test duration.

Adding or dropping test groups mid-experiment should only be done using proper adaptive sequential design methods, otherwise the error probabilities of the A/B test will suffer.

Related A/B Testing terms

Control Group


Like this glossary entry? For an in-depth and comprehensive reading on A/B testing stats, check out the book "Statistical Methods in Online A/B Testing" by the author of this glossary, Georgi Georgiev.


Purchase Statistical Methods in Online A/B Testing


Glossary Index by Letter

ABCDEFGHIKLMNOPRSTUVZ

Select a letter to see all A/B testing terms starting with that letter or visit the Glossary homepage to see all.