What does "Representative Sample" mean?

Definition of Representative Sample in the context of A/B testing (online controlled experiments).

What is a Representative Sample?

A sample is considered representative if it achieves a satisfactory level of generalizability of the outcomes from an online controlled experiment. Having a representative sample of the whole population of interest ensures the external validity of the results and thus has a direct effect on their predictive value.

The most common way to acquire a more representative sample is to increase the sample size and the test duration: gathering more users makes all possible treats to generalizability more susceptible to measure, allowing one to get a better sense of the effect of time-related factors such as seasonality, of the learning effect (novelty effect), as well as to detect any population-level changes.

Gathering more data is not without a costs, mostly opportunity cost and the risk of running with an inferior variant for longer than necessary. It is one of the inevitable trade-offs in A/B testing.

A different kind of trade-off is that while running tests for longer tends to improve external validity it can often compromise internal validity due to things like cookie churn which happens when user identification does not persist across devices and over time.

Like this glossary entry? For an in-depth and comprehensive reading on A/B testing stats, check out the book "Statistical Methods in Online A/B Testing" by the author of this glossary, Georgi Georgiev.

Articles on Representative Sample

Representative samples and generalizability of A/B testing results

Purchase Statistical Methods in Online A/B Testing

Statistical Methods in Online A/B Testing

Take your A/B testing program to the next level with the most comprehensive book on user testing statistics in e-commerce.

Learn more

Glossary index by letter

Select a letter to see all A/B testing terms starting with that letter or visit the Glossary homepage to see all.