What does "Representative Sample" mean?

What is a Representative Sample?

A sample is considered representative if it achieves a satisfactory level of generalizability of the outcomes from an online controlled experiment. Having a representative sample of the whole population of interest ensures the external validity of the results and thus has a direct effect on their predictive value.

The most common way to acquire a more representative sample is to increase the sample size and the test duration: gathering more users makes all possible treats to generalizability more susceptible to measure, allowing one to get a better sense of the effect of time-related factors such as seasonality, of the learning effect (novelty effect), as well as to detect any population-level changes.

Gathering more data is not without a costs, mostly opportunity cost and the risk of running with an inferior variant for longer than necessary. It is one of the inevitable trade-offs in A/B testing.

A different kind of trade-off is that while running tests for longer tends to improve external validity it can often compromise internal validity due to things like cookie churn which happens when user identification does not persist across devices and over time.

Like this glossary entry? For an in-depth and comprehensive reading on A/B testing stats, check out the book "Statistical Methods in Online A/B Testing" by the author of this glossary, Georgi Georgiev.

Articles on Representative Sample

Representative samples and generalizability of A/B testing results
blog.analytics-toolkit.com