What does "Segmentation" mean?

Definition of Segmentation in the context of A/B testing (online controlled experiments).

What is Segmentation?

Aliases: subgroup analysis

Segmentation in an A/B test is the practice of analyzing the performance of a primary or secondary outcome variable for particular sub-groups of the users in the experiment. For example, one might compare the effect of the treatment based on the type of device used (desktop, mobile, tablet), the geographical location of the user, the user browser, demographic information, historical interactions with the website or business and so on.

The benefit of analyzing segments is that they can uncover highly uneven effect sizes and even effects of opposite sign which might be cancelling each other out in the overall metric. Segmentation is valuable when one can act on any observed significant difference. For example segmenting by device type or location is useful since, to an extent, the user experience can be tailored based on these parameters. Segmenting based on non-actionable parameters is only useful in an inferential manner.

Performing a significance test on multiple segments amounts to multiple testing if one would act on the test as a whole based on the performance of a singe segment. p-value adjustments using the Sidak correction would be necessary in this case. If one would act only on the segment analyzed the situation becomes a bit more complex: on one hand, one is only considering the data for that segment in the p-value calculation, but on the other a CRO should still be mindful of the fact that by doing significance tests on many segments the probability of some of them to come up statistically significant is quite high, even if there is no particular effect in any segment.

A sound approach seems to be to actually consider the analysis of each segment as a more or less separate test, performing a risk-reward analysis to establish an acceptable significance threshold based on the particular segment. Naturally, it would be best if this is done before the main A/B test is performed, but it increases the complexity of the overall design. If done entirely post-hoc such a form risk/reward analysis can still be helpful in determining what level of significance would be acceptable for this particular segment given the cost of implementation and the risk if things go wrong.

When analyzing segments who are unevenly weighted a conversion rate optimization specialist should be aware of the potential for the Simpson's paradox.

Related A/B Testing terms

Multiple Testing

Articles on Segmentation

Like this glossary entry? For an in-depth and comprehensive reading on A/B testing stats, check out the book "Statistical Methods in Online A/B Testing" by the author of this glossary, Georgi Georgiev.

Purchase Statistical Methods in Online A/B Testing

Glossary Index by Letter


Select a letter to see all A/B testing terms starting with that letter or visit the Glossary homepage to see all.