## What is Segmentation?

Aliases: *subgroup analysis*

Segmentation in an A/B test is the practice of **analyzing the performance of a primary or secondary outcome variable for particular sub-groups** of the users in the experiment. For example, one might compare the effect of the treatment based on the type of device used (desktop, mobile, tablet), the geographical location of the user, the user browser, demographic information, historical interactions with the website or business and so on.

The benefit of analyzing segments is that they can uncover highly uneven effect sizes and even effects of opposite sign which might be cancelling each other out in the overall metric. Segmentation is valuable when one can act on any observed significant difference. For example segmenting by device type or location is useful since, to an extent, the user experience can be tailored based on these parameters. Segmenting based on non-actionable parameters is only useful in an inferential manner.

Performing a significance test on multiple segments amounts to multiple testing if one would act on the test as a whole based on the performance of a singe segment. p-value adjustments using the Sidak correction would be necessary in this case. If one would act only on the segment analyzed the situation becomes a bit more complex: on one hand, one is only considering the data for that segment in the p-value calculation, but on the other a CRO should still be mindful of the fact that by doing significance tests on many segments the probability of some of them to come up statistically significant is quite high, even if there is no particular effect in any segment.

A sound approach seems to be to actually consider the analysis of each segment as a more or less separate test, performing a risk-reward analysis to establish an acceptable significance threshold based on the particular segment. Naturally, it would be best if this is done before the main A/B test is performed, but it increases the complexity of the overall design. If done entirely post-hoc such a form risk/reward analysis can still be helpful in determining what level of significance would be acceptable for this particular segment given the cost of implementation and the risk if things go wrong.

When analyzing segments who are unevenly weighted a conversion rate optimization specialist should be aware of the potential for the Simpson's paradox.