What does "Sample Ratio Mismatch" mean?

Definition of Sample Ratio Mismatch in the context of A/B testing (online controlled experiments).

What is Sample Ratio Mismatch?

Aliases: SRM, sample allocation mismatch

Sample Ratio Mismatch (SRM) in online A/B testing means that the observed allocation of users between the test groups statistically significantly differs from the expected allocation under the specified allocation proportions (sample ratio). For example, with expected allocation of 0.5/0.5, observing 120,000 users in the control group versus 118,000 users in the treatment group would constitute a serious case of Sample Ratio Mismatch.

SRM can be detected quite easily using the Chi-Square Goodness of Fit test. If the test results in a sufficiently small p-value, the hypothesis that the actual allocation was produced under the specified allocation can be rejected.

The presence of SRM is a serious issue for any online controlled experiment as it is a signal for the presence of unknown (in terms of size and direction) bias in the results of the test. It can mean that the results are somewhat off, or that they are off to the point of completely reversing the outcome of the A/B test. The usual course of action after detecting SRM in a test is to identify the cause, eliminate it and then rerun the test in order to obtain trustworthy data.

Diagnosing the root cause for an observed Sample Ratio Mismatch can be a very difficult task since the issue can originate due to a variety of reasons. It can be due to a randomizer issue (users, sessions, pageviews, clicks, etc. are not assigned in a truly random fashion from the beginning), or due to technical issues in test delivery (the probability of seeing a test variant differs), or due to telemetry issues (probability of data to be collected differs), or due to reporting bugs. Additionally, measures taken to ensure data quality such as bot filtering, anti-flicker measures, etc. can in fact have a negative impact on the proper allocation of users between the test groups and thus result in SRM.

References:
[1] "Fabijan et al 2019 - Diagnosing Sample Ratio Mismatch in Online Controlled Experiments - A Taxonomy and Rules of Thumb for Practitioners", KDD '19 Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, Pages 2156-2164

Like this glossary entry? For an in-depth and comprehensive reading on A/B testing stats, check out the book "Statistical Methods in Online A/B Testing" by the author of this glossary, Georgi Georgiev.