What does "Cookie Churn" mean?

Definition of Cookie Churn in the context of A/B testing (online controlled experiments).

What is Cookie Churn?

Cookie churn or cookie churn rate is the rate at which cookies are deleted from user's browser over a period of time, usually weeks or months. Cookie churn is important in experiments where the users cannot be uniquely identified across browsers and devices and the only way to persist an experiment treatment across sessions is to rely on a cookie identifier. This is an issue since once a cookie is deleted and a user returns to the website they will get a new cookie ID and be randomized (see randomization) again and may be assigned to a different test group, polluting the data.

This leads to cross-over effects due to inconsistent experience and the statistical model is no longer adequate vis-a-vis the actual experiment. In particular, the assumption that the sample observations of the parameter of interest (whatever key performance indicator one is measuring) are an independent variable is no longer valid. The severity of the violation may not be proportional to the severity with which the outcome is affected.

The rate of cookie churn / cookie deletion is hard to estimate, though some estimate it at 75% per two-month period while others estimate it at 25-33% on a monthly basis [1].

Cookie churn can happen due to a multitude of reasons. For example, browsers allow so-called private/incognito browsing during which persistent cookies and even session cookies are not transferred from regular browsing. Persistent cookies are also not set when in such a mode. Another issue is browser settings such as "delete all cookies on exit" and the like as well as the simple action of manually deleting cookies. Anti-tracking software further exacerbates the issue.

For online controlled experiments that only include users based on a pool of existing cookies and do not allow new users to enter the experiment cookie churn is also an issue since it increases selection bias - as duration of the study increases users who keep their cookies become less and less representative of the overall population.

This can also cause the issue of survivorship bias on "per user" metrics if these are evaluated based on a selection of users, e.g. those who engaged with the website over the last half of the test, and not on all users who enter the experiment since as time goes by users who dislike the intervention will decrease the frequency of their visits or stop visiting the website altogether while the remaining will have improved metrics simply due to them "surviving" the treatment. In such cases the cookie refresh rate can also bias the results if it differs across interventions due to the nature of the intervention.

References:
[1] Dmitriev P. et al. (2016) "Pitfalls of Long-Term Online Controlled Experiments"


Glossary Index by Letter

ABCDEFGHIKLMNOPRSTUVZ

Select a letter to see all A/B testing terms starting with that letter or visit the Glossary homepage to see all.