What does "Carry-Over Effect" mean?

What is a Carry-Over Effect?

A carry-over effect is any effect from a previous experimental treatment that carries over onto a period after the experiment has been terminated and subjects are no longer experiencing the treatment. They can either be due to learning effects that are still fading away or due to permanent changes introduced to the treatment population by way of administering the intervention. Carry-over effects are sometimes called spill-over effects, although this term can also have the meaning of interaction effects.

In most A/B tests a practitioner does not have to worry about carry-over effects from prior tests given proper randomization: both control and treatment have an equal chance of including a user with a lingering effect from prior tests.

A/B tests that attempt to establish learning effects may attempt to expose a group first to one treatment and then to another, or to a treatment and then no treatment. In such cases carry-over effects need to be taken into account, especially if it is considered possible that their duration is a significant part of the duration of the post-switch period.

An undesired and unaccounted carry-over effect may appear due to cookie churn. When a users loses his identification cookie they will be randomized again and may end up in a different test group where, in the case of presence of a learning effect there will be a carry-over effect. Such a test violates the statistical model assumptions usually employed and its data is invalidated to the extent to which the carry-over affects the outcome variable(s).

Like this glossary entry? For an in-depth and comprehensive reading on A/B testing stats, check out the book "Statistical Methods in Online A/B Testing" by the author of this glossary, Georgi Georgiev.