What does "Novelty Effect" mean?

What is a Novelty Effect?

The term "novelty effect" is usually used to describe a positive effect that is entirely due to fact that there is a change, a new design feature, module, or process being introduced, regardless of what the change is. A novelty effect wears off in time and is thus considered "illusory", meaning that it would be wrong to ascribe the effect to the change itself and to expect that it will continue to persist after the novelty effect wears off. Novelty effects can be considered a case of learning effects and are a major treat to the external validity (generalizability) of online controlled experiments.

When planning an A/B test it is usually useful to consider a longer test duration for more dramatic changes in order to allow for more time for returning users to cool down from any initial positive reaction or spike of interest due to a change that was introduced as part of a treatment. This should help ensure a more representative sample and thus better predictive value for the test outcome.

Like this glossary entry? For an in-depth and comprehensive reading on A/B testing stats, check out the book "Statistical Methods in Online A/B Testing" by the author of this glossary, Georgi Georgiev.

Articles on Novelty Effect

Representative samples and generalizability of A/B testing results
blog.analytics-toolkit.com