What does "Novelty Effect" mean?

Definition of Novelty Effect in the context of A/B testing (online controlled experiments).

What is a Novelty Effect?

The term "novelty effect" is usually used to describe a positive effect that is entirely due to fact that there is a change, a new design feature, module, or process being introduced, regardless of what the change is. A novelty effect wears off in time and is thus considered "illusory", meaning that it would be wrong to ascribe the effect to the change itself and to expect that it will continue to persist after the novelty effect wears off. Novelty effects can be considered a case of learning effects and are a major treat to the external validity (generalizability) of online controlled experiments.

When planning an A/B test it is usually useful to consider a longer test duration for more dramatic changes in order to allow for more time for returning users to cool down from any initial positive reaction or spike of interest due to a change that was introduced as part of a treatment. This should help ensure a more representative sample and thus better predictive value for the test outcome.

Related A/B Testing terms


Learning Effect


Articles on Novelty Effect

Like this glossary entry? For an in-depth and comprehensive reading on A/B testing stats, check out the book "Statistical Methods in Online A/B Testing" by the author of this glossary, Georgi Georgiev.

Purchase Statistical Methods in Online A/B Testing

Glossary Index by Letter


Select a letter to see all A/B testing terms starting with that letter or visit the Glossary homepage to see all.