What does "Nominal p-value" mean?

Definition of Nominal p-value in the context of A/B testing (online controlled experiments).

What is a Nominal p-value?

Aliases: nominal significance

The nominal p-value is a calculated observed significance based on a given statistical model. When the statistical model reflects the actual test performed the nominal and actual p-value coincide. When the model is inadequate the nominal and actual significance can differ by varying amounts and oftentimes it is not possible to calculate the actual difference.

The nominal p-value may become a meaningless number if the assumptions of the statistical model used to compute it does not hold. These vary from simple things such as performing the calculation with a predefined (fixed) sample size (no peeking) to more complex ones such as requirements about normal distribution of the error, independence and identical distribution of observations, lack of multiple comparisons / multiple testing, and so on. Violating any of the prerequisites of a significance test will render the nominal p-value more or less non-actionable.

Related A/B Testing terms

p-value

Articles on Nominal p-value


Glossary Index by Letter

ABCDEFGHIKLMNOPRSTUVZ

Select a letter to see all A/B testing terms starting with that letter or visit the Glossary homepage to see all.