What does "Nominal p-value" mean?

Definition of Nominal p-value in the context of A/B testing (online controlled experiments).

What is a Nominal p-value?

Alias: nominal significance

The nominal p-value is a calculated observed significance based on a given statistical model. When the statistical model reflects the actual test performed the nominal and actual p-value coincide. When the model is inadequate the nominal and actual significance can differ by varying amounts and oftentimes it is not possible to calculate the actual difference.

The nominal p-value may become a meaningless number if the assumptions of the statistical model used to compute it does not hold. These vary from simple things such as performing the calculation with a predefined (fixed) sample size (no peeking) to more complex ones such as requirements about normal distribution of the error, independence and identical distribution of observations, lack of multiple comparisons / multiple testing, and so on. Violating any of the prerequisites of a significance test will render the nominal p-value more or less non-actionable.

Like this glossary entry? For an in-depth and comprehensive reading on A/B testing stats, check out the book "Statistical Methods in Online A/B Testing" by the author of this glossary, Georgi Georgiev.

Articles on Nominal p-value

The Bane of AB Testing: Reaching Statistical Significance

P-value definition

Related A/B Testing terms

Purchase Statistical Methods in Online A/B Testing

Statistical Methods in Online A/B Testing

Take your A/B testing program to the next level with the most comprehensive book on user testing statistics in e-commerce.

Learn more

Glossary index by letter

Select a letter to see all A/B testing terms starting with that letter or visit the Glossary homepage to see all.