What does "Nominal p-value" mean?
Definition of Nominal p-value in the context of A/B testing (online controlled experiments).
What is a Nominal p-value?
Alias: nominal significance
The nominal p-value is a calculated observed significance based on a given statistical model. When the statistical model reflects the actual test performed the nominal and actual p-value coincide. When the model is inadequate the nominal and actual significance can differ by varying amounts and oftentimes it is not possible to calculate the actual difference.
The nominal p-value may become a meaningless number if the assumptions of the statistical model used to compute it does not hold. These vary from simple things such as performing the calculation with a predefined (fixed) sample size (no peeking) to more complex ones such as requirements about normal distribution of the error, independence and identical distribution of observations, lack of multiple comparisons / multiple testing, and so on. Violating any of the prerequisites of a significance test will render the nominal p-value more or less non-actionable.
Like this glossary entry? For an in-depth and comprehensive reading on A/B testing stats, check out the book "Statistical Methods in Online A/B Testing" by the author of this glossary, Georgi Georgiev.
Articles on Nominal p-value
The Bane of AB Testing: Reaching Statistical Significance
blog.analytics-toolkit.com
P-value definition
www.gigacalculator.com
Related A/B Testing terms
p-valueAbout the author
Georgi has over twenty years of experience in online marketing, web analytics, statistics, and design of business experiments.
Author of the book "Statistical Methods in Online A/B Testing", white papers on statistical analysis of A/B tests, and a speaker, he has been distinguished as a winner in the Data & Analytics category of the 2024 Experimentation Thought Leadership Awards.
Statistical Methods in Online A/B Testing
Take your A/B testing program to the next level with the most comprehensive book on user testing statistics in e-commerce.
Learn moreGlossary index by letter
Select a letter to see all A/B testing terms starting with that letter or visit the Glossary homepage to see all.