What does "False Negative Rate" mean?
Definition of False Negative Rate in the context of A/B testing (online controlled experiments).
What is a False Negative Rate?
False negative rate refers to the proportion of significance tests that failed to reject the null hypothesis when the null hypothesis is indeed false. The beta of of a test procedure provides a conservative (upper) bound on the rate of type II error in tests performed using this procedure relative to an effect size of certain magnitude, or higher. Assessing the actual false negative rate is impossible unless the proportion of nulls which are actually false is known.
Running tests with higher statistical power to detect smaller effect sizes will improve the rate of false negatives at the cost of larger sample size / longer test duration, assuming there are at least some false null hypotheses. If there are no false nulls, the false negative rate will be 0 regardless of any characteristics of the test procedure.
Like this glossary entry? For an in-depth and comprehensive reading on A/B testing stats, check out the book "Statistical Methods in Online A/B Testing" by the author of this glossary, Georgi Georgiev.
Statistical Methods in Online A/B Testing
Take your A/B testing program to the next level with the most comprehensive book on user testing statistics in e-commerce.
Learn moreGlossary index by letter
Select a letter to see all A/B testing terms starting with that letter or visit the Glossary homepage to see all.