What does "False Negative Rate" mean?

Definition of False Negative Rate in the context of A/B testing (online controlled experiments).

What is a False Negative Rate?

False negative rate refers to the proportion of significance tests that failed to reject the null hypothesis when the null hypothesis is indeed false. The beta of of a test procedure provides a conservative (upper) bound on the rate of type II error in tests performed using this procedure relative to an effect size of certain magnitude, or higher. Assessing the actual false negative rate is impossible unless the proportion of nulls which are actually false is known.

Running tests with higher statistical power to detect smaller effect sizes will improve the rate of false negatives at the cost of larger sample size / longer test duration, assuming there are at least some false null hypotheses. If there are no false nulls, the false negative rate will be 0 regardless of any characteristics of the test procedure.

Related A/B Testing terms

Type II Error

False Discovery Rate

Like this glossary entry? For an in-depth and comprehensive reading on A/B testing stats, check out the book "Statistical Methods in Online A/B Testing" by the author of this glossary, Georgi Georgiev.

Purchase Statistical Methods in Online A/B Testing

Glossary Index by Letter


Select a letter to see all A/B testing terms starting with that letter or visit the Glossary homepage to see all.