What does "False Negative Rate" mean?

Definition of False Negative Rate in the context of A/B testing (online controlled experiments).

What is a False Negative Rate?

False negative rate refers to the proportion of significance tests that failed to reject the null hypothesis when the null hypothesis is indeed false. The beta of of a test procedure provides a conservative (upper) bound on the rate of type II error in tests performed using this procedure relative to an effect size of certain magnitude, or higher. Assessing the actual false negative rate is impossible unless the proportion of nulls which are actually false is known.

Running tests with higher statistical power to detect smaller effect sizes will improve the rate of false negatives at the cost of larger sample size / longer test duration, assuming there are at least some false null hypotheses. If there are no false nulls, the false negative rate will be 0 regardless of any characteristics of the test procedure.

Related A/B Testing terms

Type II Error

False Discovery Rate

Glossary Index by Letter


Select a letter to see all A/B testing terms starting with that letter or visit the Glossary homepage to see all.