What does "False Discovery Rate" mean?

What is a False Discovery Rate?

Alias: FDR

False discovery rate refers to the proportion of significance tests that reject a null hypothesis when it is in fact true. The alpha of of a test procedure provides a conservative (upper) bound on the rate of type I error in tests performed using this procedure. Assessing the actual false discovery rate is impossible unless the proportion of nulls which are actually true is known.

Running tests with higher sample size / longer test duration can result in a larger number of false positives with smaller effect sizes, assuming there are at least some true null hypotheses. If there are no true nulls, the false discovery rate will be 0 regardless of any characteristics of the test procedure.

There are also procedures for controlling the Familiy-Wise Error Rate in the presence of multiple testing: the Benjamini-Hochberg and the Benjamini-Hochberg-Yekutieli False Discovery Rate corrections. They are p-value corrections that limit the proportion of false discoveries to a specified proportion of them. Their application in A/B testing is of limited utility.

Like this glossary entry? For an in-depth and comprehensive reading on A/B testing stats, check out the book "Statistical Methods in Online A/B Testing" by the author of this glossary, Georgi Georgiev.