What does "False Discovery Rate" mean?
Definition of False Discovery Rate in the context of A/B testing (online controlled experiments).
What is a False Discovery Rate?
False discovery rate refers to the proportion of significance tests that reject a null hypothesis when it is in fact true. The alpha of of a test procedure provides a conservative (upper) bound on the rate of type I error in tests performed using this procedure. Assessing the actual false discovery rate is impossible unless the proportion of nulls which are actually true is known.
Running tests with higher sample size / longer test duration can result in a larger number of false positives with smaller effect sizes, assuming there are at least some true null hypotheses. If there are no true nulls, the false discovery rate will be 0 regardless of any characteristics of the test procedure.
There are also procedures for controlling the Familiy-Wise Error Rate in the presence of multiple testing: the Benjamini-Hochberg and the Benjamini-Hochberg-Yekutieli False Discovery Rate corrections. They are p-value corrections that limit the proportion of false discoveries to a specified proportion of them. Their application in A/B testing is of limited utility.
Like this glossary entry? For an in-depth and comprehensive reading on A/B testing stats, check out the book "Statistical Methods in Online A/B Testing" by the author of this glossary, Georgi Georgiev.
Statistical Methods in Online A/B Testing
Take your A/B testing program to the next level with the most comprehensive book on user testing statistics in e-commerce.Learn more
Glossary index by letter
Select a letter to see all A/B testing terms starting with that letter or visit the Glossary homepage to see all.