What does "Family-Wise Error Rate" mean?

Definition of Family-Wise Error Rate in the context of A/B testing (online controlled experiments).

What is Family-Wise Error Rate?

Aliases: FWER

The Family-Wise Error Rate (FWER) is a concept describing the overall error rate under a composite null of no effect across a series of significance tests. There are many situations in which this can occur with the most common one being conducting a Multivariate Test which results in multiple comparisons: between the control group and each of the treatment groups. Other examples of multiple testing include testing more than one Key Performance Indicator (KPI) with a change in one of them deemed enough to call the test, as well as performing a series of significance tests on accumulating data such as in sequential testing.

There are a number of statistical methods developed to control the type I error at the family level, where family refers to a logical group of tests. Such methods include the Sidak correction, Dunnett's correction and the Bonferroni correction and each is appropriate under different assumptions. Notably, when FWER control is required one needs to make the necessary adjustments during power analysis and sample size calculations otherwise the nominal power will be different than the actual one.

Like this glossary entry? For an in-depth and comprehensive reading on A/B testing stats, check out the book "Statistical Methods in Online A/B Testing" by the author of this glossary, Georgi Georgiev.

Purchase Statistical Methods in Online A/B Testing

Glossary Index by Letter


Select a letter to see all A/B testing terms starting with that letter or visit the Glossary homepage to see all.