What does "Family-Wise Error Rate" mean?

What is Family-Wise Error Rate?

Aliases: FWER

The Family-Wise Error Rate (FWER) is a concept describing the overall error rate under a composite null of no effect across a series of significance tests. There are many situations in which this can occur with the most common one being conducting a Multivariate Test which results in multiple comparisons: between the control group and each of the treatment groups. Other examples of multiple testing include testing more than one Key Performance Indicator (KPI) with a change in one of them deemed enough to call the test, as well as performing a series of significance tests on accumulating data such as in sequential testing.

There are a number of statistical methods developed to control the type I error at the family level, where family refers to a logical group of tests. Such methods include the Sidak correction, Dunnett's correction and the Bonferroni correction and each is appropriate under different assumptions. Notably, when FWER control is required one needs to make the necessary adjustments during power analysis and sample size calculations otherwise the nominal power will be different than the actual one.

Related A/B Testing terms

Dunnett's Correction

Multiple Comparisons

Multiple Testing

Sidak Correction