Analytics Code Wizard
Analytics Copy/Paste
Easy View Filters
Auto Spam Filters
Check Health Status
E-commerce Data Validator
Advanced URL Builder
Import Cost Data
A/B Testing Calculator
Analytics Dashboard
Statistical Calculators
A/B Test ROI Calculator
Smart Data Explorer
Google Update CheckerWhat is Family-Wise Error Rate?
Aliases: FWER
The Family-Wise Error Rate (FWER) is a concept describing the overall error rate under a composite null of no effect across a series of significance tests. There are many situations in which this can occur with the most common one being conducting a Multivariate Test which results in multiple comparisons: between the control group and each of the treatment groups. Other examples of multiple testing include testing more than one Key Performance Indicator (KPI) with a change in one of them deemed enough to call the test, as well as performing a series of significance tests on accumulating data such as in sequential testing.
There are a number of statistical methods developed to control the type I error at the family level, where family refers to a logical group of tests. Such methods include the Sidak correction, Dunnett's correction and the Bonferroni correction and each is appropriate under different assumptions. Notably, when FWER control is required one needs to make the necessary adjustments during power analysis and sample size calculations otherwise the nominal power will be different than the actual one.
Like this glossary entry? For an in-depth and comprehensive reading on A/B testing stats, check out the book "Statistical Methods in Online A/B Testing" by the author of this glossary, Georgi Georgiev.
