What does "Type II Error" mean?

What is a Type II Error?

Alias: false negative

A type II error a.k.a. an error of the second kind is committed when we fail to reject a false null hypothesis (erroneously accept the alternative hypothesis). For example, assuming the null hypothesis is that of no difference, even though there is in fact a difference of magnitude (μ) between the means of the Control Gropu and Test Group(s) we fail to observe a statistically significant difference between them after performing an online controlled experiment.

After an A/B Test is completed we have either committed a type II error, or we have not. The type II error rate is thus, inevitably, a characteristic of the testing procedure, not of the tested hypothesis. A properly designed and executed significance test offers conservative guarantees regarding the probability of committing a type II error. In fact, we know that with many different statistics we can devise an α-Uniformly Most Powerful Test so that we have optimal power given a certain sample size. However, the type II error guarantees only hold if the test actually performed conforms to the test as it was planned.

The type II error of a test is at odds with the type I error: increasing one leads to decreasing the other, and vice versa, assuming fixed variance, sample size and minimum effect of interest.

Like this glossary entry? For an in-depth and comprehensive reading on A/B testing stats, check out the book "Statistical Methods in Online A/B Testing" by the author of this glossary, Georgi Georgiev.

Articles on Type II Error

The Importance of Statistical Power in Online A/B Testing
blog.analytics-toolkit.com