What does "Type II Error" mean?

Definition of Type II Error in the context of A/B testing (online controlled experiments).

What is a Type II Error?

Aliases: false negative

A type II error a.k.a. an error of the second kind is committed when we fail to reject a false null hypothesis (erroneously accept the alternative hypothesis). For example, assuming the null hypothesis is that of no difference, even though there is in fact a difference of magnitude (μ) between the means of the Control Gropu and Test Group(s) we fail to observe a statistically significant difference between them after performing an online controlled experiment.

After an A/B Test is completed we have either committed a type II error, or we have not. The type II error rate is thus, inevitably, a characteristic of the testing procedure, not of the tested hypothesis. A properly designed and executed significance test offers conservative guarantees regarding the probability of committing a type II error. In fact, we know that with many different statistics we can devise an α-Uniformly Most Powerful Test so that we have optimal power given a certain sample size. However, the type II error guarantees only hold if the test actually performed conforms to the test as it was planned.

The type II error of a test is at odds with the type I error: increasing one leads to decreasing the other, and vice versa, assuming fixed variance, sample size and minimum effect of interest.

Articles on Type II Error

Glossary Index by Letter


Select a letter to see all A/B testing terms starting with that letter or visit the Glossary homepage to see all.